View Issue Details

IDProjectCategoryView StatusLast Update
0000982OpenFOAMBugpublic2016-02-03 16:32
Reporteruser19Assigned Tohenry  
Status resolvedResolutionfixed 
Summary0000982: libhelptypes segfaults/call stack overflows on large DTAGS files and is very slow
DescriptionRunning foamHelp with my 18Mb DTAGS file first crashes because the recursive doxygenXmlParse::skipForward() function causes a call stack overlow, and once that's fixed by replacing the recursion trivially with a loop, parsing is hopelessly slow (I aborted after 50% and 20 minutes of waiting... And yes, I added a progress indicator to find out.)
Additional InformationAttached patch replaces the hand-crafted parser with the MIT-licensed, lightweight pugixml library. This results in almost instantaneous loading of the DTAGS file and much simplified and easier to understand data extraction.



2013-08-30 21:29



2016-02-03 09:47

manager   ~0005891

Adding the pugixml library adds a lot of complexity to maintain. Given that we only need simple parsing of the XML it might be possible to improve parsing performance with minor change to the existing code. Have you performed profiling to find out where the algorithm needs to be improved?


2016-02-03 16:32

manager   ~0005900

For me the parsing of the DTAGS file took ~2s rather than ~40min you found.

I did some profiling on doxygenXmlParser and optimised it:

    doxygenXmlParser: pre-construct and compile constant regular expressions
    and pre-construct constant strings and names
    Speeds-up parsing of DTAGS so that lookup of a name takes ~1s

Resolved in OpenFOAM-dev by commit b2d5ff67b06a32ddcd90c0855fd5e9543094ccb8

Issue History

Date Modified Username Field Change
2013-08-30 21:29 user19 New Issue
2013-08-30 21:29 user19 File Added: 0001-ENH-Use-pugixml-as-the-XML-engine-in-libhelptypes.patch
2015-08-17 01:23 wyldckat Tag Attached: contribution
2016-02-03 09:47 henry Note Added: 0005891
2016-02-03 16:32 henry Note Added: 0005900
2016-02-03 16:32 henry Status new => resolved
2016-02-03 16:32 henry Resolution open => fixed
2016-02-03 16:32 henry Assigned To => henry