|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0002688||OpenFOAM||Feature||public||2017-09-05 16:00||2018-07-10 11:19|
|Target Version||Fixed in Version|
|Summary||0002688: Barycentric Tracking Makes DSMC Simulations Significantly Slower|
|Description||Since barycentric particle tracking was introduced, DSMC simulations have become significantly slower. I have performed the same simululation on three different versions of OpenFOAM, all compiled from source on my PC with the same optimisation flags. OpenFOAM-3.0.x is from before the new barycentric tracking was introduced, 5.0 and dev with all the latest commits (up to 5/9/17) are from after. The results for how long each takes to complete 410 time steps are below:|
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-dev = 1151s
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-5.0 = 1184s
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-3.0.x = 743s
I performed some basic code profiling and found that the majoritry of the computational time is spent in the lagrangian shared object library:
12539581 100.000 dsmcFoam
8969869 71.5324 liblagrangian.so
2092447 16.6867 dsmcFoam
922917 7.3600 libOpenFOAM.so
283972 2.2646 libc-2.19.so
145144 1.1575 libstdc++.so.6.0.19
From the call graph, I identified these functions as the most expensive:
23.5699% liblagrangian.so Foam::particle::stationaryTetGeometry(Foam::Vector<double>&, Foam::Vector<double>&, Foam::Vector<double>&, Foam::Vector<double>&) const
20.7832% liblagrangian.so Foam::particle::changeFace(int)
14.5903% liblagrangian.so Foam::particle::trackToStationaryTri(Foam::Vector<double> const&, double, int&)
4.5102% liblagrangian.so Foam::particle::stationaryTetReverseTransform(Foam::Vector<double>&, double&, Foam::BarycentricTensor<double>&) const
|Steps To Reproduce||Simulation I used to get the timings is attached.|
|Tags||No tags attached.|
|Corrected timings in original note|
There are two issues...
1) The main one is that the "p.constrainToMeshCentre()" step that happens before tracking now actually performs a track itself. The old code just used to move the particle's position, regardless of how the topology might change. Barycentric tracking is not tolerant to this, so we track instead. Unfortunately this is essentially doubling the number of tracking calls. This will only be an issue on 2D cases like this one. If you compare the speed of 3D cases, this problem will not be evident. We can optimise this away by adding the correction to the tracking displacement and performing a single track.
2) The other issue is inlining. The old tracking functions were all implemented in the header and were explicitly inlined. I put the new ones in the source file and removed the inlining. Developing when everything is inlined is a complete pain, and I thought the functions were large enough that the benefit of inlining would be negligible. Turns out, I was wrong on that last point. If we inline the entirety of the particle class, we get a noticeable speed up.
My timings are as follows (note that my computer is clearly not as good as yours):
4.x: 1192 s
dev: 1845 s (+54.8%)
dev with fix (1): 1364 s (+14.4%)
dev with fixes (1) and (2): 1199 s (+0.6%)
I don't want to inline the entire particle class if I don't have to. It would be beneficial if someone could go though the particle class, inlining a few functions at a time (small ones first), to see which ones are important.
A patch which fixes issue (1) is attached. Note that this change will also need making to the kinematic parcel and (possibly) the injection models.
Fix (1) has been pushed to dev as commit 6832a4375
Has any investigation been made as to the effects of inlining parts of the particle class? @craigwhite87, is this something you would be able to do?
Thanks for this so far Will. I have pulled the changes and can confirm that the timing on my PC is now:
OpenFOAM-dev: 918 s (~20% speed up from when I submitted the report).
I will try to inline some of the tracking functions. It's not something I have a lot of experience with, but I will use the source code from older versions of OpenFOAM as examples.
|Please reopen when there is progress on the inlining.|
|2017-09-05 16:00||craigwhite87||New Issue|
|2017-09-05 16:00||craigwhite87||File Added: dacMonacoFlatPlateNormal.tar.gz|
|2017-09-05 16:00||craigwhite87||Tag Attached: Lagrangian; barycentric particle tracking|
|2017-09-05 16:09||craigwhite87||Note Added: 0008698|
|2017-09-05 16:38||chris||Description Updated||View Revisions|
|2017-09-05 16:38||chris||Note Deleted: 0008698|
|2017-09-05 16:38||chris||Note Added: 0008701|
|2017-09-07 09:55||will||File Added: patch|
|2017-09-07 09:55||will||Note Added: 0008726|
|2017-09-12 08:52||will||Note Added: 0008754|
|2017-09-12 13:10||craigwhite87||Note Added: 0008759|
|2017-11-02 11:29||will||Assigned To||=> will|
|2017-11-02 11:29||will||Status||new => closed|
|2017-11-02 11:29||will||Resolution||open => fixed|
|2017-11-02 11:29||will||Note Added: 0008979|
|2017-11-02 11:30||will||Status||closed => resolved|
|2018-07-10 11:19||administrator||Tag Detached: Lagrangian; barycentric particle tracking|