2018-03-21 03:21 GMT

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0002688OpenFOAMFeaturepublic2018-03-08 13:52
Assigned Towill 
PlatformGNU/LinuxOSUbuntuOS Version14.04
Product Versiondev 
Target VersionFixed in Version 
Summary0002688: Barycentric Tracking Makes DSMC Simulations Significantly Slower
DescriptionSince barycentric particle tracking was introduced, DSMC simulations have become significantly slower. I have performed the same simululation on three different versions of OpenFOAM, all compiled from source on my PC with the same optimisation flags. OpenFOAM-3.0.x is from before the new barycentric tracking was introduced, 5.0 and dev with all the latest commits (up to 5/9/17) are from after. The results for how long each takes to complete 410 time steps are below:

Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-dev = 1151s
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-5.0 = 1184s
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-3.0.x = 743s

I performed some basic code profiling and found that the majoritry of the computational time is spent in the lagrangian shared object library:

12539581 100.000 dsmcFoam
      samples| %|
      8969869 71.5324 liblagrangian.so
      2092447 16.6867 dsmcFoam
       922917 7.3600 libOpenFOAM.so
       283972 2.2646 libc-2.19.so
       145144 1.1575 libstdc++.so.6.0.19

From the call graph, I identified these functions as the most expensive:

23.5699% liblagrangian.so Foam::particle::stationaryTetGeometry(Foam::Vector<double>&, Foam::Vector<double>&, Foam::Vector<double>&, Foam::Vector<double>&) const

20.7832% liblagrangian.so Foam::particle::changeFace(int)

14.5903% liblagrangian.so Foam::particle::trackToStationaryTri(Foam::Vector<double> const&, double, int&)

4.5102% liblagrangian.so Foam::particle::stationaryTetReverseTransform(Foam::Vector<double>&, double&, Foam::BarycentricTensor<double>&) const
Steps To ReproduceSimulation I used to get the timings is attached.


TagsLagrangian; barycentric particle tracking
Attached Files
  • gz file icon dacMonacoFlatPlateNormal.tar.gz (3,273 bytes) 2017-09-05 16:00
  • ? file icon patch (1,187 bytes) 2017-09-07 09:55 -
    diff --git a/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C b/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C
    index c5c4e5623..177149cd4 100644
    --- a/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C
    +++ b/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C
    @@ -53,8 +53,11 @@ bool Foam::DSMCParcel<ParcelType>::move
         while (td.keepParticle && !td.switchProcessor && p.stepFraction() < 1)
    -        // Apply correction to position for reduced-D cases
    -        p.constrainToMeshCentre();
    +        // Calculate the correction to the position for reduced-D cases
    +        const vector pos0 = p.position();
    +        vector pos1 = pos0;
    +        meshTools::constrainToMeshCentre(p.mesh(), pos1);
    +        const vector dPos = pos1 - pos0;
             Utracking = U_;
    @@ -63,7 +66,7 @@ bool Foam::DSMCParcel<ParcelType>::move
             meshTools::constrainDirection(mesh, mesh.solutionD(), Utracking);
             const scalar f = 1 - p.stepFraction();
    -        p.trackToAndHitFace(f*trackTime*Utracking, f, cloud, td);
    +        p.trackToAndHitFace(f*trackTime*Utracking + dPos, f, cloud, td);
         return td.keepParticle;
    ? file icon patch (1,187 bytes) 2017-09-07 09:55 +




chris (manager)

Corrected timings in original note


will (manager)

There are two issues...

1) The main one is that the "p.constrainToMeshCentre()" step that happens before tracking now actually performs a track itself. The old code just used to move the particle's position, regardless of how the topology might change. Barycentric tracking is not tolerant to this, so we track instead. Unfortunately this is essentially doubling the number of tracking calls. This will only be an issue on 2D cases like this one. If you compare the speed of 3D cases, this problem will not be evident. We can optimise this away by adding the correction to the tracking displacement and performing a single track.

2) The other issue is inlining. The old tracking functions were all implemented in the header and were explicitly inlined. I put the new ones in the source file and removed the inlining. Developing when everything is inlined is a complete pain, and I thought the functions were large enough that the benefit of inlining would be negligible. Turns out, I was wrong on that last point. If we inline the entirety of the particle class, we get a noticeable speed up.

My timings are as follows (note that my computer is clearly not as good as yours):

4.x: 1192 s
dev: 1845 s (+54.8%)
dev with fix (1): 1364 s (+14.4%)
dev with fixes (1) and (2): 1199 s (+0.6%)

I don't want to inline the entire particle class if I don't have to. It would be beneficial if someone could go though the particle class, inlining a few functions at a time (small ones first), to see which ones are important.

A patch which fixes issue (1) is attached. Note that this change will also need making to the kinematic parcel and (possibly) the injection models.


will (manager)

Fix (1) has been pushed to dev as commit 6832a4375

Has any investigation been made as to the effects of inlining parts of the particle class? @craigwhite87, is this something you would be able to do?


craigwhite87 (reporter)

Thanks for this so far Will. I have pulled the changes and can confirm that the timing on my PC is now:

OpenFOAM-dev: 918 s (~20% speed up from when I submitted the report).

I will try to inline some of the tracking functions. It's not something I have a lot of experience with, but I will use the source code from older versions of OpenFOAM as examples.


will (manager)

Please reopen when there is progress on the inlining.

-Issue History
Date Modified Username Field Change
2017-09-05 16:00 craigwhite87 New Issue
2017-09-05 16:00 craigwhite87 File Added: dacMonacoFlatPlateNormal.tar.gz
2017-09-05 16:00 craigwhite87 Tag Attached: Lagrangian; barycentric particle tracking
2017-09-05 16:09 craigwhite87 Note Added: 0008698
2017-09-05 16:38 chris Description Updated View Revisions
2017-09-05 16:38 chris Note Deleted: 0008698
2017-09-05 16:38 chris Note Added: 0008701
2017-09-07 09:55 will File Added: patch
2017-09-07 09:55 will Note Added: 0008726
2017-09-12 08:52 will Note Added: 0008754
2017-09-12 13:10 craigwhite87 Note Added: 0008759
2017-11-02 11:29 will Assigned To => will
2017-11-02 11:29 will Status new => closed
2017-11-02 11:29 will Resolution open => fixed
2017-11-02 11:29 will Note Added: 0008979
2017-11-02 11:30 will Status closed => resolved
+Issue History