View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0002688OpenFOAMFeaturepublic2017-09-05 16:002018-07-10 11:19
Reportercraigwhite87 Assigned Towill  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
PlatformGNU/LinuxOSUbuntuOS Version14.04
Product Versiondev 
Summary0002688: Barycentric Tracking Makes DSMC Simulations Significantly Slower
DescriptionSince barycentric particle tracking was introduced, DSMC simulations have become significantly slower. I have performed the same simululation on three different versions of OpenFOAM, all compiled from source on my PC with the same optimisation flags. OpenFOAM-3.0.x is from before the new barycentric tracking was introduced, 5.0 and dev with all the latest commits (up to 5/9/17) are from after. The results for how long each takes to complete 410 time steps are below:

Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-dev = 1151s
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-5.0 = 1184s
Time to reach 0.000127182s of simulated time (410 time steps) OpenFOAM-3.0.x = 743s

I performed some basic code profiling and found that the majoritry of the computational time is spent in the lagrangian shared object library:

12539581 100.000 dsmcFoam
    CPU_CLK_UNHALT...|
      samples| %|
    ------------------
      8969869 71.5324 liblagrangian.so
      2092447 16.6867 dsmcFoam
       922917 7.3600 libOpenFOAM.so
       283972 2.2646 libc-2.19.so
       145144 1.1575 libstdc++.so.6.0.19

From the call graph, I identified these functions as the most expensive:

23.5699% liblagrangian.so Foam::particle::stationaryTetGeometry(Foam::Vector<double>&, Foam::Vector<double>&, Foam::Vector<double>&, Foam::Vector<double>&) const

20.7832% liblagrangian.so Foam::particle::changeFace(int)

14.5903% liblagrangian.so Foam::particle::trackToStationaryTri(Foam::Vector<double> const&, double, int&)

4.5102% liblagrangian.so Foam::particle::stationaryTetReverseTransform(Foam::Vector<double>&, double&, Foam::BarycentricTensor<double>&) const
Steps To ReproduceSimulation I used to get the timings is attached.

blockMesh
dsmcInitialise
dsmcFoam

TagsNo tags attached.

Activities

craigwhite87

2017-09-05 16:00

reporter  

dacMonacoFlatPlateNormal.tar.gz (3,273 bytes)

chris

2017-09-05 16:38

manager   ~0008701

Corrected timings in original note

will

2017-09-07 09:55

manager  

patch (1,187 bytes)    
diff --git a/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C b/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C
index c5c4e5623..177149cd4 100644
--- a/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C
+++ b/src/lagrangian/DSMC/parcels/Templates/DSMCParcel/DSMCParcel.C
@@ -53,8 +53,11 @@ bool Foam::DSMCParcel<ParcelType>::move
 
     while (td.keepParticle && !td.switchProcessor && p.stepFraction() < 1)
     {
-        // Apply correction to position for reduced-D cases
-        p.constrainToMeshCentre();
+        // Calculate the correction to the position for reduced-D cases
+        const vector pos0 = p.position();
+        vector pos1 = pos0;
+        meshTools::constrainToMeshCentre(p.mesh(), pos1);
+        const vector dPos = pos1 - pos0;
 
         Utracking = U_;
 
@@ -63,7 +66,7 @@ bool Foam::DSMCParcel<ParcelType>::move
         meshTools::constrainDirection(mesh, mesh.solutionD(), Utracking);
 
         const scalar f = 1 - p.stepFraction();
-        p.trackToAndHitFace(f*trackTime*Utracking, f, cloud, td);
+        p.trackToAndHitFace(f*trackTime*Utracking + dPos, f, cloud, td);
     }
 
     return td.keepParticle;
patch (1,187 bytes)   

will

2017-09-07 09:55

manager   ~0008726

There are two issues...

1) The main one is that the "p.constrainToMeshCentre()" step that happens before tracking now actually performs a track itself. The old code just used to move the particle's position, regardless of how the topology might change. Barycentric tracking is not tolerant to this, so we track instead. Unfortunately this is essentially doubling the number of tracking calls. This will only be an issue on 2D cases like this one. If you compare the speed of 3D cases, this problem will not be evident. We can optimise this away by adding the correction to the tracking displacement and performing a single track.

2) The other issue is inlining. The old tracking functions were all implemented in the header and were explicitly inlined. I put the new ones in the source file and removed the inlining. Developing when everything is inlined is a complete pain, and I thought the functions were large enough that the benefit of inlining would be negligible. Turns out, I was wrong on that last point. If we inline the entirety of the particle class, we get a noticeable speed up.

My timings are as follows (note that my computer is clearly not as good as yours):

4.x: 1192 s
dev: 1845 s (+54.8%)
dev with fix (1): 1364 s (+14.4%)
dev with fixes (1) and (2): 1199 s (+0.6%)

I don't want to inline the entire particle class if I don't have to. It would be beneficial if someone could go though the particle class, inlining a few functions at a time (small ones first), to see which ones are important.

A patch which fixes issue (1) is attached. Note that this change will also need making to the kinematic parcel and (possibly) the injection models.

will

2017-09-12 08:52

manager   ~0008754

Fix (1) has been pushed to dev as commit 6832a4375

Has any investigation been made as to the effects of inlining parts of the particle class? @craigwhite87, is this something you would be able to do?

craigwhite87

2017-09-12 13:10

reporter   ~0008759

Thanks for this so far Will. I have pulled the changes and can confirm that the timing on my PC is now:

OpenFOAM-dev: 918 s (~20% speed up from when I submitted the report).

I will try to inline some of the tracking functions. It's not something I have a lot of experience with, but I will use the source code from older versions of OpenFOAM as examples.

will

2017-11-02 11:29

manager   ~0008979

Please reopen when there is progress on the inlining.

Issue History

Date Modified Username Field Change
2017-09-05 16:00 craigwhite87 New Issue
2017-09-05 16:00 craigwhite87 File Added: dacMonacoFlatPlateNormal.tar.gz
2017-09-05 16:00 craigwhite87 Tag Attached: Lagrangian; barycentric particle tracking
2017-09-05 16:38 chris Description Updated
2017-09-05 16:38 chris Note Added: 0008701
2017-09-07 09:55 will File Added: patch
2017-09-07 09:55 will Note Added: 0008726
2017-09-12 08:52 will Note Added: 0008754
2017-09-12 13:10 craigwhite87 Note Added: 0008759
2017-11-02 11:29 will Assigned To => will
2017-11-02 11:29 will Status new => closed
2017-11-02 11:29 will Resolution open => fixed
2017-11-02 11:29 will Note Added: 0008979
2017-11-02 11:30 will Status closed => resolved
2018-07-10 11:19 administrator Tag Detached: Lagrangian; barycentric particle tracking