View Issue Details

IDProjectCategoryView StatusLast Update
0002145OpenFOAMBugpublic2016-07-10 21:09
ReporterAlmostSurelyRob Assigned Tohenry  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionno change required 
PlatformPPC64leOSRed Hat Enterprise Linux OS Version7.2
Summary0002145: y calculation in patchDist leads to large initialisation time
DescriptionThe calculation of patch distance present in kOmegaSST and other models causes a large initialization overhead for 1000+ domains with init time dominating over the solution.

I initially came across this error in version 2.2.2, but after reviewing the code in recent versions I believe the problem is still there. I believe a pre-filtering of patches taken for consideration can overcome this problem. It seems to me that the inter-processor boundaries are taken into account which leads to N**2 complexity.

Please see the attached patch file to see my proposed change. I have only changed the patchDist.C,H files. This is for version 2.2.2. I am happy to propose something for the latest version, but I see that the implementation has significantly changed there.
Steps To ReproduceMotorbike tutorial case after suitable rescaling (multiply each x, y and z cell counts by a factor of 20) and run on 500> mpi processes.
Additional Informationfoam-extend implementation of distance calculation has the filtering I am proposing above.
TagsNo tags attached.

Activities

AlmostSurelyRob

2016-07-10 14:25

reporter  

wallDist.patch (3,207 bytes)   
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.C finiteVolume/fvMesh/wallDist/patchDist.C
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.C	2013-10-03 06:28:16.000000000 -0400
+++ finiteVolume/fvMesh/wallDist/patchDist.C	2016-07-09 19:49:34.120800000 -0400
@@ -27,6 +27,7 @@
 #include "patchWave.H"
 #include "fvMesh.H"
 #include "emptyFvPatchFields.H"
+#include "wallFvPatch.H"
 
 // * * * * * * * * * * * * * * * * Constructors  * * * * * * * * * * * * * * //
 
@@ -48,7 +49,6 @@
         mesh,
         dimensionedScalar("y", dimLength, GREAT)
     ),
-    patchIDs_(patchIDs),
     correctWalls_(correctWalls),
     nUnset_(0)
 {
@@ -67,7 +67,16 @@
 void Foam::patchDist::correct()
 {
     // Calculate distance starting from patch faces
-    patchWave wave(mesh(), patchIDs_, correctWalls_);
+    const polyBoundaryMesh& bMesh = mesh().boundaryMesh();
+    labelHashSet wallPatchIDs(bMesh.size());
+    forAll(bMesh, patchI)
+    {
+        if (isA<wallFvPatch>(bMesh[patchI]))
+        {
+            wallPatchIDs.insert(patchI);
+        }
+    }
+    patchWave wave(mesh(), wallPatchIDs, correctWalls_);
 
     // Transfer cell values from wave into *this
     transfer(wave.distance());
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.dep finiteVolume/fvMesh/wallDist/patchDist.dep
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.dep	2016-04-21 11:28:06.701115702 -0400
+++ finiteVolume/fvMesh/wallDist/patchDist.dep	2016-07-09 19:50:03.921491600 -0400
@@ -508,6 +508,8 @@
 fvMesh/wallDist/patchDist.dep: lnInclude/emptyFvPatchField.H
 fvMesh/wallDist/patchDist.dep: lnInclude/emptyFvPatch.H
 fvMesh/wallDist/patchDist.dep: lnInclude/emptyFvPatchField.C
+fvMesh/wallDist/patchDist.dep: lnInclude/wallFvPatch.H
+fvMesh/wallDist/patchDist.dep: $(WM_PROJECT_DIR)/src/OpenFOAM/lnInclude/wallPolyPatch.H
 $(OBJECTS_DIR)/patchDist.o: $(EXE_DEP)
 $(OBJECTS_DIR)/patchDist.o:
 	@SOURCE_DIR=fvMesh/wallDist
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.H finiteVolume/fvMesh/wallDist/patchDist.H
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.H	2013-10-03 06:28:16.000000000 -0400
+++ finiteVolume/fvMesh/wallDist/patchDist.H	2016-07-09 19:45:03.760264000 -0400
@@ -84,9 +84,6 @@
 
     // Private Member Data
 
-        //- Set of patch IDs
-        labelHashSet patchIDs_;
-
         //- Do accurate distance calculation for near-wall cells.
         bool correctWalls_;
 
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/wallDist.C finiteVolume/fvMesh/wallDist/wallDist.C
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/wallDist.C	2013-10-03 06:28:16.000000000 -0400
+++ finiteVolume/fvMesh/wallDist/wallDist.C	2016-07-09 19:50:01.696823000 -0400
@@ -41,7 +41,8 @@
         mesh.boundaryMesh().findPatchIDs<wallPolyPatch>(),
         correctWalls
     )
-{}
+{
+}
 
 
 // * * * * * * * * * * * * * * * * Destructor  * * * * * * * * * * * * * * * //
wallDist.patch (3,207 bytes)   

henry

2016-07-10 14:57

manager   ~0006516

Have you tried the wallDist methods in OpenFOAM-4.0/4.x and dev? The standard meshWave method is indeed slow running in parallel on large cases (to guarantee parallel consistency) but Poisson and advectionDiffusion methods are VERY fast and scale well in parallel.

henry

2016-07-10 15:01

manager   ~0006517

The patch you propose will run the patchWave or each processor separately which will indeed be faster but the distance to the wall will not be correct. It would be better to use the Poisson or advectionDiffusion methods which provide smooth distance functions and are parallel consistent.

henry

2016-07-10 15:28

manager   ~0006518

Poisson and advectionDiffusion methods are also available in OpenFOAM-3.?.?

AlmostSurelyRob

2016-07-10 20:41

reporter   ~0006519

Thanks Henry, I will look at it presently. I think I looked at 3.0.0 as my latest version, but perhaps I didn't understand the abstraction.

Issue History

Date Modified Username Field Change
2016-07-10 14:25 AlmostSurelyRob New Issue
2016-07-10 14:25 AlmostSurelyRob File Added: wallDist.patch
2016-07-10 14:57 henry Note Added: 0006516
2016-07-10 15:01 henry Note Added: 0006517
2016-07-10 15:28 henry Note Added: 0006518
2016-07-10 20:41 AlmostSurelyRob Note Added: 0006519
2016-07-10 21:09 henry Status new => resolved
2016-07-10 21:09 henry Fixed in Version => 3.0.0
2016-07-10 21:09 henry Resolution open => no change required
2016-07-10 21:09 henry Assigned To => henry