View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002145 | OpenFOAM | Bug | public | 2016-07-10 14:25 | 2016-07-10 21:09 |
| Reporter | AlmostSurelyRob | Assigned To | henry | ||
| Priority | normal | Severity | major | Reproducibility | always |
| Status | resolved | Resolution | no change required | ||
| Platform | PPC64le | OS | Red Hat Enterprise Linux | OS Version | 7.2 |
| Summary | 0002145: y calculation in patchDist leads to large initialisation time | ||||
| Description | The calculation of patch distance present in kOmegaSST and other models causes a large initialization overhead for 1000+ domains with init time dominating over the solution. I initially came across this error in version 2.2.2, but after reviewing the code in recent versions I believe the problem is still there. I believe a pre-filtering of patches taken for consideration can overcome this problem. It seems to me that the inter-processor boundaries are taken into account which leads to N**2 complexity. Please see the attached patch file to see my proposed change. I have only changed the patchDist.C,H files. This is for version 2.2.2. I am happy to propose something for the latest version, but I see that the implementation has significantly changed there. | ||||
| Steps To Reproduce | Motorbike tutorial case after suitable rescaling (multiply each x, y and z cell counts by a factor of 20) and run on 500> mpi processes. | ||||
| Additional Information | foam-extend implementation of distance calculation has the filtering I am proposing above. | ||||
| Tags | No tags attached. | ||||
|
|
wallDist.patch (3,207 bytes)
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.C finiteVolume/fvMesh/wallDist/patchDist.C
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.C 2013-10-03 06:28:16.000000000 -0400
+++ finiteVolume/fvMesh/wallDist/patchDist.C 2016-07-09 19:49:34.120800000 -0400
@@ -27,6 +27,7 @@
#include "patchWave.H"
#include "fvMesh.H"
#include "emptyFvPatchFields.H"
+#include "wallFvPatch.H"
// * * * * * * * * * * * * * * * * Constructors * * * * * * * * * * * * * * //
@@ -48,7 +49,6 @@
mesh,
dimensionedScalar("y", dimLength, GREAT)
),
- patchIDs_(patchIDs),
correctWalls_(correctWalls),
nUnset_(0)
{
@@ -67,7 +67,16 @@
void Foam::patchDist::correct()
{
// Calculate distance starting from patch faces
- patchWave wave(mesh(), patchIDs_, correctWalls_);
+ const polyBoundaryMesh& bMesh = mesh().boundaryMesh();
+ labelHashSet wallPatchIDs(bMesh.size());
+ forAll(bMesh, patchI)
+ {
+ if (isA<wallFvPatch>(bMesh[patchI]))
+ {
+ wallPatchIDs.insert(patchI);
+ }
+ }
+ patchWave wave(mesh(), wallPatchIDs, correctWalls_);
// Transfer cell values from wave into *this
transfer(wave.distance());
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.dep finiteVolume/fvMesh/wallDist/patchDist.dep
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.dep 2016-04-21 11:28:06.701115702 -0400
+++ finiteVolume/fvMesh/wallDist/patchDist.dep 2016-07-09 19:50:03.921491600 -0400
@@ -508,6 +508,8 @@
fvMesh/wallDist/patchDist.dep: lnInclude/emptyFvPatchField.H
fvMesh/wallDist/patchDist.dep: lnInclude/emptyFvPatch.H
fvMesh/wallDist/patchDist.dep: lnInclude/emptyFvPatchField.C
+fvMesh/wallDist/patchDist.dep: lnInclude/wallFvPatch.H
+fvMesh/wallDist/patchDist.dep: $(WM_PROJECT_DIR)/src/OpenFOAM/lnInclude/wallPolyPatch.H
$(OBJECTS_DIR)/patchDist.o: $(EXE_DEP)
$(OBJECTS_DIR)/patchDist.o:
@SOURCE_DIR=fvMesh/wallDist
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.H finiteVolume/fvMesh/wallDist/patchDist.H
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/patchDist.H 2013-10-03 06:28:16.000000000 -0400
+++ finiteVolume/fvMesh/wallDist/patchDist.H 2016-07-09 19:45:03.760264000 -0400
@@ -84,9 +84,6 @@
// Private Member Data
- //- Set of patch IDs
- labelHashSet patchIDs_;
-
//- Do accurate distance calculation for near-wall cells.
bool correctWalls_;
diff -Naur /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/wallDist.C finiteVolume/fvMesh/wallDist/wallDist.C
--- /gpfs/gpfs_stage1/bbopenfoam/OpenFOAM-2.2.2/src/finiteVolume/fvMesh/wallDist/wallDist.C 2013-10-03 06:28:16.000000000 -0400
+++ finiteVolume/fvMesh/wallDist/wallDist.C 2016-07-09 19:50:01.696823000 -0400
@@ -41,7 +41,8 @@
mesh.boundaryMesh().findPatchIDs<wallPolyPatch>(),
correctWalls
)
-{}
+{
+}
// * * * * * * * * * * * * * * * * Destructor * * * * * * * * * * * * * * * //
|
|
|
Have you tried the wallDist methods in OpenFOAM-4.0/4.x and dev? The standard meshWave method is indeed slow running in parallel on large cases (to guarantee parallel consistency) but Poisson and advectionDiffusion methods are VERY fast and scale well in parallel. |
|
|
The patch you propose will run the patchWave or each processor separately which will indeed be faster but the distance to the wall will not be correct. It would be better to use the Poisson or advectionDiffusion methods which provide smooth distance functions and are parallel consistent. |
|
|
Poisson and advectionDiffusion methods are also available in OpenFOAM-3.?.? |
|
|
Thanks Henry, I will look at it presently. I think I looked at 3.0.0 as my latest version, but perhaps I didn't understand the abstraction. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2016-07-10 14:25 | AlmostSurelyRob | New Issue | |
| 2016-07-10 14:25 | AlmostSurelyRob | File Added: wallDist.patch | |
| 2016-07-10 14:57 | henry | Note Added: 0006516 | |
| 2016-07-10 15:01 | henry | Note Added: 0006517 | |
| 2016-07-10 15:28 | henry | Note Added: 0006518 | |
| 2016-07-10 20:41 | AlmostSurelyRob | Note Added: 0006519 | |
| 2016-07-10 21:09 | henry | Status | new => resolved |
| 2016-07-10 21:09 | henry | Fixed in Version | => 3.0.0 |
| 2016-07-10 21:09 | henry | Resolution | open => no change required |
| 2016-07-10 21:09 | henry | Assigned To | => henry |
