View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0002401OpenFOAMBugpublic2016-12-16 09:212016-12-19 16:25
Reporterkevnor Assigned Tohenry  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Summary0002401: Bad parallel performance of volRegion function objects
DescriptionThe performance of the volRegion function object on large parallel cases is bad (and in extreme cases can lead to out-of-memory issues), due to the reconstruction of the entire field on the master process before processing is performed.

A better solution would be for the relevant processing to be performed on the separate processors and then combined as appropriate onto the master.
Steps To ReproduceConstruct any large case with e.g. > 100M cells and perform parallel decomposition.

Run parallel post-processing with any volRegion function object and note that evaluation of function object is expensive!
TagsfunctionObject, volRegion

Activities

henry

2016-12-16 09:55

manager   ~0007495

I agree that the current way 'volRegion' works is inefficient running in parallel and should be re-written. Can you provide a patch for this?

kevnor

2016-12-16 15:26

reporter   ~0007503

Here's a potential patch. I've tested it with all available operations on a small number of cases in both serial and parallel and it appears to give identical results to the previous implementation, but functions significantly faster in parallel.

I guess that something similar should be done for the surfaceRegion functionObject as well, though there's less of an efficiency issue there due to the lower dimensionality.
volRegionTemplates.patch (3,027 bytes)    
diff --git src/functionObjects/field/fieldValues/volRegion/volRegionTemplates.C src/functionObjects/field/fieldValues/volRegion/volRegionTemplates.C
index d3741f7..616cac7 100644
--- src/functionObjects/field/fieldValues/volRegion/volRegionTemplates.C
+++ src/functionObjects/field/fieldValues/volRegion/volRegionTemplates.C
@@ -84,52 +84,52 @@ Type Foam::functionObjects::fieldValues::volRegion::processValues
     {
         case opSum:
         {
-            result = sum(values);
+            result = gSum(values);
             break;
         }
         case opSumMag:
         {
-            result = sum(cmptMag(values));
+            result = gSum(cmptMag(values));
             break;
         }
         case opAverage:
         {
-            result = sum(values)/values.size();
+	  result = gSum(values)/nCells_;
             break;
         }
         case opWeightedAverage:
         {
-            result = sum(weightField*values)/sum(weightField);
+            result = gSum(weightField*values)/gSum(weightField);
             break;
         }
         case opVolAverage:
         {
-            result = sum(V*values)/sum(V);
+            result = gSum(V*values)/volume_;
             break;
         }
         case opWeightedVolAverage:
         {
-            result = sum(weightField*V*values)/sum(weightField*V);
+            result = gSum(weightField*V*values)/gSum(weightField*V);
             break;
         }
         case opVolIntegrate:
         {
-            result = sum(V*values);
+            result = gSum(V*values);
             break;
         }
         case opMin:
         {
-            result = min(values);
+            result = gMin(values);
             break;
         }
         case opMax:
         {
-            result = max(values);
+            result = gMax(values);
             break;
         }
         case opCoV:
         {
-            Type meanValue = sum(values*V)/sum(V);
+            Type meanValue = gSum(values*V)/volume_;
 
             const label nComp = pTraits<Type>::nComponents;
 
@@ -139,7 +139,7 @@ Type Foam::functionObjects::fieldValues::volRegion::processValues
                 scalar mean = component(meanValue, d);
                 scalar& res = setComponent(result, d);
 
-                res = sqrt(sum(V*sqr(vals - mean))/sum(V))/mean;
+                res = sqrt(gSum(V*sqr(vals - mean))/volume_)/mean;
             }
 
             break;
@@ -173,14 +173,10 @@ bool Foam::functionObjects::fieldValues::volRegion::writeValues
             weightField = setFieldValues<scalar>(weightFieldName_, true);
         }
 
-        // Combine onto master
-        combineFields(values);
-        combineFields(V);
-        combineFields(weightField);
+	Type result = processValues(values, V, weightField);
 
         if (Pstream::master())
         {
-            Type result = processValues(values, V, weightField);
 
             // Add to result dictionary, over-writing any previous entry
             resultDict_.add(fieldName, result, true);
volRegionTemplates.patch (3,027 bytes)   

henry

2016-12-19 16:25

manager   ~0007531

Thanks for the patch. I have updated it to conform to
http://openfoam.org/dev/coding-style-guide/

and applied it to OpenFOAM-4.x:

commit a7c2bd6cacf12110af6587378c96b98f003645d5

However it cannot be applied to OpenFOAM-dev due to significant changes in the handling of volRegion. I will make the corresponding changes shortly.

Issue History

Date Modified Username Field Change
2016-12-16 09:21 kevnor New Issue
2016-12-16 09:21 kevnor Tag Attached: functionObject
2016-12-16 09:21 kevnor Tag Attached: volRegion
2016-12-16 09:55 henry Note Added: 0007495
2016-12-16 15:26 kevnor File Added: volRegionTemplates.patch
2016-12-16 15:26 kevnor Note Added: 0007503
2016-12-19 16:25 henry Assigned To => henry
2016-12-19 16:25 henry Status new => resolved
2016-12-19 16:25 henry Resolution open => fixed
2016-12-19 16:25 henry Note Added: 0007531