View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0003432OpenFOAMBugpublic2020-01-18 13:372020-01-21 08:54
Reporterjherb Assigned Tohenry  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
PlatformUbuntu 18.04.3 LTSOSLinuxOS Version(please specify)
Summary0003432: code freeze with #calc in file included by #includeFunc in controlDict if run in parallel
DescriptionThis seems to be a follow up bug of https://bugs.openfoam.org/view.php?id=997

If I use #includeFunc in controlDict to include another file like surfaces and in that file a #calc statement is present, the case hangs if run in parallel, but works if run in serial mode.

Here is a test based on the tutorial $FOAM_TUTORIALS/incompressible/simpleFoam/pipeCyclic/

Add this to the controlDict:
functions
{
     #includeFunc "surfaces";
}

with this system/surfaces file:

/*--------------------------------*- C++ -*----------------------------------*\
  ========= |
  \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
   \\ / O peration | Website: https://openfoam.org
    \\ / A nd | Version: 7
     \\/ M anipulation |
-------------------------------------------------------------------------------
Description
    Writes out surface files with interpolated field data in VTK format, e.g.
    cutting planes, iso-surfaces and patch boundary surfaces.

    This file includes a selection of example surfaces, each of which the user
    should configure and/or remove.

\*---------------------------------------------------------------------------*/

#includeEtc "caseDicts/postProcessing/visualization/surfaces.cfg"

posZ #calc "0.25 * 1.0";

fields (p U);

surfaces
(
    zNormal
    {
        $cuttingPlane;
        pointAndNormalDict
        {
            basePoint (0 0 $posZ); // Overrides default basePoint (0 0 0)
            normalVector $z; // $z: macro for (0 0 1)
        }
    }

);

This will hang with the following log output:
/*---------------------------------------------------------------------------*\
  ========= |
  \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
   \\ / O peration | Website: https://openfoam.org
    \\ / A nd | Version: 7
     \\/ M anipulation |
\*---------------------------------------------------------------------------*/
Build : 7-ca808c8420bf
Exec : simpleFoam -parallel
Date : Jan 18 2020
Time : 13:35:28
Host : "ip-172-31-11-62"
PID : 9241
I/O : uncollated
Case : /home/ubuntu/OpenFOAM/ubuntu-7/run/pipeCyclic
nProcs : 5
Slaves :
4
(
"ip-172-31-3-105.9974"
"ip-172-31-9-103.9949"
"ip-172-31-2-198.9691"
"ip-172-31-1-202.9688"
)

Pstream initialized with:
    floatTransfer : 0
    nProcsSimpleSum : 0
    commsType : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Using #calcEntry at line 19 in file "/home/ubuntu/OpenFOAM/ubuntu-7/run/pipeCyclic/system/surfaces"
Using #codeStream with "/home/ubuntu/OpenFOAM/ubuntu-7/run/pipeCyclic/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_7d9b040dc14b182f4607f826baa84547b7963879.so"

If #calc is used directly in controlDict, it seems to work.

Setting fileModificationChecking timeStamp seems not work. If run in serial mode, it output for the OptimisationSwitches appears after the output for doing the dynamic code compilation, so it looks like, it is too late.

The case is stored on an NFS volume imported by the slaves from the master.
TagsNo tags attached.

Relationships

duplicate of 0003414 resolvedhenry #includeFunc doesn't work in parallel with codeStream 

Activities

henry

2020-01-18 14:17

manager   ~0011079

It is likely this it fixed in OpenFOAM-dev by:

commit a227fa623844e750ec9f471655e8e0182513a954
Author: Henry Weller <http://openfoam.org>
Date: Thu Jan 9 22:05:17 2020 +0000

    functionEntry: Resolved issue with the parallel transfer of unexpanded functionEntries

commit c26037e4d9fb972f88abd221dbf60785790c4727
Author: Henry Weller <http://openfoam.org>
Date: Fri Jan 10 12:59:17 2020 +0000

    dictionary::functionEntry: Added support for efficiently passing un-expanded functionEnties between processors

jherb

2020-01-18 14:57

reporter   ~0011080

I tried the Ubuntu package of 20200112, but with this the tutorial pipeCyclic fails in decomposePar step (without my changes in controlDict):

/*---------------------------------------------------------------------------*\
  ========= |
  \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
   \\ / O peration | Website: https://openfoam.org
    \\ / A nd | Version: dev
     \\/ M anipulation |
\*---------------------------------------------------------------------------*/
Build : dev-7fdfe4224aad
Exec : decomposePar -cellDist
Date : Jan 18 2020
Time : 14:56:01
Host : "ip-172-31-1-188"
PID : 7922
I/O : uncollated
Case : /home/ubuntu/OpenFOAM/ubuntu-dev/run/pipeOrig
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time



Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod scotch

Finished decomposition in 0 s

Calculating original mesh data

Distributing cells to processors

Distributing faces to processors

Distributing points to processors

Constructing processor meshes
Reading hexRef8 data : cellLevel
Reading hexRef8 data : pointLevel
Reading hexRef8 data : level0Edge
Reading hexRef8 data : refinementHistory


--> FOAM FATAL ERROR:
names array index 192 out of range 0-2

    From function const char* Foam::NamedEnum<Enum, nEnum>::operator[](Enum) const [with Enum = Foam::coupledPolyPatch::orderingType; unsigned int nEnum = 3]
    in file lnInclude/NamedEnum.C at line 134.

FOAM exiting

henry

2020-01-18 15:06

manager   ~0011081

commit 2771c1f85c267e9925c2796b52b3ae010fae6179
Author: Henry Weller <http://openfoam.org>
Date: Mon Jan 13 19:55:18 2020 +0000

    coupledPolyPatch: Added initialisation of ordering_ data member
    
    Resolves bug-report https://bugs.openfoam.org/view.php?id=3428

jherb

2020-01-18 15:06

reporter   ~0011082

Regarding the original problem:

If I use calc in surfaces (in another case), it seems to work.

If I use it in decomposeParDict, , like:
factor #calc "8";

numberOfSubdomains $factor;

The run hangs again:
/*---------------------------------------------------------------------------*\
  ========= |
  \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
   \\ / O peration | Website: https://openfoam.org
    \\ / A nd | Version: dev
     \\/ M anipulation |
\*---------------------------------------------------------------------------*/
Build : dev-7fdfe4224aad
Exec : simpleFoam -parallel
Date : Jan 18 2020
Time : 15:05:30
Host : "ip-172-31-1-188"
PID : 11357
I/O : uncollated
Using #calcEntry at line 20 in file "/home/ubuntu/OpenFOAM/ubuntu-dev/run/clusterTest/pitzDaily_3d_01/system/decomposeParDict"
Using #codeStream with "/home/ubuntu/OpenFOAM/ubuntu-dev/run/clusterTest/pitzDaily_3d_01/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_40f05f944f1a7116c0274cafd380097e7eb37a77.so"

henry

2020-01-18 16:58

manager   ~0011085

I cannot reproduce the problem in either OpenFOAM-7 or OpenFOAM-dev. Try making the change

commit a227fa623844e750ec9f471655e8e0182513a954
Author: Henry Weller <http://openfoam.org>
Date: Thu Jan 9 22:05:17 2020 +0000

    functionEntry: Resolved issue with the parallel transfer of unexpanded functionEntries

in the OpenFOAM-7 installation you have, for me it works fine with or without this change.

jherb

2020-01-19 14:55

reporter   ~0011086

I compiled the latest master from github of OpenFOAM-dev. The I run the tutorial from $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak/
Allrun just uses single process solver, but there is a decomposeParDict. Running decomposePar and then mpirun -hostfile hostfile -n 4 interFoam -parallel
 with the unmodifed decomposeParDict works. (hostfile is a single line with localhost slots=72).

Then adding the following line to decomposeParDict results in hanging of the solver:
nos #calc "4";

(no need to use this variable somewhere, just adding #calc results in the problem)
--------------------------------------------------------------------------
/*---------------------------------------------------------------------------*\
  ========= |
  \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
   \\ / O peration | Website: https://openfoam.org
    \\ / A nd | Version: dev
     \\/ M anipulation |
\*---------------------------------------------------------------------------*/
Build : dev
Exec : interFoam -parallel
Date : Jan 19 2020
Time : 14:54:50
Host : "ip-172-31-12-218"
PID : 35121
I/O : uncollated
Using #calcEntry at line 18 in file "/home/ubuntu/OpenFOAM/ubuntu-dev/run/damBreak/system/decomposeParDict"
Using #codeStream with "/home/ubuntu/OpenFOAM/ubuntu-dev/run/damBreak/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_afb18158add5835709ac19c57d4ac4745a28c245.so"

top shows, that all four processes of interFoam run with 100 % CPU power.

jherb

2020-01-19 14:58

reporter   ~0011087

I run the whole test on a c5n.18xlarge instance on AWS using Ubuntu Server 18.04 LTS (HVM) AMI, then followed the instruction on https://openfoam.org/download/dev-ubuntu/ to install all dependencies and then downloaded OpenFOAM-dev/ThirdParty-dev to compile the latest version.

henry

2020-01-19 15:26

manager   ~0011088

Currently #calc cannot be used in decomposeParDict because it is read before the parallel IO settings are complete. This is a different problem to the one you originally reported here:

"code freeze with #calc in file included by #includeFunc in controlDict if run in parallel"

 Could you clarify if that issue is resolved?

jherb

2020-01-19 15:48

reporter   ~0011089

Yes, you are right:

#calc in places like controlDict or surfaces, included by #includeFunc seems to work in openfoam-dev (but not in openfoam-7)

#calc in decomposeParDict does not work (in both versions)

Should I open a new bug report?

henry

2020-01-19 15:58

manager   ~0011090

> #calc in places like controlDict or surfaces, included by #includeFunc seems to work in openfoam-dev (but not in openfoam-7)

I tested this in OpenFOAM-dev, 7 and 6 and it worked in all three versions.

> #calc in decomposeParDict does not work (in both versions)
> Should I open a new bug report?

This is currently a limitation rather than a bug but you can report it if it is a general issue.

henry

2020-01-19 16:01

manager   ~0011091

If I put

nos #calc "4";

in controlDict in OpenFOAM-7/tutorials/multiphase/interFoam/laminar/damBreak/damBreak

and run in parallel I get:
.
.
.
Pstream initialized with:
    floatTransfer : 0
    nProcsSimpleSum : 0
    commsType : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
SetNaN : Initialising allocated memory to NaN (FOAM_SETNAN).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Using #calcEntry at line 18 in file "/home/dm2/henry/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interFoam/laminar/damBreak/damBreak/system/controlDict"
Using #codeStream with "/home/dm2/henry/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interFoam/laminar/damBreak/damBreak/dynamicCode/platforms/linux64GccDPInt32Opt/lib/libcodeStream_afb18158add5835709ac19c57d4ac4745a28c245.so"
Create mesh for time = 0

PIMPLE: No convergence criteria found

PIMPLE: Operating solver in transient mode with 1 outer corrector
PIMPLE: Operating solver in PISO mode
.
.
.

and it runs to completion.

jherb

2020-01-19 18:08

reporter   ~0011092

This seems to be computer dependent (unfortunately).

If I add #calc to controlDict, it seems to work on all computers (with OpenFOAM-7).

If I add #calc to surfaces (#includeFunc surfaces in controlDict), it works on my local computer, but it stalls on EC2. (with OpenFOAM-7). Using OpenFOAM-dev, this works.

henry

2020-01-20 10:50

manager   ~0011093

It is not clear if you are running the latest OpenFOAM-7, you will need

commit 37fa4bc3407472cc4faede31236cd3916f033a6a
Author: Henry Weller <http://openfoam.org>
Date: Thu Dec 19 16:49:35 2019 +0000

    functionObjectList: Set parent dictionary for temporary function dictionary
    
    so that it inherits the IO properties of the top dictionary.
    
    Resolves bug-report https://bugs.openfoam.org/view.php?id=3414

jherb

2020-01-20 23:49

reporter   ~0011094

I can confirm, that using the lasted Ubuntu package (20200120) for OpenFOAM-7 solves the problem also on AWS. (I had used OpenFOAM-7 git repository on the local PC already)

So the original problem has been solved (probably by the commit mentioned in comment https://bugs.openfoam.org/view.php?id=3432#c11093)

Thank you for pointing this out.

Issue History

Date Modified Username Field Change
2020-01-18 13:37 jherb New Issue
2020-01-18 14:17 henry Note Added: 0011079
2020-01-18 14:57 jherb Note Added: 0011080
2020-01-18 15:06 henry Note Added: 0011081
2020-01-18 15:06 jherb Note Added: 0011082
2020-01-18 16:58 henry Note Added: 0011085
2020-01-19 14:55 jherb Note Added: 0011086
2020-01-19 14:58 jherb Note Added: 0011087
2020-01-19 15:26 henry Note Added: 0011088
2020-01-19 15:48 jherb Note Added: 0011089
2020-01-19 15:58 henry Note Added: 0011090
2020-01-19 16:01 henry Note Added: 0011091
2020-01-19 18:08 jherb Note Added: 0011092
2020-01-20 10:50 henry Note Added: 0011093
2020-01-20 23:49 jherb Note Added: 0011094
2020-01-21 08:54 henry Assigned To => henry
2020-01-21 08:54 henry Status new => closed
2020-01-21 08:54 henry Resolution open => fixed
2020-01-21 08:54 henry Fixed in Version => 7
2020-01-21 08:54 henry Relationship added duplicate of 0003414