View Issue Details

IDProjectCategoryView StatusLast Update
0001664OpenFOAMBugpublic2015-10-22 09:51
Reporteruser1044Assigned Tohenry  
PriorityhighSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
PlatformLinuxOSOpenSuSEOS Version11.2
Summary0001664: Pressure discontinuity in the parallel calculation of pimpleDyMFoam with cyclicACMI patches
DescriptionIn the parallel calculation of pimpleDyMFoam with cyclicACMI patches by a workaround of the scotch decomposition, which is shown in the Bug Issue ID:1450, the pressure distribution is broken in the first time step.
Note that in the single calculation of pimpleDyMFoam with the same model, this symptom is not reproduced.
Steps To Reproduce1.I created the OpenFOAM Case files with cyclicAMI patches, which includes the revolving body, and have been correctly executed the Case with both of the single and parallel calculations of pimpleDyMFoam, and confirmed the correct pressure distributions up to 10 time steps.

2.I executed the Case files, in which the cyclicAMI patches are replaced to the cyclicACMI patches, with the pimpleDyMFoam, and compared the pressure distributions of the 1st and 10th time step in both of the single and parallel calculations. The pressure discontinuities appeared only in the 1st step of the parallel calculation.
Note that I used a workaround of the scotch decomposition, which is shown in the Bug Issue ID:1450, for the purpose of the parallel execution of the cyclicACMI case.

3.Further, I executed the continuous calculations from the 10th time step to the 20th time step, and could find the similar discontinuity in the 11th time step of the parallel calculation again.
Additional InformationI will attach the pdf file of the explanation with some figures and the OpenFOAM Case files including the revolving body with the cyclicACMI patches .
TagsACMI, cyclicACMI, decomposePar, Solver

Activities

user1044

2015-04-20 07:56

 

user1044

2015-04-20 08:05

 

user1044

2015-04-20 08:06

 

user1044

2015-04-20 08:06

 

user1044

2015-04-20 08:07

 

ACMI_pimpleDyMFoam_RotatingMotion.tar.ba_ (375 bytes)   
copy /b "ACMI_pimpleDyMFoam_RotatingMotion.tar.0"  "ACMI_pimpleDyMFoam_RotatingMotion.tar.gz"
copy /b "ACMI_pimpleDyMFoam_RotatingMotion.tar.gz" + "ACMI_pimpleDyMFoam_RotatingMotion.tar.1"  "ACMI_pimpleDyMFoam_RotatingMotion.tar.gz"
copy /b "ACMI_pimpleDyMFoam_RotatingMotion.tar.gz" + "ACMI_pimpleDyMFoam_RotatingMotion.tar.2"  "ACMI_pimpleDyMFoam_RotatingMotion.tar.gz"

user4

2015-04-20 09:44

  ~0004629

There should be no real difference in parallel and non-parallel behaviour. Do you see the same issue if you
- make sure the nonOverlapPatch does not use fixedValue, e.g. use uniformFixedValue instead of fixedValue. The ACMI overwrites the value field and expect the nonOverlap boundary conditions to be able to recreate its value.
- and decompose with all of the ACMI (and nonOverlapPatch patch) on one processor? (singleProcessorFaceSets option in decomposeParDict)

user1044

2015-04-21 01:39

  ~0004638

I'm not sure what mattijs mean as nonOverlapPatch is Domain_Interface_1_Side_1_blockage etc., nor how to use uniformFixedValue type, but I changed 0/U as follows,

0/U
    Domain_Interface_1_Side_1_blockage
    {
// type fixedValue;
// value uniform (0 0 0);
        type uniformFixedValue;
        uniformValue constant (0 0 0);
    }
Then, I added the following lines in the decomposeParDict,
singleProcessorFaceSets ((Domain_Interface_1_Side_1_zone -1));
singleProcessorFaceSets ((Domain_Interface_1_Side_2_zone -1));
singleProcessorFaceSets ((Domain_Interface_2_Side_1_zone -1));
singleProcessorFaceSets ((Domain_Interface_2_Side_2_zone -1));

Unfortunately, both is not effective, and I got the same results.
Actually, this issue is not supposed to be associated with nonOverlapPatch, because all of the patches are exactly the OverlapPatch, as you can see the cyclicAMI type can be applied to the same patches.
If I misunderstand how to use "type uniformFixedValue" and "singleProcessorFaceSets", please teach me them.

user4

2015-04-29 14:11

 

topoSetDict (1,676 bytes)   
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      topoSetDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

actions
(
    {
        name    couple1;
        type    faceSet;
        action  new;
        source  patchToFace;
        sourceInfo
        {
            name cyclicACMI_Domain_Interface_1_Side_1;
        }
    }
    {
        name    couple1;
        type    faceSet;
        action  add;
        source  patchToFace;
        sourceInfo
        {
            name cyclicACMI_Domain_Interface_1_Side_2;
        }
    }



    {
        name    couple2;
        type    faceSet;
        action  new;
        source  patchToFace;
        sourceInfo
        {
            name cyclicACMI_Domain_Interface_2_Side_1;
        }
    }
    {
        name    couple2;
        type    faceSet;
        action  add;
        source  patchToFace;
        sourceInfo
        {
            name cyclicACMI_Domain_Interface_2_Side_2;
        }
    }
);

// ************************************************************************* //
topoSetDict (1,676 bytes)   

user4

2015-04-29 14:11

 

decomposeParDict (1,360 bytes)   
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 8;

method          scotch;


//- Keep all of faceSet on a single processor. This puts all cells
//  connected with a point, edge or face on the same processor.
//  (just having face connected cells might not guarantee a balanced
//  decomposition)
// The processor can be -1 (the decompositionMethod chooses the processor
// for a good load balance) or explicitly provided (upsets balance).
//singleProcessorFaceSets ((couple1 -1) (couple2 -1));
singleProcessorFaceSets ((couple1 0) (couple2 0));


// ************************************************************************* //
decomposeParDict (1,360 bytes)   

user4

2015-04-29 14:13

  ~0004692

I've uploaded a topoSetDict and decomposeParDict which force all cells on the ACMI patches to be on processor 0. You can check with

decomposePar -cellDist

which writes a cellDist field giving the decomposition.

The problem still persists though - still investigating.

user4

2015-05-07 20:46

 

acmi.patch (3,878 bytes)   
diff --git a/src/meshTools/AMIInterpolation/patches/cyclicACMI/cyclicACMIPolyPatch/cyclicACMIPolyPatch.C b/src/meshTools/AMIInterpolation/patches/cyclicACMI/cyclicACMIPolyPatch/cyclicACMIPolyPatch.C
index 7a862ee..84028eb 100644
--- a/src/meshTools/AMIInterpolation/patches/cyclicACMI/cyclicACMIPolyPatch/cyclicACMIPolyPatch.C
+++ b/src/meshTools/AMIInterpolation/patches/cyclicACMI/cyclicACMIPolyPatch/cyclicACMIPolyPatch.C
@@ -44,7 +44,7 @@ const Foam::scalar Foam::cyclicACMIPolyPatch::tolerance_ = 1e-6;
 
 void Foam::cyclicACMIPolyPatch::initPatchFaceAreas() const
 {
-    if (!empty() && faceAreas0_.empty())
+    if (!empty() && (faceAreas0_.empty() || boundaryMesh().mesh().moving()))
     {
         faceAreas0_ = faceAreas();
     }
@@ -52,9 +52,13 @@ void Foam::cyclicACMIPolyPatch::initPatchFaceAreas() const
     const cyclicACMIPolyPatch& nbrACMI =
         refCast<const cyclicACMIPolyPatch>(this->neighbPatch());
 
-    if (!nbrACMI.empty() && nbrACMI.faceAreas0().empty())
+    if
+    (
+        !nbrACMI.empty()
+     && (nbrACMI.faceAreas0().empty() || boundaryMesh().mesh().moving())
+    )
     {
-        nbrACMI.initPatchFaceAreas();
+        nbrACMI.faceAreas0_ = nbrACMI.faceAreas();
     }
 }
 
@@ -136,11 +140,13 @@ void Foam::cyclicACMIPolyPatch::setNeighbourFaceAreas() const
 
 void Foam::cyclicACMIPolyPatch::initGeometry(PstreamBuffers& pBufs)
 {
+    // Note: cyclicAMIPolyPatch clears AMI so do first
+    cyclicAMIPolyPatch::initGeometry(pBufs);
+
     // Initialise the AMI so that base geometry (e.g. cell volumes) are
-    // correctly evaluated
+    // correctly evaluated before e.g. any of the processor patches gets
+    // hit (since uses cell volumes in its initGeometry)
     resetAMI();
-
-    cyclicAMIPolyPatch::initGeometry(pBufs);
 }
 
 
@@ -156,7 +162,13 @@ void Foam::cyclicACMIPolyPatch::initMovePoints
     const pointField& p
 )
 {
+    // Note: cyclicAMIPolyPatch clears AMI so do first
     cyclicAMIPolyPatch::initMovePoints(pBufs, p);
+
+    // Initialise the AMI so that base geometry (e.g. cell volumes) are
+    // correctly evaluated before e.g. any of the processor patches gets
+    // hit (since uses cell volumes in its initGeometry)
+    resetAMI();
 }
 
 
diff --git a/src/meshTools/AMIInterpolation/patches/cyclicAMI/cyclicAMIPolyPatch/cyclicAMIPolyPatch.C b/src/meshTools/AMIInterpolation/patches/cyclicAMI/cyclicAMIPolyPatch/cyclicAMIPolyPatch.C
index 9d33b27..6ec4c63 100644
--- a/src/meshTools/AMIInterpolation/patches/cyclicAMI/cyclicAMIPolyPatch/cyclicAMIPolyPatch.C
+++ b/src/meshTools/AMIInterpolation/patches/cyclicAMI/cyclicAMIPolyPatch/cyclicAMIPolyPatch.C
@@ -410,6 +410,9 @@ void Foam::cyclicAMIPolyPatch::resetAMI
 
 void Foam::cyclicAMIPolyPatch::initGeometry(PstreamBuffers& pBufs)
 {
+    // The AMI is no longer valid. Leave it up to demand-driven calculation
+    AMIPtr_.clear();
+
     polyPatch::initGeometry(pBufs);
 }
 
@@ -435,6 +438,9 @@ void Foam::cyclicAMIPolyPatch::initMovePoints
     const pointField& p
 )
 {
+    // The AMI is no longer valid. Leave it up to demand-driven calculation
+    AMIPtr_.clear();
+
     polyPatch::initMovePoints(pBufs, p);
 
     // See below. Clear out any local geometry
@@ -451,19 +457,15 @@ void Foam::cyclicAMIPolyPatch::movePoints
     polyPatch::movePoints(pBufs, p);
 
     calcTransforms();
-
-    // Note: resetAMI is called whilst in geometry update. So the slave
-    // side might not have reached 'movePoints'. Is explicitly handled by
-    // - clearing geometry of neighbour inside initMovePoints
-    // - not using localPoints() inside resetAMI
-    resetAMI();
 }
 
 
 void Foam::cyclicAMIPolyPatch::initUpdateMesh(PstreamBuffers& pBufs)
 {
-    polyPatch::initUpdateMesh(pBufs);
+    // The AMI is no longer valid. Leave it up to demand-driven calculation
     AMIPtr_.clear();
+
+    polyPatch::initUpdateMesh(pBufs);
 }
 
 
acmi.patch (3,878 bytes)   

user4

2015-05-07 20:54

  ~0004727

I've uploaded a patch for cyclicAMIPolyPatch.C and cyclicACMIPolyPatch.C. Could you try this on your case and also some AMI case (it should not affect cyclicAMI behaviour).

In the original code
- the unmasked, stored face areas were not being updated for moving mesh cases
- the face areas were corrected for the overlap after the cell centres were calculated (triggered by processorPolyPatch - so only happened in parallel). So the cell centres were incorrect and this caused the jump across the interface

With this there should be no need anymore for the special decomposition options but I have not tried this.

user1044

2015-05-08 02:09

 

user1044

2015-05-08 02:12

  ~0004728

Last edited: 2015-05-08 06:42

I calculated the original Case files in which the uniformFixedValue type is not applied to any boundary conditions and the special decomposition options is not used. Then, the pressure discontinuities have disappeared.
However, a small issue still seems to remain. Some discontinuities appear in the other positions shown as blue circles in figures of the attached pdf file, which are supposed to be decomposition boundaries. But, this issue might be negligible.

user1044

2015-05-08 02:24

 

user1044

2015-05-08 02:31

  ~0004729

Last edited: 2015-05-08 02:32

I'm wrong about some small discontinuies on the decompositon boundaries. I reconstructed the results to be already obtained and visualized them in the same way, and then the small discontinuities have disappeared as shown in the attached png file.
Thank you for your bug fix.

user1044

2015-05-08 07:28

  ~0004730

I also confirmed the uploaded patch does not affect cyclicAMI behaviour on the AMI case.

henry

2015-10-22 09:50

manager   ~0005457

Resolved in OpenFOAM-dev by commit c9e4d5c13592fe33592c3fe2354d4ab5289e3460

Issue History

Date Modified Username Field Change
2015-04-20 07:56 user1044 New Issue
2015-04-20 07:56 user1044 File Added: ACMI_decompose_rev20150420.pdf
2015-04-20 08:05 user1044 File Added: ACMI_pimpleDyMFoam_RotatingMotion.tar.0
2015-04-20 08:06 user1044 File Added: ACMI_pimpleDyMFoam_RotatingMotion.tar.1
2015-04-20 08:06 user1044 File Added: ACMI_pimpleDyMFoam_RotatingMotion.tar.2
2015-04-20 08:07 user1044 File Added: ACMI_pimpleDyMFoam_RotatingMotion.tar.ba_
2015-04-20 08:17 user1044 Tag Attached: ACMI
2015-04-20 08:17 user1044 Tag Attached: cyclicACMI
2015-04-20 08:17 user1044 Tag Attached: decomposePar
2015-04-20 08:17 user1044 Tag Attached: Solver
2015-04-20 09:44 user4 Note Added: 0004629
2015-04-21 01:39 user1044 Note Added: 0004638
2015-04-29 14:11 user4 File Added: topoSetDict
2015-04-29 14:11 user4 File Added: decomposeParDict
2015-04-29 14:13 user4 Note Added: 0004692
2015-05-07 20:46 user4 File Added: acmi.patch
2015-05-07 20:54 user4 Note Added: 0004727
2015-05-08 02:09 user1044 File Added: ACMI_decompose_rev20150508.pdf
2015-05-08 02:12 user1044 Note Added: 0004728
2015-05-08 02:24 user1044 File Added: ACMI_trn_decomp_0.0031416-2015-05-08_reconstructPar.png
2015-05-08 02:31 user1044 Note Added: 0004729
2015-05-08 02:32 user1044 Note Edited: 0004729
2015-05-08 06:42 user1044 Note Edited: 0004728
2015-05-08 07:28 user1044 Note Added: 0004730
2015-10-22 09:50 henry Note Added: 0005457
2015-10-22 09:50 henry Status new => resolved
2015-10-22 09:50 henry Resolution open => fixed
2015-10-22 09:50 henry Assigned To => henry