View Issue Details

IDProjectCategoryView StatusLast Update
0002315OpenFOAM[All Projects] Bugpublic2016-11-05 21:22
ReporterilyapopovAssigned Tohenry 
PrioritynormalSeveritycrashReproducibilityalways
Status resolvedResolutionfixed 
PlatformGNU/LinuxOSLinux UbuntuOS Version16.10
Product Version 
Fixed in Version 
Summary0002315: lagrangian Particle Tracking hangs in parallel
DescriptionI was working with lagrangian particle tracking. In parallel simulation, sometimes we encountered a hang up or a crash. The problem appears to depend on decomposition and initial position relative to processor boundaries. There is no problem detected when running on single CPU.

I was able to reproduce the problem on several systems, including Ubuntu 16.04 and 16.10, RHEL and on both OpenFOAM 3.0.x and 4.x.

Small reproduction case attached. It works in serial and hangs in parallel.

Steps To ReproduceUnpack the case and run ./run_parallel.sh

The simulation hangs on the first time step.
TagsNo tags attached.

Activities

ilyapopov

2016-11-01 16:05

reporter  

minimal.tgz (3,055 bytes)

wyldckat

2016-11-01 18:41

updater   ~0007084

@ilyapopov: This is excellent! I stumbled on this bug a couple of weeks ago, I've had the fix for it stuck in my time-schedule because I was missing a unit test case which which to double-check if the fix worked as intended. You case is what I was missing!

In attachment is the file "Cloud.C.4.x" meant for updating the file "src/lagrangian/basic/Cloud/Cloud.C".
After you replace the one you have in OpenFOAM 4.x, please run "Allwmake" in the main OpenFOAM-4.x folder so that all dependencies are updated accordingly.

Then please let us know if this also works for your other case(s) as well, not just this test unit.


I'll have the updated file for OpenFOAM-dev as well in a few minutes, after I've tested it without and with the patch, to see if it's actually working or not.

----
Background description of the actual bug and fix:

Attached is also the file "Cloud.C.patch" which provides the summary of changes needed, which is essentially a missing check on the flag "td.switchProcessor". Without this check, the particles that land exactly on a face that is shared between 2 or more processors, ends up getting caught in a constant switching between processors, hence the infinite loop we witnessed.

The weird part is that once the particle lands on another processor subdomain, it was meant to keep travelling on that other processor... problem is when the particle is not meant to continue travelling for the current global time step, which is why it gets stuck in an infinite loop.

Cloud.C_4.x (13,194 bytes)

wyldckat

2016-11-01 18:41

updater  

Cloud.C.patch (776 bytes)
diff --git a/src/lagrangian/basic/Cloud/Cloud.C b/src/lagrangian/basic/Cloud/Cloud.C
index 4bbfc9f..7bbbc91 100644
--- a/src/lagrangian/basic/Cloud/Cloud.C
+++ b/src/lagrangian/basic/Cloud/Cloud.C
@@ -261,7 +261,12 @@ void Foam::Cloud<ParticleType>::move(TrackData& td, const scalar trackTime)
             {
                 // If we are running in parallel and the particle is on a
                 // boundary face
-                if (Pstream::parRun() && p.face() >= pMesh().nInternalFaces())
+                if
+                (
+                    Pstream::parRun()
+                 && td.switchProcessor
+                 && p.face() >= pMesh().nInternalFaces()
+                )
                 {
                     label patchi = pbm.whichPatch(p.face());
 
Cloud.C.patch (776 bytes)

wyldckat

2016-11-01 21:37

updater  

Cloud.C_dev (13,097 bytes)

wyldckat

2016-11-01 21:39

updater   ~0007085

Attached is the file "Cloud.C_dev" for OpenFOAM-dev, synced to commit 2d3ddd4204934ffb39d4377ba3e74c45dee41558.

This updated file was tested successfully with the attached case and OpenFOAM-dev too.

ilyapopov

2016-11-02 08:32

reporter   ~0007086

Thank you very much for your quick reply!

I will test the fix on the real case today.

henry

2016-11-05 21:22

manager   ~0007109

Thanks Bruno

Resolved in OpenFOAM-4.x by commit 61195404f7cd198407ad068f45a14eba5e348491

Resolved in OpenFOAM-dev by commit 379c16d410f96b4671d7b3c368828044151ab515

Issue History

Date Modified Username Field Change
2016-11-01 16:05 ilyapopov New Issue
2016-11-01 16:05 ilyapopov File Added: minimal.tgz
2016-11-01 18:41 wyldckat File Added: Cloud.C_4.x
2016-11-01 18:41 wyldckat Note Added: 0007084
2016-11-01 18:41 wyldckat File Added: Cloud.C.patch
2016-11-01 21:37 wyldckat File Added: Cloud.C_dev
2016-11-01 21:39 wyldckat Note Added: 0007085
2016-11-02 08:32 ilyapopov Note Added: 0007086
2016-11-05 21:22 henry Assigned To => henry
2016-11-05 21:22 henry Status new => resolved
2016-11-05 21:22 henry Resolution open => fixed
2016-11-05 21:22 henry Fixed in Version => 4.x
2016-11-05 21:22 henry Note Added: 0007109