View Issue Details

IDProjectCategoryView StatusLast Update
0003214OpenFOAMBugpublic2020-02-05 20:20
Reporterctsang Assigned Tohenry  
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionno change required 
PlatformGNU/LinuxOSOtherOS Version(please specify)
Summary0003214: Memory corruption when initializing parcels
DescriptionRunning icoUncoupledKinematicParcelFoam with the tutorial case, hopperInitialState. When using 100 million parcels, parallel simulations crashed when initializing parcels. Attached shows the error message. Serial run does not have this problem. Are there any issues with MPI?

Also have tried MPPICFoam. Same problem occurred. If parcel count is 80 millions, both serial and parallel run fine.
Steps To ReproduceMethod to generate 200 million parcels in hopperInitialState:

In constant/kinematicParcelPositions, copy and paste all the coordinates 33333 times.
TagsNo tags attached.

Activities

ctsang

2019-04-04 23:12

reporter  

log (67,592 bytes)   
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : icoUncoupledKinematicParcelFoam -parallel
Date   : Apr 04 2019
Time   : 16:06:26
Host   : "backend-1-06.local"
PID    : 52032
I/O    : uncollated
Case   : /home/ua18762/hopperInitialState
nProcs : 2
Slaves : 1("backend-1-06.local.52033")
Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0


Reading g
Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting turbulence model type laminar
Selecting laminar stress model Stokes
Constructing kinematicCloud kinematicCloud
Constructing particle forces
    Selecting particle force sphereDrag
    Selecting particle force gravity
Constructing cloud functions
    none
Constructing particle injection models
Creating injector: model1
Selecting injection model manualInjection
    Constructing 3-D injection
    Choosing nParticle to be a fixed value, massTotal variable now does not determine anything.
*** Error in `icoUncoupledKinematicParcelFoam': malloc(): memory corruption: 0x00000000029fde40 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7dd7d)[0x7f6f0952dd7d]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7f6f0953010c]
/lib64/libc.so.6(vasprintf+0x22)[0x7f6f09523e62]
/lib64/libc.so.6(asprintf+0x87)[0x7f6f09501a87]
/share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(+0x3cbe0)[0x7f6f060f5be0]
/share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(ompi_mpi_errors_are_fatal_comm_handler+0xb4)[0x7f6f060f64b4]
/share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(ompi_errhandler_invoke+0x115)[0x7f6f060f5895]
/share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(PMPI_Send+0x163)[0x7f6f06133c83]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so(_ZN4Foam9UOPstream5writeENS_8UPstream10commsTypesEiPKclii+0x587)[0x7f6f092a5ad7]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam9UOPstreamD1Ev+0x5e)[0x7f6f0a6926ae]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam14fileOperations23uncollatedFileOperation4readERNS_11regIOobjectEbNS_8IOstream12streamFormatERKNS_4wordE+0x352)[0x7f6f0a5d85c2]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam11regIOobject12readHeaderOkENS_8IOstream12streamFormatERKNS_4wordE+0x5f)[0x7f6f0a6cc97f]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so(_ZN4Foam13GlobalIOFieldINS_6VectorIdEEEC2ERKNS_8IOobjectE+0x52)[0x7f6f0ff28c22]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so(_ZN4Foam15ManualInjectionINS_14KinematicCloudINS_5CloudINS_15CollidingParcelINS_15KinematicParcelINS_8particleEEEEEEEEEEC1ERKNS_10dictionaryERS9_RKNS_4wordE+0xf5)[0x7f6f0ffcf7c5]
/share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so(_ZN4Foam14InjectionModelINS_14KinematicCloudINS_5CloudINS_15CollidingParcelINS_15KinematicParcelINS_8particleEEEEEEEEEE31adddictionaryConstructorToTableINS_15ManualInjectionIS9_EEE3NewERKNS_10dictionaryERS9_RKNS_4wordE+0x32)[0x7f6f0ffd0022]
icoUncoupledKinematicParcelFoam[0x43cfb8]
icoUncoupledKinematicParcelFoam[0x43d18b]
icoUncoupledKinematicParcelFoam[0x4592f2]
icoUncoupledKinematicParcelFoam[0x45a21c]
icoUncoupledKinematicParcelFoam[0x41c7f2]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6f094d1c05]
icoUncoupledKinematicParcelFoam[0x41ecb3]
======= Memory map: ========
00400000-0047c000 r-xp 00000000 00:2c 361082572866                       /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/bin/icoUncoupledKinematicParcelFoam
0067b000-0067d000 r--p 0007b000 00:2c 361082572866                       /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/bin/icoUncoupledKinematicParcelFoam
0067d000-0067e000 rw-p 0007d000 00:2c 361082572866                       /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/bin/icoUncoupledKinematicParcelFoam
02591000-11fec3000 rw-p 00000000 00:00 0                                 [heap]
7f6e5c000000-7f6e5c021000 rw-p 00000000 00:00 0 
7f6e5c021000-7f6e60000000 ---p 00000000 00:00 0 
7f6e6361f000-7f6ef2085000 rw-p 00000000 00:00 0 
7f6ef2085000-7f6ef2091000 r-xp 00000000 08:02 658339                     /usr/lib64/libnss_files-2.17.so
7f6ef2091000-7f6ef2290000 ---p 0000c000 08:02 658339                     /usr/lib64/libnss_files-2.17.so
7f6ef2290000-7f6ef2291000 r--p 0000b000 08:02 658339                     /usr/lib64/libnss_files-2.17.so
7f6ef2291000-7f6ef2292000 rw-p 0000c000 08:02 658339                     /usr/lib64/libnss_files-2.17.so
7f6ef2292000-7f6ef35ab000 rw-p 00000000 00:00 0 
7f6ef35ab000-7f6ef39ac000 rw-s 00000000 08:11 71                         /state/partition1/21705.txc7ptorque01.local/ompi.backend-1-06.8762/pid.52027/1/1/vader_segment.backend-1-06.1
7f6ef39ac000-7f6ef39d8000 r-xp 00000000 00:2c 274909530271               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_rdma.so
7f6ef39d8000-7f6ef3bd7000 ---p 0002c000 00:2c 274909530271               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_rdma.so
7f6ef3bd7000-7f6ef3bd8000 r--p 0002b000 00:2c 274909530271               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_rdma.so
7f6ef3bd8000-7f6ef3bd9000 rw-p 0002c000 00:2c 274909530271               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_rdma.so
7f6ef3bd9000-7f6ef3bf7000 r-xp 00000000 00:2c 274909530269               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_pt2pt.so
7f6ef3bf7000-7f6ef3df6000 ---p 0001e000 00:2c 274909530269               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_pt2pt.so
7f6ef3df6000-7f6ef3df7000 r--p 0001d000 00:2c 274909530269               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_pt2pt.so
7f6ef3df7000-7f6ef3df8000 rw-p 0001e000 00:2c 274909530269               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_pt2pt.so
7f6ef3df8000-7f6ef3dfe000 r-xp 00000000 00:2c 274909530267               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_sm.so
7f6ef3dfe000-7f6ef3ffd000 ---p 00006000 00:2c 274909530267               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_sm.so
7f6ef3ffd000-7f6ef3ffe000 r--p 00005000 00:2c 274909530267               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_sm.so
7f6ef3ffe000-7f6ef4000000 rw-p 00006000 00:2c 274909530267               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_osc_sm.so
7f6ef4000000-7f6ef4021000 rw-p 00000000 00:00 0 
7f6ef4021000-7f6ef8000000 ---p 00000000 00:00 0 
7f6ef8112000-7f6ef8121000 r-xp 00000000 00:2c 274909530247               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_tuned.so
7f6ef8121000-7f6ef8320000 ---p 0000f000 00:2c 274909530247               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_tuned.so
7f6ef8320000-7f6ef8321000 r--p 0000e000 00:2c 274909530247               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_tuned.so
7f6ef8321000-7f6ef8322000 rw-p 0000f000 00:2c 274909530247               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_tuned.so
7f6ef8322000-7f6ef8325000 r-xp 00000000 00:2c 274909530245               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sync.so
7f6ef8325000-7f6ef8525000 ---p 00003000 00:2c 274909530245               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sync.so
7f6ef8525000-7f6ef8526000 r--p 00003000 00:2c 274909530245               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sync.so
7f6ef8526000-7f6ef8527000 rw-p 00004000 00:2c 274909530245               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sync.so
7f6ef8527000-7f6ef8529000 r-xp 00000000 00:2c 270961156106               /share/apps/openmpi/openmpi-3.0.0/lib/libmca_common_sm.so.40.0.0
7f6ef8529000-7f6ef8728000 ---p 00002000 00:2c 270961156106               /share/apps/openmpi/openmpi-3.0.0/lib/libmca_common_sm.so.40.0.0
7f6ef8728000-7f6ef8729000 r--p 00001000 00:2c 270961156106               /share/apps/openmpi/openmpi-3.0.0/lib/libmca_common_sm.so.40.0.0
7f6ef8729000-7f6ef872a000 rw-p 00002000 00:2c 270961156106               /share/apps/openmpi/openmpi-3.0.0/lib/libmca_common_sm.so.40.0.0
7f6ef872a000-7f6ef8730000 r-xp 00000000 00:2c 274909530243               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sm.so
7f6ef8730000-7f6ef892f000 ---p 00006000 00:2c 274909530243               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sm.so
7f6ef892f000-7f6ef8930000 r--p 00005000 00:2c 274909530243               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sm.so
7f6ef8930000-7f6ef8931000 rw-p 00006000 00:2c 274909530243               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_sm.so
7f6ef8931000-7f6ef8933000 r-xp 00000000 00:2c 274909530241               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_self.so
7f6ef8933000-7f6ef8b33000 ---p 00002000 00:2c 274909530241               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_self.so
7f6ef8b33000-7f6ef8b34000 r--p 00002000 00:2c 274909530241               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_self.so
7f6ef8b34000-7f6ef8b35000 rw-p 00003000 00:2c 274909530241               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_self.so
7f6ef8b35000-7f6ef8b50000 r-xp 00000000 00:2c 274909530239               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_libnbc.so
7f6ef8b50000-7f6ef8d4f000 ---p 0001b000 00:2c 274909530239               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_libnbc.so
7f6ef8d4f000-7f6ef8d50000 r--p 0001a000 00:2c 274909530239               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_libnbc.so
7f6ef8d50000-7f6ef8d51000 rw-p 0001b000 00:2c 274909530239               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_libnbc.so
7f6ef8d51000-7f6ef8d5e000 r-xp 00000000 00:2c 274909530235               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_basic.so
7f6ef8d5e000-7f6ef8f5d000 ---p 0000d000 00:2c 274909530235               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_basic.so
7f6ef8f5d000-7f6ef8f5e000 r--p 0000c000 00:2c 274909530235               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_basic.so
7f6ef8f5e000-7f6ef8f5f000 rw-p 0000d000 00:2c 274909530235               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_basic.so
7f6ef8f5f000-7f6ef8f82000 r-xp 00000000 00:2c 274909530277               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pml_ob1.so
7f6ef8f82000-7f6ef9182000 ---p 00023000 00:2c 274909530277               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pml_ob1.so
7f6ef9182000-7f6ef9183000 r--p 00023000 00:2c 274909530277               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pml_ob1.so
7f6ef9183000-7f6ef9184000 rw-p 00024000 00:2c 274909530277               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pml_ob1.so
7f6ef938e000-7f6ef9392000 r-xp 00000000 00:2c 274909530237               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_inter.so
7f6ef9392000-7f6ef9591000 ---p 00004000 00:2c 274909530237               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_inter.so
7f6ef9591000-7f6ef9592000 r--p 00003000 00:2c 274909530237               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_inter.so
7f6ef9592000-7f6ef9593000 rw-p 00004000 00:2c 274909530237               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_coll_inter.so
7f6ef9593000-7f6ef9994000 rw-s 00000000 08:11 1610614595                 /state/partition1/21705.txc7ptorque01.local/ompi.backend-1-06.8762/pid.52027/1/0/vader_segment.backend-1-06.0
7f6ef9994000-7f6ef999b000 r-xp 00000000 00:2c 274909528505               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_vader.so
7f6ef999b000-7f6ef9b9a000 ---p 00007000 00:2c 274909528505               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_vader.so
7f6ef9b9a000-7f6ef9b9b000 r--p 00006000 00:2c 274909528505               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_vader.so
7f6ef9b9b000-7f6ef9b9d000 rw-p 00007000 00:2c 274909528505               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_vader.so
7f6ef9b9d000-7f6ef9bab000 r-xp 00000000 00:2c 274909528503               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_tcp.so
7f6ef9bab000-7f6ef9dab000 ---p 0000e000 00:2c 274909528503               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_tcp.so
7f6ef9dab000-7f6ef9dac000 r--p 0000e000 00:2c 274909528503               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_tcp.so
7f6ef9dac000-7f6ef9dae000 rw-p 0000f000 00:2c 274909528503               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_tcp.so
7f6ef9dae000-7f6ef9daf000 r-xp 00000000 00:2c 274909528501               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_sm.so
7f6ef9daf000-7f6ef9fae000 ---p 00001000 00:2c 274909528501               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_sm.so
7f6ef9fae000-7f6ef9faf000 r--p 00000000 00:2c 274909528501               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_sm.so
7f6ef9faf000-7f6ef9fb0000 rw-p 00001000 00:2c 274909528501               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_sm.so
7f6ef9fb0000-7f6ef9fb3000 r-xp 00000000 00:2c 274909528499               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_self.so
7f6ef9fb3000-7f6efa1b2000 ---p 00003000 00:2c 274909528499               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_self.so
7f6efa1b2000-7f6efa1b3000 r--p 00002000 00:2c 274909528499               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_self.so
7f6efa1b3000-7f6efa1b4000 rw-p 00003000 00:2c 274909528499               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_btl_self.so
7f6efa1b4000-7f6efa1b8000 r-xp 00000000 00:2c 274909530233               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_bml_r2.so
7f6efa1b8000-7f6efa3b7000 ---p 00004000 00:2c 274909530233               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_bml_r2.so
7f6efa3b7000-7f6efa3b8000 r--p 00003000 00:2c 274909530233               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_bml_r2.so
7f6efa3b8000-7f6efa3b9000 rw-p 00004000 00:2c 274909530233               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_bml_r2.so
7f6efa3b9000-7f6efa3bc000 r-xp 00000000 00:2c 274909530113               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_mpool_hugepage.so
7f6efa3bc000-7f6efa5bb000 ---p 00003000 00:2c 274909530113               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_mpool_hugepage.so
7f6efa5bb000-7f6efa5bc000 r--p 00002000 00:2c 274909530113               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_mpool_hugepage.so
7f6efa5bc000-7f6efa5bd000 rw-p 00003000 00:2c 274909530113               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_mpool_hugepage.so
7f6efa5bd000-7f6efa5c0000 r-xp 00000000 00:2c 274909530125               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rcache_grdma.so
7f6efa5c0000-7f6efa7c0000 ---p 00003000 00:2c 274909530125               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rcache_grdma.so
7f6efa7c0000-7f6efa7c1000 r--p 00003000 00:2c 274909530125               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rcache_grdma.so
7f6efa7c1000-7f6efa7c2000 rw-p 00004000 00:2c 274909530125               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rcache_grdma.so
7f6efa7c2000-7f6efa7c4000 r-xp 00000000 00:2c 274909528497               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_bucket.so
7f6efa7c4000-7f6efa9c3000 ---p 00002000 00:2c 274909528497               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_bucket.so
7f6efa9c3000-7f6efa9c4000 r--p 00001000 00:2c 274909528497               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_bucket.so
7f6efa9c4000-7f6efa9c5000 rw-p 00002000 00:2c 274909528497               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_bucket.so
7f6efa9c5000-7f6efa9c8000 r-xp 00000000 00:2c 274909530199               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rml_oob.so
7f6efa9c8000-7f6efabc7000 ---p 00003000 00:2c 274909530199               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rml_oob.so
7f6efabc7000-7f6efabc8000 r--p 00002000 00:2c 274909530199               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rml_oob.so
7f6efabc8000-7f6efabc9000 rw-p 00003000 00:2c 274909530199               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_rml_oob.so
7f6efabc9000-7f6efabdb000 r-xp 00000000 00:2c 274909530175               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_oob_tcp.so
7f6efabdb000-7f6efaddb000 ---p 00012000 00:2c 274909530175               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_oob_tcp.so
7f6efaddb000-7f6efaddc000 r--p 00012000 00:2c 274909530175               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_oob_tcp.so
7f6efaddc000-7f6efaddd000 rw-p 00013000 00:2c 274909530175               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_oob_tcp.so
7f6efaddd000-7f6efade0000 r-xp 00000000 00:2c 274909530207               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_radix.so
7f6efade0000-7f6efafdf000 ---p 00003000 00:2c 274909530207               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_radix.so
7f6efafdf000-7f6efafe0000 r--p 00002000 00:2c 274909530207               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_radix.so
7f6efafe0000-7f6efafe1000 rw-p 00003000 00:2c 274909530207               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_radix.so
7f6efafe1000-7f6efafe3000 r-xp 00000000 00:2c 274909530205               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_direct.so
7f6efafe3000-7f6efb1e2000 ---p 00002000 00:2c 274909530205               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_direct.so
7f6efb1e2000-7f6efb1e3000 r--p 00001000 00:2c 274909530205               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_direct.so
7f6efb1e3000-7f6efb1e4000 rw-p 00002000 00:2c 274909530205               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_direct.so
7f6efb1e4000-7f6efb1e6000 r-xp 00000000 00:2c 274909530203               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_debruijn.so
7f6efb1e6000-7f6efb3e5000 ---p 00002000 00:2c 274909530203               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_debruijn.so
7f6efb3e5000-7f6efb3e6000 r--p 00001000 00:2c 274909530203               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_debruijn.so
7f6efb3e6000-7f6efb3e7000 rw-p 00002000 00:2c 274909530203               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_debruijn.so
7f6efb3e7000-7f6efb3e9000 r-xp 00000000 00:2c 274909530201               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_binomial.so
7f6efb3e9000-7f6efb5e9000 ---p 00002000 00:2c 274909530201               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_binomial.so
7f6efb5e9000-7f6efb5ea000 r--p 00002000 00:2c 274909530201               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_binomial.so
7f6efb5ea000-7f6efb5eb000 rw-p 00003000 00:2c 274909530201               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_routed_binomial.so
7f6efb7eb000-7f6efb7ed000 r-xp 00000000 00:2c 274909528495               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_basic.so
7f6efb7ed000-7f6efb9ed000 ---p 00002000 00:2c 274909528495               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_basic.so
7f6efb9ed000-7f6efb9ee000 r--p 00002000 00:2c 274909528495               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_basic.so
7f6efb9ee000-7f6efb9ef000 rw-p 00003000 00:2c 274909528495               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_allocator_basic.so
7f6efb9ef000-7f6efb9f5000 r-xp 00000000 00:2c 274909530133               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_dfs_app.so
7f6efb9f5000-7f6efbbf5000 ---p 00006000 00:2c 274909530133               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_dfs_app.so
7f6efbbf5000-7f6efbbf6000 r--p 00006000 00:2c 274909530133               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_dfs_app.so
7f6efbbf6000-7f6efbbf7000 rw-p 00007000 00:2c 274909530133               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_dfs_app.so
7f6efbbf7000-7f6efbbfc000 r-xp 00000000 00:2c 274909530163               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_grpcomm_direct.so
7f6efbbfc000-7f6efbdfb000 ---p 00005000 00:2c 274909530163               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_grpcomm_direct.so
7f6efbdfb000-7f6efbdfc000 r--p 00004000 00:2c 274909530163               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_grpcomm_direct.so
7f6efbdfc000-7f6efbdfd000 rw-p 00005000 00:2c 274909530163               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_grpcomm_direct.so
7f6efbdfd000-7f6efbdff000 r-xp 00000000 00:2c 274909530139               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_errmgr_default_app.so
7f6efbdff000-7f6efbffe000 ---p 00002000 00:2c 274909530139               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_errmgr_default_app.so
7f6efbffe000-7f6efbfff000 r--p 00001000 00:2c 274909530139               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_errmgr_default_app.so
7f6efbfff000-7f6efc000000 rw-p 00002000 00:2c 274909530139               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_errmgr_default_app.so
7f6efc000000-7f6efc021000 rw-p 00000000 00:00 0 
7f6efc021000-7f6f00000000 ---p 00000000 00:00 0 
7f6f001c0000-7f6f001c2000 r-xp 00000000 00:2c 274909530219               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_state_app.so
7f6f001c2000-7f6f003c1000 ---p 00002000 00:2c 274909530219               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_state_app.so
7f6f003c1000-7f6f003c2000 r--p 00001000 00:2c 274909530219               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_state_app.so
7f6f003c2000-7f6f003c3000 rw-p 00002000 00:2c 274909530219               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_state_app.so
7f6f003c3000-7f6f007c3000 r--s 00000000 08:11 536872581                  /state/partition1/21705.txc7ptorque01.local/ompi.backend-1-06.8762/pid.52027/pmix_dstor_52027/smdataseg-4088659969-0
7f6f007c3000-7f6f00bc3000 r--s 00000000 08:11 536872580                  /state/partition1/21705.txc7ptorque01.local/ompi.backend-1-06.8762/pid.52027/pmix_dstor_52027/smseg-4088659969-0
7f6f00bc3000-7f6f00bc4000 ---p 00000000 00:00 0 
7f6f00bc4000-7f6f017c4000 rw-p 00000000 00:00 0                          [stack:52035]
7f6f017c4000-7f6f017c5000 r-xp 00000000 00:2c 309570969620               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_none.so
7f6f017c5000-7f6f019c4000 ---p 00001000 00:2c 309570969620               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_none.so
7f6f019c4000-7f6f019c5000 r--p 00000000 00:2c 309570969620               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_none.so
7f6f019c5000-7f6f019c6000 rw-p 00001000 00:2c 309570969620               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_none.so
7f6f019c6000-7f6f019c8000 r-xp 00000000 00:2c 309570969618               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_native.so
7f6f019c8000-7f6f01bc7000 ---p 00002000 00:2c 309570969618               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_native.so
7f6f01bc7000-7f6f01bc8000 r--p 00001000 00:2c 309570969618               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_native.so
7f6f01bc8000-7f6f01bc9000 rw-p 00002000 00:2c 309570969618               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_psec_native.so
7f6f01bc9000-7f6f01bd2000 r-xp 00000000 00:2c 309570969626               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_tcp.so
7f6f01bd2000-7f6f01dd1000 ---p 00009000 00:2c 309570969626               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_tcp.so
7f6f01dd1000-7f6f01dd2000 r--p 00008000 00:2c 309570969626               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_tcp.so
7f6f01dd2000-7f6f01dd3000 rw-p 00009000 00:2c 309570969626               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_tcp.so
7f6f01dd3000-7f6f01e8f000 r-xp 00000000 00:2c 274909530121               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pmix_pmix2x.so
7f6f01e8f000-7f6f0208e000 ---p 000bc000 00:2c 274909530121               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pmix_pmix2x.so
7f6f0208e000-7f6f02090000 r--p 000bb000 00:2c 274909530121               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pmix_pmix2x.so
7f6f02090000-7f6f02094000 rw-p 000bd000 00:2c 274909530121               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_pmix_pmix2x.so
7f6f02094000-7f6f02097000 rw-p 00000000 00:00 0 
7f6f02097000-7f6f0209c000 r-xp 00000000 00:2c 309570969628               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_usock.so
7f6f0209c000-7f6f0229b000 ---p 00005000 00:2c 309570969628               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_usock.so
7f6f0229b000-7f6f0229c000 r--p 00004000 00:2c 309570969628               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_usock.so
7f6f0229c000-7f6f0229d000 rw-p 00005000 00:2c 309570969628               /share/apps/openmpi/openmpi-3.0.0/lib/pmix/mca_ptl_usock.so
7f6f0229d000-7f6f0229e000 ---p 00000000 00:00 0 
7f6f0229e000-7f6f02e9e000 rw-p 00000000 00:00 0                          [stack:52034]
7f6f02e9e000-7f6f02ea2000 r-xp 00000000 00:2c 274909530153               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_ess_pmi.so
7f6f02ea2000-7f6f030a1000 ---p 00004000 00:2c 274909530153               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_ess_pmi.so
7f6f030a1000-7f6f030a2000 r--p 00003000 00:2c 274909530153               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_ess_pmi.so
7f6f030a2000-7f6f030a3000 rw-p 00004000 00:2c 274909530153               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_ess_pmi.so
7f6f030a3000-7f6f030a5000 r-xp 00000000 00:2c 274909530217               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_slurm.so
7f6f030a5000-7f6f032a4000 ---p 00002000 00:2c 274909530217               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_slurm.so
7f6f032a4000-7f6f032a5000 r--p 00001000 00:2c 274909530217               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_slurm.so
7f6f032a5000-7f6f032a6000 rw-p 00002000 00:2c 274909530217               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_slurm.so
7f6f032a6000-7f6f032a7000 r-xp 00000000 00:2c 274909530215               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_orte.so
7f6f032a7000-7f6f034a7000 ---p 00001000 00:2c 274909530215               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_orte.so
7f6f034a7000-7f6f034a8000 r--p 00001000 00:2c 274909530215               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_orte.so
7f6f034a8000-7f6f034a9000 rw-p 00002000 00:2c 274909530215               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_orte.so
7f6f034a9000-7f6f034b1000 r-xp 00000000 00:2c 274909530213               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_ompi.so
7f6f034b1000-7f6f036b0000 ---p 00008000 00:2c 274909530213               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_ompi.so
7f6f036b0000-7f6f036b1000 r--p 00007000 00:2c 274909530213               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_ompi.so
7f6f036b1000-7f6f036b3000 rw-p 00008000 00:2c 274909530213               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_ompi.so
7f6f036b3000-7f6f036b4000 r-xp 00000000 00:2c 274909530211               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_flux.so
7f6f036b4000-7f6f038b4000 ---p 00001000 00:2c 274909530211               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_flux.so
7f6f038b4000-7f6f038b5000 r--p 00001000 00:2c 274909530211               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_flux.so
7f6f038b5000-7f6f038b6000 rw-p 00002000 00:2c 274909530211               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_schizo_flux.so
7f6f038b6000-7f6f038b9000 r-xp 00000000 00:2c 274909530127               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_shmem_mmap.so
7f6f038b9000-7f6f03ab8000 ---p 00003000 00:2c 274909530127               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_shmem_mmap.so
7f6f03ab8000-7f6f03ab9000 r--p 00002000 00:2c 274909530127               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_shmem_mmap.so
7f6f03ab9000-7f6f03aba000 rw-p 00003000 00:2c 274909530127               /share/apps/openmpi/openmpi-3.0.0/lib/openmpi/mca_shmem_mmap.so
7f6f03aba000-7f6f03ac9000 r-xp 00000000 08:02 658961                     /usr/lib64/libbz2.so.1.0.6
7f6f03ac9000-7f6f03cc8000 ---p 0000f000 08:02 658961                     /usr/lib64/libbz2.so.1.0.6
7f6f03cc8000-7f6f03cc9000 r--p 0000e000 08:02 658961                     /usr/lib64/libbz2.so.1.0.6
7f6f03cc9000-7f6f03cca000 rw-p 0000f000 08:02 658961                     /usr/lib64/libbz2.so.1.0.6
7f6f03cca000-7f6f03cef000 r-xp 00000000 08:02 658794                     /usr/lib64/liblzma.so.5.2.2
7f6f03cef000-7f6f03eee000 ---p 00025000 08:02 658794                     /usr/lib64/liblzma.so.5.2.2
7f6f03eee000-7f6f03eef000 r--p 00024000 08:02 658794                     /usr/lib64/liblzma.so.5.2.2
7f6f03eef000-7f6f03ef0000 rw-p 00025000 08:02 658794                     /usr/lib64/liblzma.so.5.2.2
7f6f03ef0000-7f6f03f07000 r-xp 00000000 08:02 659028                     /usr/lib64/libelf-0.168.so
7f6f03f07000-7f6f04106000 ---p 00017000 08:02 659028                     /usr/lib64/libelf-0.168.so
7f6f04106000-7f6f04107000 r--p 00016000 08:02 659028                     /usr/lib64/libelf-0.168.so
7f6f04107000-7f6f04108000 rw-p 00017000 08:02 659028                     /usr/lib64/libelf-0.168.so
7f6f04108000-7f6f0410c000 r-xp 00000000 08:02 659055                     /usr/lib64/libattr.so.1.1.0
7f6f0410c000-7f6f0430b000 ---p 00004000 08:02 659055                     /usr/lib64/libattr.so.1.1.0
7f6f0430b000-7f6f0430c000 r--p 00003000 08:02 659055                     /usr/lib64/libattr.so.1.1.0
7f6f0430c000-7f6f0430d000 rw-p 00004000 08:02 659055                     /usr/lib64/libattr.so.1.1.0
7f6f0430d000-7f6f04351000 r-xp 00000000 08:02 660719                     /usr/lib64/libdw-0.168.so
7f6f04351000-7f6f04551000 ---p 00044000 08:02 660719                     /usr/lib64/libdw-0.168.so
7f6f04551000-7f6f04553000 r--p 00044000 08:02 660719                     /usr/lib64/libdw-0.168.so
7f6f04553000-7f6f04554000 rw-p 00046000 08:02 660719                     /usr/lib64/libdw-0.168.so
7f6f04554000-7f6f04558000 r-xp 00000000 08:02 659058                     /usr/lib64/libcap.so.2.22
7f6f04558000-7f6f04757000 ---p 00004000 08:02 659058                     /usr/lib64/libcap.so.2.22
7f6f04757000-7f6f04758000 r--p 00003000 08:02 659058                     /usr/lib64/libcap.so.2.22
7f6f04758000-7f6f04759000 rw-p 00004000 08:02 659058                     /usr/lib64/libcap.so.2.22
7f6f04759000-7f6f04776000 r-xp 00000000 00:2c 618851831893               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libextrudeModel.so
7f6f04776000-7f6f04976000 ---p 0001d000 00:2c 618851831893               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libextrudeModel.so
7f6f04976000-7f6f04977000 r--p 0001d000 00:2c 618851831893               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libextrudeModel.so
7f6f04977000-7f6f04978000 rw-p 0001e000 00:2c 618851831893               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libextrudeModel.so
7f6f04978000-7f6f0497a000 r-xp 00000000 08:02 658355                     /usr/lib64/libutil-2.17.so
7f6f0497a000-7f6f04b79000 ---p 00002000 08:02 658355                     /usr/lib64/libutil-2.17.so
7f6f04b79000-7f6f04b7a000 r--p 00001000 08:02 658355                     /usr/lib64/libutil-2.17.so
7f6f04b7a000-7f6f04b7b000 rw-p 00002000 08:02 658355                     /usr/lib64/libutil-2.17.so
7f6f04b7b000-7f6f04b82000 r-xp 00000000 08:02 658351                     /usr/lib64/librt-2.17.so
7f6f04b82000-7f6f04d81000 ---p 00007000 08:02 658351                     /usr/lib64/librt-2.17.so
7f6f04d81000-7f6f04d82000 r--p 00006000 08:02 658351                     /usr/lib64/librt-2.17.so
7f6f04d82000-7f6f04d83000 rw-p 00007000 08:02 658351                     /usr/lib64/librt-2.17.so
7f6f04d83000-7f6f04e73000 r-xp 00000000 00:2c 270961156102               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40.0.0
7f6f04e73000-7f6f05073000 ---p 000f0000 00:2c 270961156102               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40.0.0
7f6f05073000-7f6f05076000 r--p 000f0000 00:2c 270961156102               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40.0.0
7f6f05076000-7f6f0507c000 rw-p 000f3000 00:2c 270961156102               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-pal.so.40.0.0
7f6f0507c000-7f6f05083000 rw-p 00000000 00:00 0 
7f6f05083000-7f6f0512d000 r-xp 00000000 00:2c 270961156110               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-rte.so.40.0.0
7f6f0512d000-7f6f0532c000 ---p 000aa000 00:2c 270961156110               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-rte.so.40.0.0
7f6f0532c000-7f6f0532e000 r--p 000a9000 00:2c 270961156110               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-rte.so.40.0.0
7f6f0532e000-7f6f05332000 rw-p 000ab000 00:2c 270961156110               /share/apps/openmpi/openmpi-3.0.0/lib/libopen-rte.so.40.0.0
7f6f05332000-7f6f05334000 rw-p 00000000 00:00 0 
7f6f05334000-7f6f053ac000 r-xp 00000000 00:2c 618851831892               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libconversion.so
7f6f053ac000-7f6f055ac000 ---p 00078000 00:2c 618851831892               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libconversion.so
7f6f055ac000-7f6f055ae000 r--p 00078000 00:2c 618851831892               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libconversion.so
7f6f055ae000-7f6f055af000 rw-p 0007a000 00:2c 618851831892               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libconversion.so
7f6f055af000-7f6f055b0000 rw-p 00000000 00:00 0 
7f6f055b0000-7f6f05a5a000 r-xp 00000000 00:2c 618851831894               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicMesh.so
7f6f05a5a000-7f6f05c5a000 ---p 004aa000 00:2c 618851831894               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicMesh.so
7f6f05c5a000-7f6f05c66000 r--p 004aa000 00:2c 618851831894               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicMesh.so
7f6f05c66000-7f6f05c6d000 rw-p 004b6000 00:2c 618851831894               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicMesh.so
7f6f05c6d000-7f6f05c6f000 rw-p 00000000 00:00 0 
7f6f05c6f000-7f6f05c9b000 r-xp 00000000 00:2c 618851831912               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidSpecie.so
7f6f05c9b000-7f6f05e9a000 ---p 0002c000 00:2c 618851831912               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidSpecie.so
7f6f05e9a000-7f6f05e9c000 r--p 0002b000 00:2c 618851831912               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidSpecie.so
7f6f05e9c000-7f6f05e9d000 rw-p 0002d000 00:2c 618851831912               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidSpecie.so
7f6f05e9d000-7f6f05eb4000 r-xp 00000000 08:02 658347                     /usr/lib64/libpthread-2.17.so
7f6f05eb4000-7f6f060b3000 ---p 00017000 08:02 658347                     /usr/lib64/libpthread-2.17.so
7f6f060b3000-7f6f060b4000 r--p 00016000 08:02 658347                     /usr/lib64/libpthread-2.17.so
7f6f060b4000-7f6f060b5000 rw-p 00017000 08:02 658347                     /usr/lib64/libpthread-2.17.so
7f6f060b5000-7f6f060b9000 rw-p 00000000 00:00 0 
7f6f060b9000-7f6f06190000 r-xp 00000000 00:2c 270961156114               /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40.0.0
7f6f06190000-7f6f06390000 ---p 000d7000 00:2c 270961156114               /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40.0.0
7f6f06390000-7f6f06391000 r--p 000d7000 00:2c 270961156114               /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40.0.0
7f6f06391000-7f6f063a2000 rw-p 000d8000 00:2c 270961156114               /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40.0.0
7f6f063a2000-7f6f063b3000 rw-p 00000000 00:00 0 
7f6f063b3000-7f6f063c8000 r-xp 00000000 08:02 658780                     /usr/lib64/libz.so.1.2.7
7f6f063c8000-7f6f065c7000 ---p 00015000 08:02 658780                     /usr/lib64/libz.so.1.2.7
7f6f065c7000-7f6f065c8000 r--p 00014000 08:02 658780                     /usr/lib64/libz.so.1.2.7
7f6f065c8000-7f6f065c9000 rw-p 00015000 08:02 658780                     /usr/lib64/libz.so.1.2.7
7f6f065c9000-7f6f0664f000 r-xp 00000000 00:2c 618851831878               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfileFormats.so
7f6f0664f000-7f6f0684f000 ---p 00086000 00:2c 618851831878               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfileFormats.so
7f6f0684f000-7f6f06853000 r--p 00086000 00:2c 618851831878               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfileFormats.so
7f6f06853000-7f6f06854000 rw-p 0008a000 00:2c 618851831878               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfileFormats.so
7f6f06854000-7f6f06855000 rw-p 00000000 00:00 0 
7f6f06855000-7f6f0695b000 r-xp 00000000 00:2c 618851831879               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfMesh.so
7f6f0695b000-7f6f06b5b000 ---p 00106000 00:2c 618851831879               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfMesh.so
7f6f06b5b000-7f6f06b5f000 r--p 00106000 00:2c 618851831879               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfMesh.so
7f6f06b5f000-7f6f06b61000 rw-p 0010a000 00:2c 618851831879               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfMesh.so
7f6f06b61000-7f6f06bf9000 r-xp 00000000 00:2c 618851831880               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtriSurface.so
7f6f06bf9000-7f6f06df9000 ---p 00098000 00:2c 618851831880               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtriSurface.so
7f6f06df9000-7f6f06dfb000 r--p 00098000 00:2c 618851831880               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtriSurface.so
7f6f06dfb000-7f6f06dfd000 rw-p 0009a000 00:2c 618851831880               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtriSurface.so
7f6f06dfd000-7f6f06e13000 r-xp 00000000 00:2c 618851831905               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtwoPhaseMixture.so
7f6f06e13000-7f6f07013000 ---p 00016000 00:2c 618851831905               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtwoPhaseMixture.so
7f6f07013000-7f6f07014000 r--p 00016000 00:2c 618851831905               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtwoPhaseMixture.so
7f6f07014000-7f6f07015000 rw-p 00017000 00:2c 618851831905               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libtwoPhaseMixture.so
7f6f07015000-7f6f070da000 r-xp 00000000 00:2c 618851831920               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidThermo.so
7f6f070da000-7f6f072d9000 ---p 000c5000 00:2c 618851831920               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidThermo.so
7f6f072d9000-7f6f072de000 r--p 000c4000 00:2c 618851831920               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidThermo.so
7f6f072de000-7f6f072e0000 rw-p 000c9000 00:2c 618851831920               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsolidThermo.so
7f6f072e0000-7f6f07535000 r-xp 00000000 00:2c 618851831926               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTurbulenceModels.so
7f6f07535000-7f6f07735000 ---p 00255000 00:2c 618851831926               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTurbulenceModels.so
7f6f07735000-7f6f07741000 r--p 00255000 00:2c 618851831926               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTurbulenceModels.so
7f6f07741000-7f6f07745000 rw-p 00261000 00:2c 618851831926               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTurbulenceModels.so
7f6f07745000-7f6f07746000 rw-p 00000000 00:00 0 
7f6f07746000-7f6f07b10000 r-xp 00000000 00:2c 618851831895               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsampling.so
7f6f07b10000-7f6f07d10000 ---p 003ca000 00:2c 618851831895               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsampling.so
7f6f07d10000-7f6f07d1b000 r--p 003ca000 00:2c 618851831895               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsampling.so
7f6f07d1b000-7f6f07d21000 rw-p 003d5000 00:2c 618851831895               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsampling.so
7f6f07d21000-7f6f07d22000 rw-p 00000000 00:00 0 
7f6f07d22000-7f6f07d7e000 r-xp 00000000 00:2c 618851831896               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicFvMesh.so
7f6f07d7e000-7f6f07f7d000 ---p 0005c000 00:2c 618851831896               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicFvMesh.so
7f6f07f7d000-7f6f07f81000 r--p 0005b000 00:2c 618851831896               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicFvMesh.so
7f6f07f81000-7f6f07f82000 rw-p 0005f000 00:2c 618851831896               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdynamicFvMesh.so
7f6f07f82000-7f6f07f83000 rw-p 00000000 00:00 0 
7f6f07f83000-7f6f07f89000 r-xp 00000000 00:2c 618851831919               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libSLGThermo.so
7f6f07f89000-7f6f08188000 ---p 00006000 00:2c 618851831919               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libSLGThermo.so
7f6f08188000-7f6f08189000 r--p 00005000 00:2c 618851831919               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libSLGThermo.so
7f6f08189000-7f6f0818a000 rw-p 00006000 00:2c 618851831919               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libSLGThermo.so
7f6f0818a000-7f6f08b0e000 r-xp 00000000 00:2c 618851831915               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libreactionThermophysicalModels.so
7f6f08b0e000-7f6f08d0d000 ---p 00984000 00:2c 618851831915               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libreactionThermophysicalModels.so
7f6f08d0d000-7f6f08d51000 r--p 00983000 00:2c 618851831915               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libreactionThermophysicalModels.so
7f6f08d51000-7f6f08d57000 rw-p 009c7000 00:2c 618851831915               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libreactionThermophysicalModels.so
7f6f08d57000-7f6f08d58000 rw-p 00000000 00:00 0 
7f6f08d58000-7f6f08e7a000 r-xp 00000000 00:2c 618851831913               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libthermophysicalProperties.so
7f6f08e7a000-7f6f0907a000 ---p 00122000 00:2c 618851831913               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libthermophysicalProperties.so
7f6f0907a000-7f6f0907f000 r--p 00122000 00:2c 618851831913               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libthermophysicalProperties.so
7f6f0907f000-7f6f09080000 rw-p 00127000 00:2c 618851831913               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libthermophysicalProperties.so
7f6f09080000-7f6f09081000 rw-p 00000000 00:00 0 
7f6f09081000-7f6f090a0000 r-xp 00000000 00:2c 618851831890               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdistributionModels.so
7f6f090a0000-7f6f0929f000 ---p 0001f000 00:2c 618851831890               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdistributionModels.so
7f6f0929f000-7f6f092a0000 r--p 0001e000 00:2c 618851831890               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdistributionModels.so
7f6f092a0000-7f6f092a1000 rw-p 0001f000 00:2c 618851831890               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libdistributionModels.so
7f6f092a1000-7f6f092af000 r-xp 00000000 00:2c 653241081859               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so
7f6f092af000-7f6f094ae000 ---p 0000e000 00:2c 653241081859               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so
7f6f094ae000-7f6f094af000 r--p 0000d000 00:2c 653241081859               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so
7f6f094af000-7f6f094b0000 rw-p 0000e000 00:2c 653241081859               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so
7f6f094b0000-7f6f09668000 r-xp 00000000 08:02 658321                     /usr/lib64/libc-2.17.so
7f6f09668000-7f6f09868000 ---p 001b8000 08:02 658321                     /usr/lib64/libc-2.17.so
7f6f09868000-7f6f0986c000 r--p 001b8000 08:02 658321                     /usr/lib64/libc-2.17.so
7f6f0986c000-7f6f0986e000 rw-p 001bc000 08:02 658321                     /usr/lib64/libc-2.17.so
7f6f0986e000-7f6f09873000 rw-p 00000000 00:00 0 
7f6f09873000-7f6f09888000 r-xp 00000000 08:02 655385                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6f09888000-7f6f09a87000 ---p 00015000 08:02 655385                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6f09a87000-7f6f09a88000 r--p 00014000 08:02 655385                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6f09a88000-7f6f09a89000 rw-p 00015000 08:02 655385                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f6f09a89000-7f6f09b8a000 r-xp 00000000 08:02 658329                     /usr/lib64/libm-2.17.so
7f6f09b8a000-7f6f09d89000 ---p 00101000 08:02 658329                     /usr/lib64/libm-2.17.so
7f6f09d89000-7f6f09d8a000 r--p 00100000 08:02 658329                     /usr/lib64/libm-2.17.so
7f6f09d8a000-7f6f09d8b000 rw-p 00101000 08:02 658329                     /usr/lib64/libm-2.17.so
7f6f09d8b000-7f6f09e74000 r-xp 00000000 08:02 658627                     /usr/lib64/libstdc++.so.6.0.19
7f6f09e74000-7f6f0a074000 ---p 000e9000 08:02 658627                     /usr/lib64/libstdc++.so.6.0.19
7f6f0a074000-7f6f0a07c000 r--p 000e9000 08:02 658627                     /usr/lib64/libstdc++.so.6.0.19
7f6f0a07c000-7f6f0a07e000 rw-p 000f1000 08:02 658627                     /usr/lib64/libstdc++.so.6.0.19
7f6f0a07e000-7f6f0a093000 rw-p 00000000 00:00 0 
7f6f0a093000-7f6f0a095000 r-xp 00000000 08:02 658327                     /usr/lib64/libdl-2.17.so
7f6f0a095000-7f6f0a295000 ---p 00002000 08:02 658327                     /usr/lib64/libdl-2.17.so
7f6f0a295000-7f6f0a296000 r--p 00002000 08:02 658327                     /usr/lib64/libdl-2.17.so
7f6f0a296000-7f6f0a297000 rw-p 00003000 08:02 658327                     /usr/lib64/libdl-2.17.so
7f6f0a297000-7f6f0ab93000 r-xp 00000000 00:2c 618851831877               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
7f6f0ab93000-7f6f0ad92000 ---p 008fc000 00:2c 618851831877               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
7f6f0ad92000-7f6f0adc5000 r--p 008fb000 00:2c 618851831877               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
7f6f0adc5000-7f6f0add0000 rw-p 0092e000 00:2c 618851831877               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
7f6f0add0000-7f6f0adde000 rw-p 00000000 00:00 0 
7f6f0adde000-7f6f0aff2000 r-xp 00000000 00:2c 618851831930               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfaceFilmModels.so
7f6f0aff2000-7f6f0b1f1000 ---p 00214000 00:2c 618851831930               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfaceFilmModels.so
7f6f0b1f1000-7f6f0b1f9000 r--p 00213000 00:2c 618851831930               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfaceFilmModels.so
7f6f0b1f9000-7f6f0b1fd000 rw-p 0021b000 00:2c 618851831930               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libsurfaceFilmModels.so
7f6f0b1fd000-7f6f0b1fe000 rw-p 00000000 00:00 0 
7f6f0b1fe000-7f6f0b271000 r-xp 00000000 00:2c 618851831928               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libregionModels.so
7f6f0b271000-7f6f0b471000 ---p 00073000 00:2c 618851831928               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libregionModels.so
7f6f0b471000-7f6f0b473000 r--p 00073000 00:2c 618851831928               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libregionModels.so
7f6f0b473000-7f6f0b475000 rw-p 00075000 00:2c 618851831928               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libregionModels.so
7f6f0b475000-7f6f0b8f3000 r-xp 00000000 00:2c 618851831881               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libmeshTools.so
7f6f0b8f3000-7f6f0baf2000 ---p 0047e000 00:2c 618851831881               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libmeshTools.so
7f6f0baf2000-7f6f0bb06000 r--p 0047d000 00:2c 618851831881               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libmeshTools.so
7f6f0bb06000-7f6f0bb0c000 rw-p 00491000 00:2c 618851831881               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libmeshTools.so
7f6f0bb0c000-7f6f0bb11000 rw-p 00000000 00:00 0 
7f6f0bb11000-7f6f0bcd1000 r-xp 00000000 00:2c 618851831950               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfvOptions.so
7f6f0bcd1000-7f6f0bed0000 ---p 001c0000 00:2c 618851831950               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfvOptions.so
7f6f0bed0000-7f6f0bed9000 r--p 001bf000 00:2c 618851831950               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfvOptions.so
7f6f0bed9000-7f6f0bedc000 rw-p 001c8000 00:2c 618851831950               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfvOptions.so
7f6f0bedc000-7f6f0bedd000 rw-p 00000000 00:00 0 
7f6f0bedd000-7f6f0dabb000 r-xp 00000000 00:2c 618851831888               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
7f6f0dabb000-7f6f0dcba000 ---p 01bde000 00:2c 618851831888               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
7f6f0dcba000-7f6f0dd3b000 r--p 01bdd000 00:2c 618851831888               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
7f6f0dd3b000-7f6f0dd4c000 rw-p 01c5e000 00:2c 618851831888               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
7f6f0dd4c000-7f6f0dd57000 rw-p 00000000 00:00 0 
7f6f0dd57000-7f6f0ddd1000 r-xp 00000000 00:2c 618851831908               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTransportModels.so
7f6f0ddd1000-7f6f0dfd1000 ---p 0007a000 00:2c 618851831908               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTransportModels.so
7f6f0dfd1000-7f6f0dfd3000 r--p 0007a000 00:2c 618851831908               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTransportModels.so
7f6f0dfd3000-7f6f0dfd4000 rw-p 0007c000 00:2c 618851831908               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTransportModels.so
7f6f0dfd4000-7f6f0dfd5000 rw-p 00000000 00:00 0 
7f6f0dfd5000-7f6f0e228000 r-xp 00000000 00:2c 618851831925               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTurbulenceModels.so
7f6f0e228000-7f6f0e427000 ---p 00253000 00:2c 618851831925               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTurbulenceModels.so
7f6f0e427000-7f6f0e431000 r--p 00252000 00:2c 618851831925               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTurbulenceModels.so
7f6f0e431000-7f6f0e435000 rw-p 0025c000 00:2c 618851831925               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libincompressibleTurbulenceModels.so
7f6f0e435000-7f6f0e62d000 r-xp 00000000 00:2c 618851831924               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libturbulenceModels.so
7f6f0e62d000-7f6f0e82c000 ---p 001f8000 00:2c 618851831924               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libturbulenceModels.so
7f6f0e82c000-7f6f0e836000 r--p 001f7000 00:2c 618851831924               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libturbulenceModels.so
7f6f0e836000-7f6f0e839000 rw-p 00201000 00:2c 618851831924               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libturbulenceModels.so
7f6f0e839000-7f6f0e83a000 rw-p 00000000 00:00 0 
7f6f0e83a000-7f6f0e96a000 r-xp 00000000 00:2c 618851831923               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libradiationModels.so
7f6f0e96a000-7f6f0eb6a000 ---p 00130000 00:2c 618851831923               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libradiationModels.so
7f6f0eb6a000-7f6f0eb70000 r--p 00130000 00:2c 618851831923               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libradiationModels.so
7f6f0eb70000-7f6f0eb72000 rw-p 00136000 00:2c 618851831923               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libradiationModels.so
7f6f0eb72000-7f6f0eb73000 rw-p 00000000 00:00 0 
7f6f0eb73000-7f6f0f0aa000 r-xp 00000000 00:2c 618851831911               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libspecie.so
7f6f0f0aa000-7f6f0f2aa000 ---p 00537000 00:2c 618851831911               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libspecie.so
7f6f0f2aa000-7f6f0f2c3000 r--p 00537000 00:2c 618851831911               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libspecie.so
7f6f0f2c3000-7f6f0f2c6000 rw-p 00550000 00:2c 618851831911               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libspecie.so
7f6f0f2c6000-7f6f0f2c8000 rw-p 00000000 00:00 0 
7f6f0f2c8000-7f6f0f5da000 r-xp 00000000 00:2c 618851831914               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfluidThermophysicalModels.so
7f6f0f5da000-7f6f0f7da000 ---p 00312000 00:2c 618851831914               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfluidThermophysicalModels.so
7f6f0f7da000-7f6f0f7ef000 r--p 00312000 00:2c 618851831914               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfluidThermophysicalModels.so
7f6f0f7ef000-7f6f0f7f1000 rw-p 00327000 00:2c 618851831914               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libfluidThermophysicalModels.so
7f6f0f7f1000-7f6f0f7f2000 rw-p 00000000 00:00 0 
7f6f0f7f2000-7f6f0f7f5000 r-xp 00000000 00:2c 618851831909               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTransportModels.so
7f6f0f7f5000-7f6f0f9f4000 ---p 00003000 00:2c 618851831909               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTransportModels.so
7f6f0f9f4000-7f6f0f9f5000 r--p 00002000 00:2c 618851831909               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTransportModels.so
7f6f0f9f5000-7f6f0f9f6000 rw-p 00003000 00:2c 618851831909               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libcompressibleTransportModels.so
7f6f0f9f6000-7f6f0fa50000 r-xp 00000000 00:2c 618851831937               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianTurbulence.so
7f6f0fa50000-7f6f0fc4f000 ---p 0005a000 00:2c 618851831937               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianTurbulence.so
7f6f0fc4f000-7f6f0fc52000 r--p 00059000 00:2c 618851831937               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianTurbulence.so
7f6f0fc52000-7f6f0fc53000 rw-p 0005c000 00:2c 618851831937               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianTurbulence.so
7f6f0fc53000-7f6f1026b000 r-xp 00000000 00:2c 618851831936               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so
7f6f1026b000-7f6f1046a000 ---p 00618000 00:2c 618851831936               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so
7f6f1046a000-7f6f1048c000 r--p 00617000 00:2c 618851831936               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so
7f6f1048c000-7f6f10492000 rw-p 00639000 00:2c 618851831936               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so
7f6f10492000-7f6f10496000 rw-p 00000000 00:00 0 
7f6f10496000-7f6f104bf000 r-xp 00000000 00:2c 618851831889               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangian.so
7f6f104bf000-7f6f106bf000 ---p 00029000 00:2c 618851831889               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangian.so
7f6f106bf000-7f6f106c0000 r--p 00029000 00:2c 618851831889               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangian.so
7f6f106c0000-7f6f106c1000 rw-p 0002a000 00:2c 618851831889               /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangian.so
7f6f106c1000-7f6f106e2000 r-xp 00000000 08:02 658314                     /usr/lib64/ld-2.17.so
7f6f106f7000-7f6f108ac000 rw-p 00000000 00:00 0 
7f6f108ac000-7f6f108be000 r-xp 00000000 08:02 661626                     /usr/lib64/libudev.so.1.6.2
7f6f108be000-7f6f108bf000 ---p 00012000 08:02 661626                     /usr/lib64/libudev.so.1.6.2
7f6f108bf000-7f6f108c0000 r--p 00012000 08:02 661626                     /usr/lib64/libudev.so.1.6.2
7f6f108c0000-7f6f108c1000 rw-p 00013000 08:02 661626                     /usr/lib64/libudev.so.1.6.2
7f6f108c1000-7f6f108cc000 rw-p 00000000 00:00 0 
7f6f108d6000-7f6f108d8000 rw-p 00000000 00:00 0 
7f6f108d8000-7f6f108d9000 rw-s 00000000 08:11 536872579                  /state/partition1/21705.txc7ptorque01.local/ompi.backend-1-06.8762/pid.52027/pmix_dstor_52027/dstore_sm.lock
7f6f108d9000-7f6f108da000 r--s 00000000 08:11 536872578                  /state/partition1/21705.txc7ptorque01.local/ompi.backend-1-06.8762/pid.52027/pmix_dstor_52027/initial-pmix_shared-segment-0
7f6f108da000-7f6f108e2000 rw-p 00000000 00:00 0 
7f6f108e2000-7f6f108e3000 r--p 00021000 08:02 658314                     /usr/lib64/ld-2.17.so
7f6f108e3000-7f6f108e4000 rw-p 00022000 08:02 658314                     /usr/lib64/ld-2.17.so
7f6f108e4000-7f6f108e5000 rw-p 00000000 00:00 0 
7ffe34659000-7ffe346e0000 rw-p 00000000 00:00 0                          [stack]
7ffe346f4000-7ffe346f6000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[backend-1-06:52032] *** Process received signal ***
[backend-1-06:52032] Signal: Aborted (6)
[backend-1-06:52032] Signal code:  (-6)
[backend-1-06:52032] [ 0] /lib64/libc.so.6(+0x35270)[0x7f6f094e5270]
[backend-1-06:52032] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7f6f094e51f7]
[backend-1-06:52032] [ 2] /lib64/libc.so.6(abort+0x148)[0x7f6f094e68e8]
[backend-1-06:52032] [ 3] /lib64/libc.so.6(+0x74f47)[0x7f6f09524f47]
[backend-1-06:52032] [ 4] /lib64/libc.so.6(+0x7dd7d)[0x7f6f0952dd7d]
[backend-1-06:52032] [ 5] /lib64/libc.so.6(__libc_malloc+0x4c)[0x7f6f0953010c]
[backend-1-06:52032] [ 6] /lib64/libc.so.6(vasprintf+0x22)[0x7f6f09523e62]
[backend-1-06:52032] [ 7] /lib64/libc.so.6(asprintf+0x87)[0x7f6f09501a87]
[backend-1-06:52032] [ 8] /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(+0x3cbe0)[0x7f6f060f5be0]
[backend-1-06:52032] [ 9] /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(ompi_mpi_errors_are_fatal_comm_handler+0xb4)[0x7f6f060f64b4]
[backend-1-06:52032] [10] /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(ompi_errhandler_invoke+0x115)[0x7f6f060f5895]
[backend-1-06:52032] [11] /share/apps/openmpi/openmpi-3.0.0/lib/libmpi.so.40(PMPI_Send+0x163)[0x7f6f06133c83]
[backend-1-06:52032] [12] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/openmpi-system/libPstream.so(_ZN4Foam9UOPstream5writeENS_8UPstream10commsTypesEiPKclii+0x587)[0x7f6f092a5ad7]
[backend-1-06:52032] [13] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam9UOPstreamD1Ev+0x5e)[0x7f6f0a6926ae]
[backend-1-06:52032] [14] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZNK4Foam14fileOperations23uncollatedFileOperation4readERNS_11regIOobjectEbNS_8IOstream12streamFormatERKNS_4wordE+0x352)[0x7f6f0a5d85c2]
[backend-1-06:52032] [15] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam11regIOobject12readHeaderOkENS_8IOstream12streamFormatERKNS_4wordE+0x5f)[0x7f6f0a6cc97f]
[backend-1-06:52032] [16] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so(_ZN4Foam13GlobalIOFieldINS_6VectorIdEEEC2ERKNS_8IOobjectE+0x52)[0x7f6f0ff28c22]
[backend-1-06:52032] [17] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so(_ZN4Foam15ManualInjectionINS_14KinematicCloudINS_5CloudINS_15CollidingParcelINS_15KinematicParcelINS_8particleEEEEEEEEEEC1ERKNS_10dictionaryERS9_RKNS_4wordE+0xf5)[0x7f6f0ffcf7c5]
[backend-1-06:52032] [18] /share/apps/OpenFOAM/OF6x/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/liblagrangianIntermediate.so(_ZN4Foam14InjectionModelINS_14KinematicCloudINS_5CloudINS_15CollidingParcelINS_15KinematicParcelINS_8particleEEEEEEEEEE31adddictionaryConstructorToTableINS_15ManualInjectionIS9_EEE3NewERKNS_10dictionaryERS9_RKNS_4wordE+0x32)[0x7f6f0ffd0022]
[backend-1-06:52032] [19] icoUncoupledKinematicParcelFoam[0x43cfb8]
[backend-1-06:52032] [20] icoUncoupledKinematicParcelFoam[0x43d18b]
[backend-1-06:52032] [21] icoUncoupledKinematicParcelFoam[0x4592f2]
[backend-1-06:52032] [22] icoUncoupledKinematicParcelFoam[0x45a21c]
[backend-1-06:52032] [23] icoUncoupledKinematicParcelFoam[0x41c7f2]
[backend-1-06:52032] [24] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6f094d1c05]
[backend-1-06:52032] [25] icoUncoupledKinematicParcelFoam[0x41ecb3]
[backend-1-06:52032] *** End of error message ***
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node backend-1-06 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
log (67,592 bytes)   

henry

2019-04-04 23:40

manager   ~0010411

Are you running with 32bit or 64bit labels? To run with this many particles you will need to compile and run with 64bit labels.

ctsang

2019-04-05 05:24

reporter   ~0010412

Will recompile with 64 bit labels to see if it resolves this issue. Thanks.

ctsang

2019-04-05 20:16

reporter   ~0010413

Have recompiled and run with 64 bit labels. However, the issue persists. The log file is attached. The error message is somewhat different from the previous one with 32bit labels. I guess the real question is that why the serial run works, but parallel run fails. Thanks.
log-2 (2,219 bytes)   
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6-eb0ce5d792a1
Exec   : myicoUncoupledKinematicParcelFoam -parallel
Date   : Apr 05 2019
Time   : 14:04:32
Host   : "backend-1-04.local"
PID    : 132006
I/O    : uncollated
Case   : /home/ua18762/hopperInitialState
nProcs : 2
Slaves : 1("backend-1-04.local.132007")
Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0


Reading g
Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting turbulence model type laminar
Selecting laminar stress model Stokes
Constructing kinematicCloud kinematicCloud
Constructing particle forces
    Selecting particle force sphereDrag
    Selecting particle force gravity
Constructing cloud functions
    none
Constructing particle injection models
Creating injector: model1
Selecting injection model manualInjection
    Constructing 3-D injection
    Choosing nParticle to be a fixed value, massTotal variable now does not determine anything.
[backend-1-04:132006] *** An error occurred in MPI_Send
[backend-1-04:132006] *** reported by process [6229983233,139732466008064]
[backend-1-04:132006] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
[backend-1-04:132006] *** MPI_ERR_COUNT: invalid count argument
[backend-1-04:132006] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[backend-1-04:132006] ***    and potentially your MPI job)
log-2 (2,219 bytes)   

henry

2019-04-05 20:37

manager   ~0010414

It looks like there is an issue with the MPI you are using.

henry

2019-04-05 20:40

manager   ~0010415

> In constant/kinematicParcelPositions, copy and paste all the coordinates 33333 times.

This is not at all convenient, can you provide a simpler way of reproducing the problem?

ctsang

2019-04-05 21:33

reporter   ~0010416

1) Please download kinematicParcelPositions from this link: <https://drive.google.com/open?id=1tNo3kPHM8fZZ9gZCVwGx0ctgyHSqDJtk>. This has ~100 million parcel positions.
2) Replace the original kinematicParcelPositions with this one in hopperInitialState.
3) Run icoUncoupledKinematicParcelFoam with multiple cores.

Thanks.

henry

2019-04-06 20:37

manager   ~0010418

The problem is that MPI_Send in MPI-2.2 implementations have a message size limit of 2^31-1 elements and your positions list exceeds this.

henry

2019-04-08 08:33

manager   ~0010423

See also https://www.mpi-forum.org/mpi-40/

There is a group working on this issue:

Large Counts

    Lead: Jeff Hammond
    Scope: Understanding and fixing the issues associated with integer counts and displacements.

ctsang

2019-04-08 23:42

reporter   ~0010425

I varied the number of injectors (number of injection models) and keep the total number of parcels the same. The assumption is that MPI_send is called by each injector, and each MPI_send will pass less message size. This partly solves the issue, but still has some interesting behaviors. Please see the test summary below:

# of processors # of injectors # of parcels per injector (M) # of total parcels (M) Observation
1 1 112 112 Can run
2 1 112 112 Failed to initialize
2 2 56 112 Can run
4 4 28 112 Can run
28 2 56 112 Failed to initialize
28 28 4 112 Failed to initialize

The above shows that runs w/ 28 processors failed. Not sure why. The log file of 28 processors and 28 injectors is attached. Now the error message does not show "memory corruption" or "MPI_Send" we saw before. Thanks.
log-3 (10,380 bytes)   
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     | Website:  https://openfoam.org
    \\  /    A nd           | Version:  6
     \\/     M anipulation  |
\*---------------------------------------------------------------------------*/
Build  : 6
Exec   : MPPICFoam -parallel
Date   : Apr 08 2019
Time   : 16:44:22
Host   : "backend-1-08.local"
PID    : 106843
I/O    : uncollated
Case   : /home/ua18762/OpenFOAM/ua18762-6/run/MPPICFoam/NETL_BFB_QuanCutCell_v02_12ft_parcelStudy3
nProcs : 28
Slaves : 
27
(
"backend-1-08.local.106844"
"backend-1-08.local.106845"
"backend-1-08.local.106846"
"backend-1-08.local.106847"
"backend-1-08.local.106848"
"backend-1-08.local.106849"
"backend-1-08.local.106850"
"backend-1-08.local.106851"
"backend-1-08.local.106852"
"backend-1-08.local.106853"
"backend-1-08.local.106854"
"backend-1-08.local.106855"
"backend-1-08.local.106856"
"backend-1-08.local.106857"
"backend-1-08.local.106858"
"backend-1-08.local.106859"
"backend-1-08.local.106860"
"backend-1-08.local.106861"
"backend-1-08.local.106862"
"backend-1-08.local.106863"
"backend-1-08.local.106864"
"backend-1-08.local.106865"
"backend-1-08.local.106866"
"backend-1-08.local.106867"
"backend-1-08.local.106868"
"backend-1-08.local.106869"
"backend-1-08.local.106870"
)

Pstream initialized with:
    floatTransfer      : 0
    nProcsSimpleSum    : 0
    commsType          : nonBlocking
    polling iterations : 0
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 10)
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0


PIMPLE: No convergence criteria found


PIMPLE: Operating solver in PISO mode


Reading g
Reading field U

Reading field p

Reading/calculating continuous-phase face flux field phic

Creating turbulence model

Selecting incompressible transport model Newtonian
Creating field alphac

Constructing kinematicCloud kinematicCloud
Constructing particle forces
    Selecting particle force WenYuDrag
    Selecting particle force gravity
Constructing cloud functions
    none
Constructing particle injection models
Creating injector: model1
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model2
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model3
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model4
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model5
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model6
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model7
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model8
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model9
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model10
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model11
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model12
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model13
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model14
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model15
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model16
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model17
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model18
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model19
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model20
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model21
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model22
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model23
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model24
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model25
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model26
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
    number density      = 2.886e+06
    number of particles = 4019539
Creating injector: model27
Selecting injection model cellZoneInjection
    Constructing 3-D injection
Selecting distribution model fixedValue
    cell zone size      = 121318
    cell zone volume    = 1.39277
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 12 with PID 0 on node backend-1-08 exited on signal 9 (Killed).
--------------------------------------------------------------------------
log-3 (10,380 bytes)   

henry

2020-02-05 20:20

manager   ~0011123

This issue is in MPI:

See also https://www.mpi-forum.org/mpi-40/

There is a group working on this issue:

Large Counts

    Lead: Jeff Hammond
    Scope: Understanding and fixing the issues associated with integer counts and displacements.

Issue History

Date Modified Username Field Change
2019-04-04 23:12 ctsang New Issue
2019-04-04 23:12 ctsang File Added: log
2019-04-04 23:40 henry Note Added: 0010411
2019-04-05 05:24 ctsang Note Added: 0010412
2019-04-05 20:16 ctsang File Added: log-2
2019-04-05 20:16 ctsang Note Added: 0010413
2019-04-05 20:37 henry Note Added: 0010414
2019-04-05 20:40 henry Note Added: 0010415
2019-04-05 21:33 ctsang Note Added: 0010416
2019-04-06 20:37 henry Note Added: 0010418
2019-04-08 08:33 henry Note Added: 0010423
2019-04-08 23:42 ctsang File Added: log-3
2019-04-08 23:42 ctsang Note Added: 0010425
2020-02-05 20:20 henry Assigned To => henry
2020-02-05 20:20 henry Status new => closed
2020-02-05 20:20 henry Resolution open => no change required
2020-02-05 20:20 henry Note Added: 0011123