View Issue Details

IDProjectCategoryView StatusLast Update
0002067OpenFOAMBugpublic2016-05-18 12:13
Reporterpaukap Assigned Tohenry  
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionunable to reproduce 
PlatformIntel 64bit (Xeon Westmere)OSLinux (CentOS 7)OS Version7.1
Summary0002067: Multiple crashs when running 'tutorials'
DescriptionOpenFOAM 3.0.1 has been installed in our environment (CentOS 7, RHEl-comp.) using Intel compilers version 16.0, GCC 4.8.5 (distribution-std.), OpenMPI 1.10 and Intel MPI.

Trying to run the tutorials, we observe multiple crashes of different applications with mainly two scenarios: either both GCC and Intel compiled applications crash with (likely) the same issue, OR the Intel-compiled version fail while the GCC-compiled version have success (examples see below).

Is it known about any incompatibility issues with Intel compilers, should we compile OpenFOAM with a particular version of the Intel compiler?

Should we report all the issues en détail?

Have a nice day,
Paul Kapinos
------------------------------------------------------------------------------
example 1: same error in Intel's and GCC's versions, in
tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel

#0 Foam::error::printStack(Foam::Ostream&) at ??:?
#1 Foam::sigFpe::sigHandler(int) at ??:?
#2 ? in "/lib64/libc.so.6"
#3 ? in "/lib64/libm.so.6"
#4 Foam::H2O::mu(double, double) const at ??:?
#5 Foam::regionModels::surfaceFilmModels::liquidFilmThermo::mu() const at ??:?
#6 Foam::regionModels::surfaceFilmModels::liquidViscosity::correct(Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&) at ??:?
#7 Foam::regionModels::surfaceFilmModels::thermoSingleLayer::solveEnergy() at ??:?
#8 Foam::regionModels::surfaceFilmModels::thermoSingleLayer::evolveRegion() at ??:?
#9 Foam::regionModels::regionModel::evolve() at ??:?
#10 ? at ??:?
#11 __libc_start_main in "/lib64/libc.so.6"
#12 ? at ??:?

------------------------------------------------------------------------------
example 2: GCC's version of 'decomposePar' is OK while Intel's version fails, in
tutorials/multiphase/interFoam/laminar/damBreak
and in
tutorials/combustion/fireFoam/les/oppositeBurningPanels
.... with different error messages:

------------------------------
--> FOAM FATAL ERROR:
Attempt to cast type wall to type coupled

    From function refCast<To>(From&)
    in file /usr/local_rwth/sw/openfoam/3.0.1/intel_16.0.1.150-openmpi_1.10.1/OpenFOAM-3.0.1/src/OpenFOAM/lnInclude/typeInfo.H at line 114.

FOAM aborting

#0 Foam::error::printStack(Foam::Ostream&) at ??:?
#1 Foam::error::abort() at ??:?
#2 Foam::Ostream& Foam::operator<< <Foam::error>(Foam::Ostream&, Foam::errorManip<Foam::error>) at ??:?
#3 ? at ??:?
#4 ? at domainDecomposition.C:?
#5 ? at decomposePar.C:?
#6 __libc_start_main in "/lib64/libc.so.6"
#7 ? at ??:?

------------------------------
Constructing processor meshes
#0 Foam::error::printStack(Foam::Ostream&) at ??:?
#1 Foam::sigSegv::sigHandler(int) at ??:?
#2 ? in "/lib64/libc.so.6"
#3 Foam::PtrList<Foam::polyPatch>::operator[](int) const at ??:?
#4 ? at domainDecomposition.C:?
#5 ? at decomposePar.C:?
#6 __libc_start_main in "/lib64/libc.so.6"
#7 ? at ??:?

------------------------------------------------------------------------------

Steps To Reproduce"Allrun" in 'tutorials' or run the particular test cases (e.g. in 'tutorials/multiphase/interFoam/laminar/damBreak' run 'blockMesh; decomposePar'
TagsNo tags attached.

Activities

paukap

2016-04-25 10:57

reporter  

henry

2016-04-25 11:08

manager   ~0006171

There are indeed issues with the

    tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel

case which are resolved in OpenFOAM-dev. I will make the same changes to OpenFOAM-3.0.x today.

I have not seen any problems with the

   tutorials/multiphase/interFoam/laminar/damBreak

case with any compiler on any OS. However, I have Intel compiler version 15.0.3 installed and have not tested 16.0. It looks like the problems you see with icpc-16 are failures of the optimizer, try compiling with a lower optimization level.

Are there any other tutorial cases which do not run when compiled with gcc?

henry

2016-04-25 17:38

manager   ~0006173

tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel
    Resolved in OpenFOAM-3.0.x by commit 81f713cc3751b0594230d1359914a1b9c4221a24

henry

2016-04-25 18:28

manager   ~0006174

I have just run the tutorials in question in OpenFOAM-3.0.x compiled with icpc-15.0.3 without any problems. It looks like there is an issue with icpc-16.0 which you may wish to report to the compiler developers at Intel.

henry

2016-04-26 10:57

manager   ~0006177

I found a problem with surfaceFeatureExtract compiled with icpc; it looks like the optimizer in icpc is too agressive and incorrectly compiles some code. I resolved the problem by removing the -xHost option but -O3 seems OK. Now with icpc-15.0.3 all of the tutorials run in OpenFOAM-3.0.x and dev. I have not tested icpc-16.0.

henry

2016-05-03 15:12

manager   ~0006222

This is probably resolved, please re-open when you have answers to the open questions.

paukap

2016-05-13 11:14

reporter   ~0006261

Hi Henry,

now we see issues on 'snappyHexMesh' in 'tutorials/mesh/snappyHexMesh/flange': both intel/16 and gcc/4.8 compied versions fail with a core dump and message
> floating point exception (core dumped) snappyHexMesh -overwrite
... and below stack traces.

About intel/16: would you like to test OpenFAOM in our environment? We have a plenty of compilers here, Intel including latest and betas, PGI, GCC in many versions, even Oracle Studio... If yes please contact me via eMail.

Best
Paul

------------------------------------------------------------------------------
intel/16:
....
....
Setting refinement level of surface to be consistent with shells.
#0 Foam::error::printStack(Foam::Ostream&) at ??:?
#1 Foam::sigFpe::sigHandler(int) at ??:?
#2 ? in "/lib64/libc.so.6"
#3 ? at triangleFuncs.C:?
#4 Foam::triangleFuncs::intersectBb(Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::treeBoundBox const&) at ??:?
#5 Foam::treeDataPrimitivePatch<Foam::triSurface>::overlaps(int, Foam::treeBoundBox const&) const at ??:?
#6 ? at triSurfaceSearch.C:?
#7 ? at triSurfaceSearch.C:?
#8 ? at triSurfaceSearch.C:?
#9 ? at triSurfaceSearch.C:?
#10 ? at triSurfaceSearch.C:?
#11 ? at refinementSurfaces.C:?
#12 ? at ??:?
#13 __libc_start_main in "/lib64/libc.so.6"
#14 ? at ??:?

------------------------------------------------------------------------------
gcc/4.8:
.....
.....
Setting refinement level of surface to be consistent with shells.
#0 Foam::error::printStack(Foam::Ostream&) at ??:?
#1 Foam::sigFpe::sigHandler(int) at ??:?
#2 ? in "/lib64/libc.so.6"
#3 Foam::triangleFuncs::intersectAxesBundle(Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::Vector<double> const&, int, Foam::Field<Foam::Vector<double> > const&, double, Foam::Vector<double>&) at ??:?
#4 Foam::triangleFuncs::intersectBb(Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::treeBoundBox const&) at ??:?
#5 Foam::indexedOctree<Foam::treeDataPrimitivePatch<Foam::triSurface> >::divide(Foam::List<int> const&, Foam::treeBoundBox const&, Foam::List<Foam::List<int> >&) const at ??:?
#6 Foam::indexedOctree<Foam::treeDataPrimitivePatch<Foam::triSurface> >::divide(Foam::treeBoundBox const&, Foam::DynamicList<Foam::List<int>, 0u, 2u, 1u>&, int) const at ??:?
#7 Foam::indexedOctree<Foam::treeDataPrimitivePatch<Foam::triSurface> >::indexedOctree(Foam::treeDataPrimitivePatch<Foam::triSurface> const&, Foam::treeBoundBox const&, int, double, double) at ??:?
#8 Foam::triSurfaceSearch::tree() const at ??:?
#9 Foam::triSurfaceSearch::findNearest(Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&, Foam::List<Foam::PointIndexHit<Foam::Vector<double> > >&) const at ??:?
#10 Foam::refinementSurfaces::setMinLevelFields(Foam::shellSurfaces const&) at ??:?
#11 ? at ??:?
#12 __libc_start_main in "/lib64/libc.so.6"
#13 ? at ??:?
------------------------------------------------------------------------------

henry

2016-05-13 11:49

manager   ~0006262

I can run all of the tutorials with OpenFOAM-3.0.? compiled with gcc-4.8.?

Have you tested any other gcc versions?

paukap

2016-05-13 12:12

reporter   ~0006263

Yes, intel/14.0.3.174 and gcc_5.3.0 - exactly the same behaviour.
Could this be rooted in optimisation, or hardware?

henry

2016-05-13 12:28

manager   ~0006264

I am unable to reproduce this problem with any compiler I have tried.
Have you tried OpenFOAM-3.0.x or OpenFOAM-dev?

paukap

2016-05-13 12:44

reporter   ~0006265

We use OpenFOAM-3.0.1 (release).

paukap

2016-05-13 12:46

reporter   ~0006266

304e6a14b9e69c20989527f5fb1ed724 OpenFOAM-3.0.1.tgz
4665072d7d29ab9af5ced402f667185a ThirdParty-3.0.1.tgz

However, we enable quite well optimisation. Could this be an issue? The same on Intel and GCC side?!

henry

2016-05-13 13:04

manager   ~0006267

Have you changed the optimization options for these compilers provided with OpenFOAM? If so what changes? Also have you tried to compile with the options provided?

paukap

2016-05-13 22:03

reporter   ~0006270

We added '-ffast-math' for GCC, and for Intel added '-ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 ' for C, and additionally '-no-prec-div' and '-O3' instead of '-O2' for 'c++Opt'.

Looks like this could be dangerous idea:
> -ffast-math
> Sets -fno-math-errno, -funsafe-math-optimizations,
> -ffinite-math-only, -fno-rounding-math, -fno-signaling-nans and
> -fcx-limited-range.
>
> This option causes the preprocessor macro "__FAST_MATH__" to be
> defined.
>
> This option is not turned on by any -O option besides -Ofast since
> it can result in incorrect output for programs that depend on an
> exact implementation of IEEE or ISO rules/specifications for math
> functions. It may, however, yield faster code for programs that do
> not require the guarantees of these specifications.

Well, will switch back to original values (but keep '-ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 ' for Intel) and try is again.




------------------------------------------------------------------------------
GCC:
c++Opt ------------------------------------------------------------------------------
c++DBUG =
#c++OPT = -O3
c++OPT = -O3 -ffast-math
ROUNDING_MATH = -frounding-math

cOpt ------------------------------------------------------------------------------
cDBUG =
#cOPT = -O3
cOPT = -O3 -ffast-math
------------------------------------------------------------------------------


------------------------------------------------------------------------------
Intel:
c++Opt ------------------------------------------------------------------------------
c++DBUG =
#c++OPT = -xHost -O2
c++OPT = -O3 -ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 -no-prec-div

cOpt ------------------------------------------------------------------------------
cDBUG =
#cOPT = -O3 -no-prec-div
cOPT = -O3 -ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 -no-prec-div
------------------------------------------------------------------------------

henry

2016-05-13 22:44

manager   ~0006271

The '-ffast-math' limits the precision of some operations which causes many utilities and solvers to fail which is why we do not recommend it.

I have also tested many combinations of optimizations on the Intel compiler and only those we provide in the OpenFOAM wmake files are reliable. If you have further problems with the operation of OpenFOAM please test with the default compiler switches we provide.

paukap

2016-05-18 11:49

reporter   ~0006288

Dear henry, many thanks for the support.

Indeed, changing to very vanilla optimisation flags did it - issues on 'decomposePar' and 'snappyHexMesh' vanish. Surprizing, as we compiled many versions of OpenFOAM previously with "our" optimisation flags (which are "not that aggressive", indeed).

However, workarounding issues by looking out for yet another lower optimisation flag seem not to be very reliable way we would like to go next years. Especially because '-xHost' is a very bad idea for us (we have many different CPUs here - on newer CPUs this flag lead to non-optimal runs, on older to 'unknown instruction', likely), we opened an issue by Intel compiler developers. Maybe they will find something.





Next case: running 'Alltest' in 'tutorials' wee see failures in many examples, which are this times not core dumps et cetera, but incorrect execution of some examples. Behaviour is the same on GCC and Intel compiled versions.

a) Failures of 'reconstructPar' e.g. in 'tutorialsTest/discreteMethods/dsmcFoam/wedge15Ma5' with this message in 'log.reconstructPar':
> --> FOAM FATAL ERROR:
> No times selected
> From function reconstructPar
> in file reconstructPar.C at line 210.
> FOAM exiting

b) Failures in 'fireFoam' with messages about non-available 'controlDict' file, which *is* available in lower dir:

pk224850@cluster-beta:/work/pk224850/TEMP/OpenFOAM/20160517-TEST/intel_16.0.2.181-openmpi_1.10.2/tutorialsTest/combustion/fireFoam/les/smallPoolFire3D[651]$ find . -iname controlDict
./system/controlDict

> --> FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file
> 'controlDict'

Maybe you could take a look at this and/or report about this is known behaviour?

Have a nice day,
Paul
P.S. no interest in testing newest Intel compilers in our environment?

henry

2016-05-18 12:12

manager   ~0006289

I cannot reproduce the problem you are having running tutorials. We would be happy to test any recent versions of OpenFOAM with any compilers in your environment but this work would need to be funded either as part of a support contract or a more specific arrangement. We could also help with any cases which still do not run on your systems.

Issue History

Date Modified Username Field Change
2016-04-25 10:57 paukap New Issue
2016-04-25 10:57 paukap File Added: 20160425-errors301.tar.gz
2016-04-25 11:08 henry Note Added: 0006171
2016-04-25 17:38 henry Note Added: 0006173
2016-04-25 18:28 henry Note Added: 0006174
2016-04-26 10:57 henry Note Added: 0006177
2016-05-03 15:12 henry Note Added: 0006222
2016-05-03 15:12 henry Status new => resolved
2016-05-03 15:12 henry Resolution open => fixed
2016-05-03 15:12 henry Assigned To => henry
2016-05-13 11:14 paukap Note Added: 0006261
2016-05-13 11:14 paukap Status resolved => feedback
2016-05-13 11:14 paukap Resolution fixed => reopened
2016-05-13 11:49 henry Note Added: 0006262
2016-05-13 12:12 paukap Note Added: 0006263
2016-05-13 12:12 paukap Status feedback => assigned
2016-05-13 12:28 henry Note Added: 0006264
2016-05-13 12:44 paukap Note Added: 0006265
2016-05-13 12:46 paukap Note Added: 0006266
2016-05-13 13:04 henry Note Added: 0006267
2016-05-13 22:03 paukap Note Added: 0006270
2016-05-13 22:44 henry Note Added: 0006271
2016-05-13 22:44 henry Status assigned => closed
2016-05-13 22:44 henry Resolution reopened => no change required
2016-05-18 11:49 paukap Note Added: 0006288
2016-05-18 11:49 paukap Status closed => feedback
2016-05-18 11:49 paukap Resolution no change required => reopened
2016-05-18 12:12 henry Note Added: 0006289
2016-05-18 12:13 henry Status feedback => closed
2016-05-18 12:13 henry Resolution reopened => unable to reproduce