View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002067 | OpenFOAM | Bug | public | 2016-04-25 10:57 | 2016-05-18 12:13 |
| Reporter | paukap | Assigned To | henry | ||
| Priority | normal | Severity | crash | Reproducibility | always |
| Status | closed | Resolution | unable to reproduce | ||
| Platform | Intel 64bit (Xeon Westmere) | OS | Linux (CentOS 7) | OS Version | 7.1 |
| Summary | 0002067: Multiple crashs when running 'tutorials' | ||||
| Description | OpenFOAM 3.0.1 has been installed in our environment (CentOS 7, RHEl-comp.) using Intel compilers version 16.0, GCC 4.8.5 (distribution-std.), OpenMPI 1.10 and Intel MPI. Trying to run the tutorials, we observe multiple crashes of different applications with mainly two scenarios: either both GCC and Intel compiled applications crash with (likely) the same issue, OR the Intel-compiled version fail while the GCC-compiled version have success (examples see below). Is it known about any incompatibility issues with Intel compilers, should we compile OpenFOAM with a particular version of the Intel compiler? Should we report all the issues en détail? Have a nice day, Paul Kapinos ------------------------------------------------------------------------------ example 1: same error in Intel's and GCC's versions, in tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel #0 Foam::error::printStack(Foam::Ostream&) at ??:? #1 Foam::sigFpe::sigHandler(int) at ??:? #2 ? in "/lib64/libc.so.6" #3 ? in "/lib64/libm.so.6" #4 Foam::H2O::mu(double, double) const at ??:? #5 Foam::regionModels::surfaceFilmModels::liquidFilmThermo::mu() const at ??:? #6 Foam::regionModels::surfaceFilmModels::liquidViscosity::correct(Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&, Foam::GeometricField<double, Foam::fvPatchField, Foam::volMesh> const&) at ??:? #7 Foam::regionModels::surfaceFilmModels::thermoSingleLayer::solveEnergy() at ??:? #8 Foam::regionModels::surfaceFilmModels::thermoSingleLayer::evolveRegion() at ??:? #9 Foam::regionModels::regionModel::evolve() at ??:? #10 ? at ??:? #11 __libc_start_main in "/lib64/libc.so.6" #12 ? at ??:? ------------------------------------------------------------------------------ example 2: GCC's version of 'decomposePar' is OK while Intel's version fails, in tutorials/multiphase/interFoam/laminar/damBreak and in tutorials/combustion/fireFoam/les/oppositeBurningPanels .... with different error messages: ------------------------------ --> FOAM FATAL ERROR: Attempt to cast type wall to type coupled From function refCast<To>(From&) in file /usr/local_rwth/sw/openfoam/3.0.1/intel_16.0.1.150-openmpi_1.10.1/OpenFOAM-3.0.1/src/OpenFOAM/lnInclude/typeInfo.H at line 114. FOAM aborting #0 Foam::error::printStack(Foam::Ostream&) at ??:? #1 Foam::error::abort() at ??:? #2 Foam::Ostream& Foam::operator<< <Foam::error>(Foam::Ostream&, Foam::errorManip<Foam::error>) at ??:? #3 ? at ??:? #4 ? at domainDecomposition.C:? #5 ? at decomposePar.C:? #6 __libc_start_main in "/lib64/libc.so.6" #7 ? at ??:? ------------------------------ Constructing processor meshes #0 Foam::error::printStack(Foam::Ostream&) at ??:? #1 Foam::sigSegv::sigHandler(int) at ??:? #2 ? in "/lib64/libc.so.6" #3 Foam::PtrList<Foam::polyPatch>::operator[](int) const at ??:? #4 ? at domainDecomposition.C:? #5 ? at decomposePar.C:? #6 __libc_start_main in "/lib64/libc.so.6" #7 ? at ??:? ------------------------------------------------------------------------------ | ||||
| Steps To Reproduce | "Allrun" in 'tutorials' or run the particular test cases (e.g. in 'tutorials/multiphase/interFoam/laminar/damBreak' run 'blockMesh; decomposePar' | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
There are indeed issues with the tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel case which are resolved in OpenFOAM-dev. I will make the same changes to OpenFOAM-3.0.x today. I have not seen any problems with the tutorials/multiphase/interFoam/laminar/damBreak case with any compiler on any OS. However, I have Intel compiler version 15.0.3 installed and have not tested 16.0. It looks like the problems you see with icpc-16 are failures of the optimizer, try compiling with a lower optimization level. Are there any other tutorial cases which do not run when compiled with gcc? |
|
|
tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel Resolved in OpenFOAM-3.0.x by commit 81f713cc3751b0594230d1359914a1b9c4221a24 |
|
|
I have just run the tutorials in question in OpenFOAM-3.0.x compiled with icpc-15.0.3 without any problems. It looks like there is an issue with icpc-16.0 which you may wish to report to the compiler developers at Intel. |
|
|
I found a problem with surfaceFeatureExtract compiled with icpc; it looks like the optimizer in icpc is too agressive and incorrectly compiles some code. I resolved the problem by removing the -xHost option but -O3 seems OK. Now with icpc-15.0.3 all of the tutorials run in OpenFOAM-3.0.x and dev. I have not tested icpc-16.0. |
|
|
This is probably resolved, please re-open when you have answers to the open questions. |
|
|
Hi Henry, now we see issues on 'snappyHexMesh' in 'tutorials/mesh/snappyHexMesh/flange': both intel/16 and gcc/4.8 compied versions fail with a core dump and message > floating point exception (core dumped) snappyHexMesh -overwrite ... and below stack traces. About intel/16: would you like to test OpenFAOM in our environment? We have a plenty of compilers here, Intel including latest and betas, PGI, GCC in many versions, even Oracle Studio... If yes please contact me via eMail. Best Paul ------------------------------------------------------------------------------ intel/16: .... .... Setting refinement level of surface to be consistent with shells. #0 Foam::error::printStack(Foam::Ostream&) at ??:? #1 Foam::sigFpe::sigHandler(int) at ??:? #2 ? in "/lib64/libc.so.6" #3 ? at triangleFuncs.C:? #4 Foam::triangleFuncs::intersectBb(Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::treeBoundBox const&) at ??:? #5 Foam::treeDataPrimitivePatch<Foam::triSurface>::overlaps(int, Foam::treeBoundBox const&) const at ??:? #6 ? at triSurfaceSearch.C:? #7 ? at triSurfaceSearch.C:? #8 ? at triSurfaceSearch.C:? #9 ? at triSurfaceSearch.C:? #10 ? at triSurfaceSearch.C:? #11 ? at refinementSurfaces.C:? #12 ? at ??:? #13 __libc_start_main in "/lib64/libc.so.6" #14 ? at ??:? ------------------------------------------------------------------------------ gcc/4.8: ..... ..... Setting refinement level of surface to be consistent with shells. #0 Foam::error::printStack(Foam::Ostream&) at ??:? #1 Foam::sigFpe::sigHandler(int) at ??:? #2 ? in "/lib64/libc.so.6" #3 Foam::triangleFuncs::intersectAxesBundle(Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::Vector<double> const&, int, Foam::Field<Foam::Vector<double> > const&, double, Foam::Vector<double>&) at ??:? #4 Foam::triangleFuncs::intersectBb(Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::Vector<double> const&, Foam::treeBoundBox const&) at ??:? #5 Foam::indexedOctree<Foam::treeDataPrimitivePatch<Foam::triSurface> >::divide(Foam::List<int> const&, Foam::treeBoundBox const&, Foam::List<Foam::List<int> >&) const at ??:? #6 Foam::indexedOctree<Foam::treeDataPrimitivePatch<Foam::triSurface> >::divide(Foam::treeBoundBox const&, Foam::DynamicList<Foam::List<int>, 0u, 2u, 1u>&, int) const at ??:? #7 Foam::indexedOctree<Foam::treeDataPrimitivePatch<Foam::triSurface> >::indexedOctree(Foam::treeDataPrimitivePatch<Foam::triSurface> const&, Foam::treeBoundBox const&, int, double, double) at ??:? #8 Foam::triSurfaceSearch::tree() const at ??:? #9 Foam::triSurfaceSearch::findNearest(Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&, Foam::List<Foam::PointIndexHit<Foam::Vector<double> > >&) const at ??:? #10 Foam::refinementSurfaces::setMinLevelFields(Foam::shellSurfaces const&) at ??:? #11 ? at ??:? #12 __libc_start_main in "/lib64/libc.so.6" #13 ? at ??:? ------------------------------------------------------------------------------ |
|
|
I can run all of the tutorials with OpenFOAM-3.0.? compiled with gcc-4.8.? Have you tested any other gcc versions? |
|
|
Yes, intel/14.0.3.174 and gcc_5.3.0 - exactly the same behaviour. Could this be rooted in optimisation, or hardware? |
|
|
I am unable to reproduce this problem with any compiler I have tried. Have you tried OpenFOAM-3.0.x or OpenFOAM-dev? |
|
|
We use OpenFOAM-3.0.1 (release). |
|
|
304e6a14b9e69c20989527f5fb1ed724 OpenFOAM-3.0.1.tgz 4665072d7d29ab9af5ced402f667185a ThirdParty-3.0.1.tgz However, we enable quite well optimisation. Could this be an issue? The same on Intel and GCC side?! |
|
|
Have you changed the optimization options for these compilers provided with OpenFOAM? If so what changes? Also have you tried to compile with the options provided? |
|
|
We added '-ffast-math' for GCC, and for Intel added '-ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 ' for C, and additionally '-no-prec-div' and '-O3' instead of '-O2' for 'c++Opt'. Looks like this could be dangerous idea: > -ffast-math > Sets -fno-math-errno, -funsafe-math-optimizations, > -ffinite-math-only, -fno-rounding-math, -fno-signaling-nans and > -fcx-limited-range. > > This option causes the preprocessor macro "__FAST_MATH__" to be > defined. > > This option is not turned on by any -O option besides -Ofast since > it can result in incorrect output for programs that depend on an > exact implementation of IEEE or ISO rules/specifications for math > functions. It may, however, yield faster code for programs that do > not require the guarantees of these specifications. Well, will switch back to original values (but keep '-ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 ' for Intel) and try is again. ------------------------------------------------------------------------------ GCC: c++Opt ------------------------------------------------------------------------------ c++DBUG = #c++OPT = -O3 c++OPT = -O3 -ffast-math ROUNDING_MATH = -frounding-math cOpt ------------------------------------------------------------------------------ cDBUG = #cOPT = -O3 cOPT = -O3 -ffast-math ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Intel: c++Opt ------------------------------------------------------------------------------ c++DBUG = #c++OPT = -xHost -O2 c++OPT = -O3 -ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 -no-prec-div cOpt ------------------------------------------------------------------------------ cDBUG = #cOPT = -O3 -no-prec-div cOPT = -O3 -ip -axCORE-AVX2,AVX,SSE4.2,SSE4.1 -no-prec-div ------------------------------------------------------------------------------ |
|
|
The '-ffast-math' limits the precision of some operations which causes many utilities and solvers to fail which is why we do not recommend it. I have also tested many combinations of optimizations on the Intel compiler and only those we provide in the OpenFOAM wmake files are reliable. If you have further problems with the operation of OpenFOAM please test with the default compiler switches we provide. |
|
|
Dear henry, many thanks for the support. Indeed, changing to very vanilla optimisation flags did it - issues on 'decomposePar' and 'snappyHexMesh' vanish. Surprizing, as we compiled many versions of OpenFOAM previously with "our" optimisation flags (which are "not that aggressive", indeed). However, workarounding issues by looking out for yet another lower optimisation flag seem not to be very reliable way we would like to go next years. Especially because '-xHost' is a very bad idea for us (we have many different CPUs here - on newer CPUs this flag lead to non-optimal runs, on older to 'unknown instruction', likely), we opened an issue by Intel compiler developers. Maybe they will find something. Next case: running 'Alltest' in 'tutorials' wee see failures in many examples, which are this times not core dumps et cetera, but incorrect execution of some examples. Behaviour is the same on GCC and Intel compiled versions. a) Failures of 'reconstructPar' e.g. in 'tutorialsTest/discreteMethods/dsmcFoam/wedge15Ma5' with this message in 'log.reconstructPar': > --> FOAM FATAL ERROR: > No times selected > From function reconstructPar > in file reconstructPar.C at line 210. > FOAM exiting b) Failures in 'fireFoam' with messages about non-available 'controlDict' file, which *is* available in lower dir: pk224850@cluster-beta:/work/pk224850/TEMP/OpenFOAM/20160517-TEST/intel_16.0.2.181-openmpi_1.10.2/tutorialsTest/combustion/fireFoam/les/smallPoolFire3D[651]$ find . -iname controlDict ./system/controlDict > --> FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file > 'controlDict' Maybe you could take a look at this and/or report about this is known behaviour? Have a nice day, Paul P.S. no interest in testing newest Intel compilers in our environment? |
|
|
I cannot reproduce the problem you are having running tutorials. We would be happy to test any recent versions of OpenFOAM with any compilers in your environment but this work would need to be funded either as part of a support contract or a more specific arrangement. We could also help with any cases which still do not run on your systems. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2016-04-25 10:57 | paukap | New Issue | |
| 2016-04-25 10:57 | paukap | File Added: 20160425-errors301.tar.gz | |
| 2016-04-25 11:08 | henry | Note Added: 0006171 | |
| 2016-04-25 17:38 | henry | Note Added: 0006173 | |
| 2016-04-25 18:28 | henry | Note Added: 0006174 | |
| 2016-04-26 10:57 | henry | Note Added: 0006177 | |
| 2016-05-03 15:12 | henry | Note Added: 0006222 | |
| 2016-05-03 15:12 | henry | Status | new => resolved |
| 2016-05-03 15:12 | henry | Resolution | open => fixed |
| 2016-05-03 15:12 | henry | Assigned To | => henry |
| 2016-05-13 11:14 | paukap | Note Added: 0006261 | |
| 2016-05-13 11:14 | paukap | Status | resolved => feedback |
| 2016-05-13 11:14 | paukap | Resolution | fixed => reopened |
| 2016-05-13 11:49 | henry | Note Added: 0006262 | |
| 2016-05-13 12:12 | paukap | Note Added: 0006263 | |
| 2016-05-13 12:12 | paukap | Status | feedback => assigned |
| 2016-05-13 12:28 | henry | Note Added: 0006264 | |
| 2016-05-13 12:44 | paukap | Note Added: 0006265 | |
| 2016-05-13 12:46 | paukap | Note Added: 0006266 | |
| 2016-05-13 13:04 | henry | Note Added: 0006267 | |
| 2016-05-13 22:03 | paukap | Note Added: 0006270 | |
| 2016-05-13 22:44 | henry | Note Added: 0006271 | |
| 2016-05-13 22:44 | henry | Status | assigned => closed |
| 2016-05-13 22:44 | henry | Resolution | reopened => no change required |
| 2016-05-18 11:49 | paukap | Note Added: 0006288 | |
| 2016-05-18 11:49 | paukap | Status | closed => feedback |
| 2016-05-18 11:49 | paukap | Resolution | no change required => reopened |
| 2016-05-18 12:12 | henry | Note Added: 0006289 | |
| 2016-05-18 12:13 | henry | Status | feedback => closed |
| 2016-05-18 12:13 | henry | Resolution | reopened => unable to reproduce |
