View Issue Details

IDProjectCategoryView StatusLast Update
0002275OpenFOAM[All Projects] Bugpublic2016-10-13 11:28
Reporterjfp6Assigned Tohenry 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformGNU/LinuxOSUbuntuOS Version16.04
Product Versiondev 
Fixed in Version 
Summary0002275: scotch no longer available for decomposition when running snappyHexMesh in parallel
Descriptionscotch is a valid decomposition method when decomposing a case, but is not a valid method when running snappyHexMesh in parallel. Then, it must be ptscotch instead of scotch. If there is a reason for this apparent discrepancy, that is fine, but I am inquiring because it requires setting scotch to decompose and then switching to ptscotch to run snappyHexMesh in parallel.
Steps To ReproduceThis error can be duplicated by running the Allrun script in /tutorials/mesh/snappyHexMesh/motorBike after switching the decomposition method to scotch in decomposePar.
TagsNo tags attached.

Activities

wyldckat

2016-09-28 18:54

updater   ~0006933

Technically, "ptscotch" is a parallel implementation of "scotch", which should only be used when we already have the case decomposed in parallel and need to rebalance it.
"ptscotch" is specially another library from Scotch, which is why a one-to-one option is given.

Nonetheless, there was a change done in the code sometime ago, which allows using the same coeff-block in "decomposeParDict"...

Although... from your Summary "scotch no longer available for decomposition when running snappyHexMesh in parallel", what was the last version of OpenFOAM in which you observed this?

jfp6

2016-09-28 19:52

reporter   ~0006936

Thank you for your insight. In version 3.0.x, you could select scotch for decomposition and it also worked with snappyHexMesh(SHM) in parallel. Perhaps this was incorrect behavior that a selection of scotch worked for a parallel SHM run or maybe the code knew to switch to ptscotch when doing rebalancing. It was convenient from a scripting perspective because you didn't have to change decomposePar files.

wyldckat

2016-09-28 20:32

updater   ~0006937

Many thanks for pointing out the last known version! Sorry, I didn't remember about this feature. I do remember that this was addressed a few years ago, for that exact reason: making scripting easier.

I've tested with OpenFOAM 3.0.x, 4.x and dev:

 - 3.0.x - "scotch" is automatically interpreted as "ptscotch"
 - 4.x - "scotch" is invalid
 - dev - "scotch" is invalid

I still have to check where this was changed...

wyldckat

2016-09-28 21:18

updater   ~0006938

I've done some digging around in the Git histories of two public repositories and I've isolated the reason for this situation. So let me start by outlining what I've found:

 - In OpenFOAM-dev, this change occurred in commit 1441f8cab064, in the file "src/parallel/decompose/decompositionMethods/decompositionMethod/decompositionMethod.C": https://github.com/OpenFOAM/OpenFOAM-dev/commit/1441f8cab064a79b65e0dc7c80e86b8c45dc2f80#diff-d08c779b59aa77c6172a011c50b5a36dL56

 - Since the comment in the commit wasn't clear, I went checking the OpenFOAM-history repository and there I managed to find that this change was actually done because the "scotch" library can coexist with the "ptscotch" library, as long as Scotch 6.0.0 or newer is used. This change was done in two commits:

   - https://github.com/OpenCFD/OpenFOAM-history/commit/c1457b633e0d555e13d9ee6747d81a7be8be5138 - where the same change was done, namely where the automatic renaming of "scotch" to "ptscotch" was removed from "src/parallel/decompose/decompositionMethods/decompositionMethod/decompositionMethod.C"

   - https://github.com/OpenCFD/OpenFOAM-history/commit/264307f102d0db1b379af3e2f6d6787864f8aff1 - where "scotch" was added to snappyHexMesh's library dependency.


This last commit is missing from OpenFOAM 4.x and OpenFOAM-dev. As for why it's missing, all signs point to a fairly simple reason: the Deb packages of OpenFOAM on Ubuntu rely on using the system's Scotch version 5.1.0... which is incompatible with this feature of having simultaneous use of "scotch" and "ptscotch" when running in parallel :(


Therefore, the conclusion so far is that the automatic renaming of "scotch" to "ptscotch" has been dropped, because as of Scotch 6.0, we can also use the Scotch's serial-mode library for when running in parallel. Problem is that I have no idea as to why this might be a good idea, unless it's meant to allow for a complete re-decomposition, given that the parallel implementation may be bias to an already existing decomposition.


@MattijsJ @henry: Any chance either one of you can give us some feedback on this change? Is using the serial-mode Scotch library any good in a parallel redistribution, that justifies this just?
I know it makes it more consistent with all other decomposition methods...


@jfp6: Since this looks like a feature that is here to stay, you can rely on the file inclusion methodology that OpenFOAM has got, namely that you can include other files. Then you can modify a single smaller file that only has the method that you want to use and which the script can easily replace with either a copy command or direct creation of the file.

Rough outline:

  In "decomposeParDict":

    #include "decompositionMethodDict"

  In "decompositionMethodDict":

    method scotch;

  In the script:

    echo "method scotch;" > system/decompositionMethodDict
    #...
    echo "method ptscotch;" > system/decompositionMethodDict

jfp6

2016-09-28 21:29

reporter   ~0006939

Thank you for the clarification and recommendation. I am happy to modify my scripts; I just wanted to verify that this was not a bug that should be corrected.

MattijsJ

2016-09-29 09:02

reporter   ~0006940

I think the dependency to scotch should just be added to snappyHexMesh.

2) or in your system/controlDict just add the library yourself:

libs ("libscotchDecomp.so");

jfp6

2016-09-29 15:19

reporter   ~0006943

That is a clean approach.

henry

2016-09-30 18:54

manager   ~0006949

> Deb packages of OpenFOAM on Ubuntu rely on using the system's Scotch version 5.1.0... which is incompatible with this feature of having simultaneous use of "scotch" and "ptscotch" when running in parallel

Correct, we can resolve this issue when the maintainers of scotch release packs for version 6.0.

wyldckat

2016-10-05 23:04

updater   ~0006969

> Correct, we can resolve this issue when the maintainers of scotch release packs for version 6.0.

That's going to be a bit of a problem... the last update to the Deb packages for Scotch in Debian were done in 2013; the bug tracker refers to having to use at least Scotch 6.0.1 to not have issues with MUMPS: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731965 - but there have been no more signs of activity since then :(

Well, at least openSUSE has got Scotch 6.0.4: https://build.opensuse.org/package/revisions/science/scotch


Any chance we can use a hack in "Allwmake" and "Make/options" scripts for "snappyHexMesh" (and possibly at "etc/config.*/scotch"), so that this can be used on any other systems, except in Ubuntu/Debian? The original report does refer to OpenFOAM-dev.

henry

2016-10-05 23:21

manager   ~0006971

I don't like the idea that OpenFOAM should operate differently on different GNU/Linux distributions. What is wrong with Mattijs' suggestion, at least until the maintainers of scotch release an updated version for Ubuntu?

wyldckat

2016-10-06 11:09

updater   ~0006973

I see what you mean: less maintenance and more consistency.
Then there are only two intermediate fixes missing for this bug report:

  1- For this to be at least disclaimed in the release notes, either to update 4.0 or add it in the next release notes for 4.1.

  2- To add this to the development roadmap (if not there already), namely to create a dependence on the upstream packages of Scotch for Debian/Ubuntu.

henry

2016-10-06 11:41

manager   ~0006974

1. I am not sure what you mean; what do you want "disclaimed" in the release notes?

2. The Ubuntu packs already have a dependence on the upstream packages of Scotch, basically that is the problem because the package provided by the maintainers is out of date.

wyldckat

2016-10-06 13:06

updater   ~0006975

1. Sorry, "disclaimed" wasn't the best word for this situation, but I couldn't think of another. The following should be indicated in the release notes:

 - In versions prior to 4.0, using "scotch" in "decomposeParDict" when running "snappyHexMesh" in parallel, would switch automatically to "ptscotch".
 - Introduced in commit 1441f8cab064 is the requirement to use "ptscotch" explicitly when needed; or as an alternative, to load "libscotch.so" via "libs" in "controlDict", in order to use the main Scotch library, instead of PTScotch, with the limitation that Scotch 6.0 or newer is needed.


2. I was referring to was to keep a note somewhere that later on this needs to be updated, namely the dependency "-lscotch" in "snappyHexMesh/Make/options", once the upstream Scotch Deb packages are upgraded.
I'm point this out also due to a similar example that happened a couple of days ago, namely in issue #2280, where the somewhat recent upgrade to depend on C++11, didn't account for other features that were already known that could be upgraded as well.

We could start using the bug tracker for keeping track of these (somewhat) known limitations, but the downside is that it would potentially get mixed up with the other bug reports :(
Using a public development Roadmap would be best, but I expect this requires some additional funding to do so and maintain.

henry

2016-10-13 11:28

manager   ~0007006

Resolved in OpenFOAM-4.x/1 by commit da3fc4f29d758ae25e85d74350fda6679a458ad9

Resolved in OpenFOAM-dev by commit b6e740bd7497dd76b84359bba2dad5ab92f6dd7d

See also
    http://openfoam.org/download/dev-ubuntu-scotch-6/
    http://openfoam.org/version/4-1/

Issue History

Date Modified Username Field Change
2016-09-28 18:30 jfp6 New Issue
2016-09-28 18:54 wyldckat Note Added: 0006933
2016-09-28 19:52 jfp6 Note Added: 0006936
2016-09-28 20:32 wyldckat Note Added: 0006937
2016-09-28 21:18 wyldckat Note Added: 0006938
2016-09-28 21:29 jfp6 Note Added: 0006939
2016-09-29 09:02 MattijsJ Note Added: 0006940
2016-09-29 15:19 jfp6 Note Added: 0006943
2016-09-30 18:54 henry Note Added: 0006949
2016-10-05 23:04 wyldckat Note Added: 0006969
2016-10-05 23:21 henry Note Added: 0006971
2016-10-06 11:09 wyldckat Note Added: 0006973
2016-10-06 11:41 henry Note Added: 0006974
2016-10-06 13:06 wyldckat Note Added: 0006975
2016-10-13 11:28 henry Assigned To => henry
2016-10-13 11:28 henry Status new => resolved
2016-10-13 11:28 henry Resolution open => fixed
2016-10-13 11:28 henry Note Added: 0007006