View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002275 | OpenFOAM | Bug | public | 2016-09-28 18:30 | 2016-10-13 11:28 |
| Reporter | jfp6 | Assigned To | henry | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Platform | GNU/Linux | OS | Ubuntu | OS Version | 16.04 |
| Product Version | dev | ||||
| Summary | 0002275: scotch no longer available for decomposition when running snappyHexMesh in parallel | ||||
| Description | scotch is a valid decomposition method when decomposing a case, but is not a valid method when running snappyHexMesh in parallel. Then, it must be ptscotch instead of scotch. If there is a reason for this apparent discrepancy, that is fine, but I am inquiring because it requires setting scotch to decompose and then switching to ptscotch to run snappyHexMesh in parallel. | ||||
| Steps To Reproduce | This error can be duplicated by running the Allrun script in /tutorials/mesh/snappyHexMesh/motorBike after switching the decomposition method to scotch in decomposePar. | ||||
| Tags | No tags attached. | ||||
|
|
Technically, "ptscotch" is a parallel implementation of "scotch", which should only be used when we already have the case decomposed in parallel and need to rebalance it. "ptscotch" is specially another library from Scotch, which is why a one-to-one option is given. Nonetheless, there was a change done in the code sometime ago, which allows using the same coeff-block in "decomposeParDict"... Although... from your Summary "scotch no longer available for decomposition when running snappyHexMesh in parallel", what was the last version of OpenFOAM in which you observed this? |
|
|
Thank you for your insight. In version 3.0.x, you could select scotch for decomposition and it also worked with snappyHexMesh(SHM) in parallel. Perhaps this was incorrect behavior that a selection of scotch worked for a parallel SHM run or maybe the code knew to switch to ptscotch when doing rebalancing. It was convenient from a scripting perspective because you didn't have to change decomposePar files. |
|
|
Many thanks for pointing out the last known version! Sorry, I didn't remember about this feature. I do remember that this was addressed a few years ago, for that exact reason: making scripting easier. I've tested with OpenFOAM 3.0.x, 4.x and dev: - 3.0.x - "scotch" is automatically interpreted as "ptscotch" - 4.x - "scotch" is invalid - dev - "scotch" is invalid I still have to check where this was changed... |
|
|
I've done some digging around in the Git histories of two public repositories and I've isolated the reason for this situation. So let me start by outlining what I've found: - In OpenFOAM-dev, this change occurred in commit 1441f8cab064, in the file "src/parallel/decompose/decompositionMethods/decompositionMethod/decompositionMethod.C": https://github.com/OpenFOAM/OpenFOAM-dev/commit/1441f8cab064a79b65e0dc7c80e86b8c45dc2f80#diff-d08c779b59aa77c6172a011c50b5a36dL56 - Since the comment in the commit wasn't clear, I went checking the OpenFOAM-history repository and there I managed to find that this change was actually done because the "scotch" library can coexist with the "ptscotch" library, as long as Scotch 6.0.0 or newer is used. This change was done in two commits: - https://github.com/OpenCFD/OpenFOAM-history/commit/c1457b633e0d555e13d9ee6747d81a7be8be5138 - where the same change was done, namely where the automatic renaming of "scotch" to "ptscotch" was removed from "src/parallel/decompose/decompositionMethods/decompositionMethod/decompositionMethod.C" - https://github.com/OpenCFD/OpenFOAM-history/commit/264307f102d0db1b379af3e2f6d6787864f8aff1 - where "scotch" was added to snappyHexMesh's library dependency. This last commit is missing from OpenFOAM 4.x and OpenFOAM-dev. As for why it's missing, all signs point to a fairly simple reason: the Deb packages of OpenFOAM on Ubuntu rely on using the system's Scotch version 5.1.0... which is incompatible with this feature of having simultaneous use of "scotch" and "ptscotch" when running in parallel :( Therefore, the conclusion so far is that the automatic renaming of "scotch" to "ptscotch" has been dropped, because as of Scotch 6.0, we can also use the Scotch's serial-mode library for when running in parallel. Problem is that I have no idea as to why this might be a good idea, unless it's meant to allow for a complete re-decomposition, given that the parallel implementation may be bias to an already existing decomposition. @MattijsJ @henry: Any chance either one of you can give us some feedback on this change? Is using the serial-mode Scotch library any good in a parallel redistribution, that justifies this just? I know it makes it more consistent with all other decomposition methods... @jfp6: Since this looks like a feature that is here to stay, you can rely on the file inclusion methodology that OpenFOAM has got, namely that you can include other files. Then you can modify a single smaller file that only has the method that you want to use and which the script can easily replace with either a copy command or direct creation of the file. Rough outline: In "decomposeParDict": #include "decompositionMethodDict" In "decompositionMethodDict": method scotch; In the script: echo "method scotch;" > system/decompositionMethodDict #... echo "method ptscotch;" > system/decompositionMethodDict |
|
|
Thank you for the clarification and recommendation. I am happy to modify my scripts; I just wanted to verify that this was not a bug that should be corrected. |
|
|
I think the dependency to scotch should just be added to snappyHexMesh. 2) or in your system/controlDict just add the library yourself: libs ("libscotchDecomp.so"); |
|
|
That is a clean approach. |
|
|
> Deb packages of OpenFOAM on Ubuntu rely on using the system's Scotch version 5.1.0... which is incompatible with this feature of having simultaneous use of "scotch" and "ptscotch" when running in parallel Correct, we can resolve this issue when the maintainers of scotch release packs for version 6.0. |
|
|
> Correct, we can resolve this issue when the maintainers of scotch release packs for version 6.0. That's going to be a bit of a problem... the last update to the Deb packages for Scotch in Debian were done in 2013; the bug tracker refers to having to use at least Scotch 6.0.1 to not have issues with MUMPS: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=731965 - but there have been no more signs of activity since then :( Well, at least openSUSE has got Scotch 6.0.4: https://build.opensuse.org/package/revisions/science/scotch Any chance we can use a hack in "Allwmake" and "Make/options" scripts for "snappyHexMesh" (and possibly at "etc/config.*/scotch"), so that this can be used on any other systems, except in Ubuntu/Debian? The original report does refer to OpenFOAM-dev. |
|
|
I don't like the idea that OpenFOAM should operate differently on different GNU/Linux distributions. What is wrong with Mattijs' suggestion, at least until the maintainers of scotch release an updated version for Ubuntu? |
|
|
I see what you mean: less maintenance and more consistency. Then there are only two intermediate fixes missing for this bug report: 1- For this to be at least disclaimed in the release notes, either to update 4.0 or add it in the next release notes for 4.1. 2- To add this to the development roadmap (if not there already), namely to create a dependence on the upstream packages of Scotch for Debian/Ubuntu. |
|
|
1. I am not sure what you mean; what do you want "disclaimed" in the release notes? 2. The Ubuntu packs already have a dependence on the upstream packages of Scotch, basically that is the problem because the package provided by the maintainers is out of date. |
|
|
1. Sorry, "disclaimed" wasn't the best word for this situation, but I couldn't think of another. The following should be indicated in the release notes: - In versions prior to 4.0, using "scotch" in "decomposeParDict" when running "snappyHexMesh" in parallel, would switch automatically to "ptscotch". - Introduced in commit 1441f8cab064 is the requirement to use "ptscotch" explicitly when needed; or as an alternative, to load "libscotch.so" via "libs" in "controlDict", in order to use the main Scotch library, instead of PTScotch, with the limitation that Scotch 6.0 or newer is needed. 2. I was referring to was to keep a note somewhere that later on this needs to be updated, namely the dependency "-lscotch" in "snappyHexMesh/Make/options", once the upstream Scotch Deb packages are upgraded. I'm point this out also due to a similar example that happened a couple of days ago, namely in issue #2280, where the somewhat recent upgrade to depend on C++11, didn't account for other features that were already known that could be upgraded as well. We could start using the bug tracker for keeping track of these (somewhat) known limitations, but the downside is that it would potentially get mixed up with the other bug reports :( Using a public development Roadmap would be best, but I expect this requires some additional funding to do so and maintain. |
|
|
Resolved in OpenFOAM-4.x/1 by commit da3fc4f29d758ae25e85d74350fda6679a458ad9 Resolved in OpenFOAM-dev by commit b6e740bd7497dd76b84359bba2dad5ab92f6dd7d See also http://openfoam.org/download/dev-ubuntu-scotch-6/ http://openfoam.org/version/4-1/ |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2016-09-28 18:30 | jfp6 | New Issue | |
| 2016-09-28 18:54 | wyldckat | Note Added: 0006933 | |
| 2016-09-28 19:52 | jfp6 | Note Added: 0006936 | |
| 2016-09-28 20:32 | wyldckat | Note Added: 0006937 | |
| 2016-09-28 21:18 | wyldckat | Note Added: 0006938 | |
| 2016-09-28 21:29 | jfp6 | Note Added: 0006939 | |
| 2016-09-29 09:02 | MattijsJ | Note Added: 0006940 | |
| 2016-09-29 15:19 | jfp6 | Note Added: 0006943 | |
| 2016-09-30 18:54 | henry | Note Added: 0006949 | |
| 2016-10-05 23:04 | wyldckat | Note Added: 0006969 | |
| 2016-10-05 23:21 | henry | Note Added: 0006971 | |
| 2016-10-06 11:09 | wyldckat | Note Added: 0006973 | |
| 2016-10-06 11:41 | henry | Note Added: 0006974 | |
| 2016-10-06 13:06 | wyldckat | Note Added: 0006975 | |
| 2016-10-13 11:28 | henry | Assigned To | => henry |
| 2016-10-13 11:28 | henry | Status | new => resolved |
| 2016-10-13 11:28 | henry | Resolution | open => fixed |
| 2016-10-13 11:28 | henry | Note Added: 0007006 |
