View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0003814 | OpenFOAM | Feature | public | 2022-03-07 11:24 | 2022-03-10 09:26 |
| Reporter | petteri | Assigned To | henry | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Platform | GNU/Linux | OS | Ubuntu | OS Version | 14.04 |
| Product Version | dev | ||||
| Fixed in Version | dev | ||||
| Summary | 0003814: Zoltan configuration inconsistency | ||||
| Description | The file etc/config.sh/zoltan allows for setting the variable ZOLTAN_ARCH_DIR which allone is not sufficient to configure zoltan when a system installation for zoltan is used. The reason for this is that the zoltan library binary is assumed (by files $FOAM_SRC/renumber/Allwmake and $FOAM_SRC/renumber/zoltanRenumber/Make/options) to be located in path: $FOAM_EXT_LIBBIN/$FOAM_MPI/libzoltan.a The aforementioned is inconsistent with the metis and scotch dependencies. Scotch and metis binaries and headers are assumed to be found from $?_ARCH_PATH/lib and $?_ARCH_PATH/include, respectively, which allows for a system installation of those libraries without modifying $FOAM_SRC and only setting the ARCH_PATH correctly. Would it be possible to change the zoltan binaries to be used from $ZOLTAN_ARCH_PATH/lib in $FOAM_SRC? Or better yet, allow the user to configure ZOLTAN_LIB_PATH and ZOLTAN_INCLUDE_PATH separately for allowing even more flexibility? | ||||
| Tags | No tags attached. | ||||
|
|
I have not attempted to use a system installation for zoltan and don't have one available on my machine. Given that it is a small library try compiling the latest sources is the ThirdParty-dev directory, the download link is in the README. If you have to use the system installation I am not sure if it is in the same place on all machines and have no idea what the paths should be set to in general. But if you can provide a patch to allow users to set the variables however they want and test it on your machine I can include it provided it does not break the current method. |
|
|
Note it is likely that we will replace Zoltan with an alternative library in the future due to the very limited support for weighted redistribution. Also the native Zoltan renumbering method is rather simplistic and not as good as the Cuthill-McKee method supplied in OpenFOAM. If you can arrange for a parMeTiS licence that might be a better option if you need weighted balance for chemistry. |
|
|
Here's a patch that works on our end and allows us to use system compiled zoltan: https://diffy.org/diff/e2185b09b5338 Note that in the patch only the OpenFOAM side environment is modified. The patch should allow for the old way of running the ThirdParty-dev/Allwmake but I did not test it. Also, it would be safer to change all occurences of '$FOAM_EXT_LIBBIN/$FOAM_MPI' to $ZOLTAN_LIB_PATH also in ThirdParty-dev/Allwmake to ensure that the paths stay up to date. The reason we want to use system compiled libraries as much as possible is that we have better control of dependencies on a cluster which has different architectures. We have had strange crashes with mpi if mixed openmpi/scotch/zoltan combinations have been used when certain (older) nodes are involved. We need to be able to compile (and configure) all libraries separately and ensure that they function properly. Since we use many different versions of OpenFOAM, handling all the paths in ThirdParty-directories gets messy and is error prone and those errors typically occur during runtime. |
|
|
> Also, it would be safer to change all occurences of '$FOAM_EXT_LIBBIN/$FOAM_MPI' to $ZOLTAN_LIB_PATH also in ThirdParty-dev/Allwmake to ensure that the paths stay up to date. This is not clear, could you please make all the changes you need and test to make sure it doesn't break the current operation? It might be better to wait until after Zoltan is replaced to avoid this maintenance overhead. |
|
|
Here's a patch for the ThirdParty-dev/Allwmake: https://diffy.org/diff/aafcd0c24c9bb I tested that after rebuilding the latest dev tutorials combustion/reactingFoam/counterFlowFlame2D_GRI and interFoam/RAS/floatingObject run correctly. |
|
|
I don't see how your changes make the handling of Zoltan consistent with Scotch or Metis, you have added export ZOLTAN_INCLUDE_PATH=$ZOLTAN_ARCH_PATH/include export ZOLTAN_LIB_PATH=$FOAM_EXT_LIBBIN/$FOAM_MPI but there is no equivalent for Scotch or Metis. Have you also added SCOTCH_INCLUDE_PATH etc.? |
|
|
The changes you have made are not correct, $FOAM_EXT_LIBBIN/$FOAM_MPI/libzoltan.a is the correct path for Zoltan which is build against a particular version of MPI. ptScotch is handled in the same manner, see src/parallel/decompose/ptscotchDecomp/Make/options LIB_LIBS = \ -L$(SCOTCH_ARCH_PATH)/lib \ -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) \ -lptscotch \ -lptscotcherrexit \ -lscotch \ -lrt where -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) is used to add the path for the -lptscotch -lptscotcherrexit libraries. |
|
|
It would be possible to add a -L$(ZOLTAN_ARCH_PATH)/lib entry which you could use but it is not possible to remove the -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) as this is required by the OpenFOAM installation. |
|
|
This should do what you want without breaking current functionality: commit 18f4f2f9e724edae4ca1d4711eb6fc37f57035f9 (HEAD -> master, origin/master, origin/HEAD) Author: Henry Weller <http://cfd.direct> Date: Mon Mar 7 17:14:34 2022 +0000 Zoltan: Added alternative library location $ZOLTAN_ARCH_PATH/lib to support linking to system installations of the Zoltan library. Resolves the feature request https://bugs.openfoam.org/view.php?id=3814 |
|
|
Thanks for the update, I think we will manage from now on. Few notes: > The changes you have made are not correct, $FOAM_EXT_LIBBIN/$FOAM_MPI/libzoltan.a is the correct path for Zoltan which is build against a particular version of MPI I'm not sure if its necessary to have "correct paths" for the zoltan build (or any library for that matter) because that only enforces the user to compile third party code in a specified (OpenFOAM-dependent) location. If it is necessary for Zoltan and OpenFOAM to use the same version of mpi, having the library binaries in some named location won't enforce this. The reason why I separated ZOLTAN_INCLUDE_PATH and ZOLTAN_LIBRARY_PATH is because those are the only things required by FOAM_SRC. However, you are correct that separating the two is still incosistent with scotch and metis. Having the first part of $ZOLTAN_ARCH_PATH/lib is good and allows for ZOLTAN_ARCH_PATH to point somewhere completely independent of OpenFOAM. However, I'm not sure what is the benefit of having the 'lib' there. User may equally well want to store the binaries in some system/lib64, system/WHATEVER. But like I said this works for us. Thanks again for the quick patch! |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2022-03-07 11:24 | petteri | New Issue | |
| 2022-03-07 11:39 | henry | Note Added: 0012509 | |
| 2022-03-07 12:50 | henry | Note Added: 0012510 | |
| 2022-03-07 13:27 | petteri | Note Added: 0012511 | |
| 2022-03-07 13:34 | henry | Note Added: 0012512 | |
| 2022-03-07 14:50 | petteri | Note Added: 0012513 | |
| 2022-03-07 16:42 | henry | Note Added: 0012514 | |
| 2022-03-07 16:55 | henry | Note Added: 0012515 | |
| 2022-03-07 16:58 | henry | Note Added: 0012516 | |
| 2022-03-07 17:17 | henry | Note Added: 0012517 | |
| 2022-03-08 07:10 | petteri | Note Added: 0012518 | |
| 2022-03-08 08:37 | henry | Assigned To | => henry |
| 2022-03-08 08:37 | henry | Status | new => resolved |
| 2022-03-08 08:37 | henry | Resolution | open => fixed |
| 2022-03-08 08:37 | henry | Fixed in Version | => dev |
