View Issue Details

IDProjectCategoryView StatusLast Update
0003814OpenFOAMFeaturepublic2022-03-10 09:26
ReporterpetteriAssigned Tohenry 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformGNU/LinuxOSUbuntuOS Version14.04
Product Versiondev 
Fixed in Versiondev 
Summary0003814: Zoltan configuration inconsistency
DescriptionThe file etc/config.sh/zoltan allows for setting the variable ZOLTAN_ARCH_DIR which allone is not sufficient to configure zoltan when a system installation for zoltan is used. The reason for this is that the zoltan library binary is assumed (by files $FOAM_SRC/renumber/Allwmake and $FOAM_SRC/renumber/zoltanRenumber/Make/options) to be located in path:
$FOAM_EXT_LIBBIN/$FOAM_MPI/libzoltan.a

The aforementioned is inconsistent with the metis and scotch dependencies. Scotch and metis binaries and headers are assumed to be found from $?_ARCH_PATH/lib and $?_ARCH_PATH/include, respectively, which allows for a system installation of those libraries without modifying $FOAM_SRC and only setting the ARCH_PATH correctly. Would it be possible to change the zoltan binaries to be used from $ZOLTAN_ARCH_PATH/lib in $FOAM_SRC? Or better yet, allow the user to configure ZOLTAN_LIB_PATH and ZOLTAN_INCLUDE_PATH separately for allowing even more flexibility?
TagsNo tags attached.

Activities

henry

2022-03-07 11:39

manager   ~0012509

I have not attempted to use a system installation for zoltan and don't have one available on my machine. Given that it is a small library try compiling the latest sources is the ThirdParty-dev directory, the download link is in the README. If you have to use the system installation I am not sure if it is in the same place on all machines and have no idea what the paths should be set to in general. But if you can provide a patch to allow users to set the variables however they want and test it on your machine I can include it provided it does not break the current method.

henry

2022-03-07 12:50

manager   ~0012510

Note it is likely that we will replace Zoltan with an alternative library in the future due to the very limited support for weighted redistribution.
Also the native Zoltan renumbering method is rather simplistic and not as good as the Cuthill-McKee method supplied in OpenFOAM.

If you can arrange for a parMeTiS licence that might be a better option if you need weighted balance for chemistry.

petteri

2022-03-07 13:27

reporter   ~0012511

Here's a patch that works on our end and allows us to use system compiled zoltan:

https://diffy.org/diff/e2185b09b5338

Note that in the patch only the OpenFOAM side environment is modified. The patch should allow for the old way of running the ThirdParty-dev/Allwmake but I did not test it. Also, it would be safer to change all occurences of '$FOAM_EXT_LIBBIN/$FOAM_MPI' to $ZOLTAN_LIB_PATH also in ThirdParty-dev/Allwmake to ensure that the paths stay up to date.

The reason we want to use system compiled libraries as much as possible is that we have better control of dependencies on a cluster which has different architectures. We have had strange crashes with mpi if mixed openmpi/scotch/zoltan combinations have been used when certain (older) nodes are involved. We need to be able to compile (and configure) all libraries separately and ensure that they function properly. Since we use many different versions of OpenFOAM, handling all the paths in ThirdParty-directories gets messy and is error prone and those errors typically occur during runtime.

henry

2022-03-07 13:34

manager   ~0012512

> Also, it would be safer to change all occurences of '$FOAM_EXT_LIBBIN/$FOAM_MPI' to $ZOLTAN_LIB_PATH also in ThirdParty-dev/Allwmake to ensure that the paths stay up to date.

This is not clear, could you please make all the changes you need and test to make sure it doesn't break the current operation?

It might be better to wait until after Zoltan is replaced to avoid this maintenance overhead.

petteri

2022-03-07 14:50

reporter   ~0012513

Here's a patch for the ThirdParty-dev/Allwmake:

https://diffy.org/diff/aafcd0c24c9bb

I tested that after rebuilding the latest dev tutorials combustion/reactingFoam/counterFlowFlame2D_GRI and interFoam/RAS/floatingObject run correctly.

henry

2022-03-07 16:42

manager   ~0012514

I don't see how your changes make the handling of Zoltan consistent with Scotch or Metis, you have added

export ZOLTAN_INCLUDE_PATH=$ZOLTAN_ARCH_PATH/include
export ZOLTAN_LIB_PATH=$FOAM_EXT_LIBBIN/$FOAM_MPI

but there is no equivalent for Scotch or Metis. Have you also added SCOTCH_INCLUDE_PATH etc.?

henry

2022-03-07 16:55

manager   ~0012515

The changes you have made are not correct, $FOAM_EXT_LIBBIN/$FOAM_MPI/libzoltan.a is the correct path for Zoltan which is build against a particular version of MPI. ptScotch is handled in the same manner,
see src/parallel/decompose/ptscotchDecomp/Make/options

LIB_LIBS = \
    -L$(SCOTCH_ARCH_PATH)/lib \
    -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) \
    -lptscotch \
    -lptscotcherrexit \
    -lscotch \
    -lrt

where -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) is used to add the path for the -lptscotch -lptscotcherrexit libraries.

henry

2022-03-07 16:58

manager   ~0012516

It would be possible to add a -L$(ZOLTAN_ARCH_PATH)/lib entry which you could use but it is not possible to remove the -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) as this is required by the OpenFOAM installation.

henry

2022-03-07 17:17

manager   ~0012517

This should do what you want without breaking current functionality:

commit 18f4f2f9e724edae4ca1d4711eb6fc37f57035f9 (HEAD -> master, origin/master, origin/HEAD)
Author: Henry Weller <http://cfd.direct>
Date: Mon Mar 7 17:14:34 2022 +0000

    Zoltan: Added alternative library location $ZOLTAN_ARCH_PATH/lib
    
    to support linking to system installations of the Zoltan library.
    
    Resolves the feature request https://bugs.openfoam.org/view.php?id=3814

petteri

2022-03-08 07:10

reporter   ~0012518

Thanks for the update, I think we will manage from now on. Few notes:

> The changes you have made are not correct, $FOAM_EXT_LIBBIN/$FOAM_MPI/libzoltan.a is the correct path for Zoltan which is build against a particular version of MPI

I'm not sure if its necessary to have "correct paths" for the zoltan build (or any library for that matter) because that only enforces the user to compile third party code in a specified (OpenFOAM-dependent) location. If it is necessary for Zoltan and OpenFOAM to use the same version of mpi, having the library binaries in some named location won't enforce this.

The reason why I separated ZOLTAN_INCLUDE_PATH and ZOLTAN_LIBRARY_PATH is because those are the only things required by FOAM_SRC. However, you are correct that separating the two is still incosistent with scotch and metis. Having the first part of $ZOLTAN_ARCH_PATH/lib is good and allows for ZOLTAN_ARCH_PATH to point somewhere completely independent of OpenFOAM. However, I'm not sure what is the benefit of having the 'lib' there. User may equally well want to store the binaries in some system/lib64, system/WHATEVER. But like I said this works for us.

Thanks again for the quick patch!

Issue History

Date Modified Username Field Change
2022-03-07 11:24 petteri New Issue
2022-03-07 11:39 henry Note Added: 0012509
2022-03-07 12:50 henry Note Added: 0012510
2022-03-07 13:27 petteri Note Added: 0012511
2022-03-07 13:34 henry Note Added: 0012512
2022-03-07 14:50 petteri Note Added: 0012513
2022-03-07 16:42 henry Note Added: 0012514
2022-03-07 16:55 henry Note Added: 0012515
2022-03-07 16:58 henry Note Added: 0012516
2022-03-07 17:17 henry Note Added: 0012517
2022-03-08 07:10 petteri Note Added: 0012518
2022-03-08 08:37 henry Assigned To => henry
2022-03-08 08:37 henry Status new => resolved
2022-03-08 08:37 henry Resolution open => fixed
2022-03-08 08:37 henry Fixed in Version => dev