View Issue Details

IDProjectCategoryView StatusLast Update
0003300OpenFOAMFeaturepublic2019-07-12 12:49
ReporterpaulmedwardsAssigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
PlatformGNU/LinuxOSOtherOS Version(please specify)
Product Versiondev 
Fixed in Version 
Summary0003300: Allow args to be passed to mpirun with runParallel
DescriptionI would like to pass arguments to mpirun (e.g. setting processes per node and hostfile etc.) with runParallel. I can't see a way to do it other than edit the file myself or rewrite the command using mpirun in the Allrun scripts.
Additional InformationHere is a possible patch:

From d1c1eb24207c801cc59ae506c4a56cd0ade6c6f6 Mon Sep 17 00:00:00 2001
From: edwardsp <paul.m.edwards@gmail.com>
Date: Mon, 1 Jul 2019 09:40:19 +0100
Subject: [PATCH] Add mpirun args from environment with runParallel

---
 bin/tools/RunFunctions | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bin/tools/RunFunctions b/bin/tools/RunFunctions
index 02c1b174cb..8376be8278 100644
--- a/bin/tools/RunFunctions
+++ b/bin/tools/RunFunctions
@@ -139,9 +139,9 @@ runParallel()
     else
         echo "Running $APP_RUN in parallel on $PWD using $nProcs processes"
         if [ "$LOG_APPEND" = "true" ]; then
- ( mpirun -np $nProcs $APP_RUN -parallel "$@" < /dev/null >> log.$LOG_SUFFIX 2>&1 )
+ ( mpirun $runParallelMpirunArgs -np $nProcs $APP_RUN -parallel "$@" < /dev/null >> log.$LOG_SUFFIX 2>&1 )
         else
- ( mpirun -np $nProcs $APP_RUN -parallel "$@" < /dev/null > log.$LOG_SUFFIX 2>&1 )
+ ( mpirun $runParallelMpirunArgs -np $nProcs $APP_RUN -parallel "$@" < /dev/null > log.$LOG_SUFFIX 2>&1 )
         fi
     fi
 }
TagsNo tags attached.

Activities

henry

2019-07-02 09:30

manager   ~0010540

We are trying to reduce the environment clutter OpenFOAM needs and generates. Wouldn't it be better to pass the additional information as arguments?
Alternatively you could write a version of runParallel which handles the additional information.

Note that runParallel was designed to run the tutorial cases and similar, we didn't intend it to be a completely general purpose way of launching OpenFOAM jobs.
You could try foamJob as a more general approach to OpenFOAM job launch and management but you would need to enhance the argument handling to pass additional mpirun arguments.

paulmedwards

2019-07-02 09:50

reporter   ~0010542

I could create a patch to add an argument to runParallel - how about "--mpirun-args <MPIRUN_ARGS>"?

I am compiling this for Azure and wanted an easy way to run the tutorials to demonstrate OpenFOAM.

henry

2019-07-02 09:59

manager   ~0010544

runParallel wasn't really designed to be used in this way, it would be better to extend the foamJob script which is more flexible than first sourcing $WM_PROJECT_DIR/bin/tools/RunFunctions and using the functions.
foamJob is very old and not well maintained and could probably do with a rewrite.

wyldckat

2019-07-05 19:16

updater   ~0010552

I've been thinking of rewriting foamJob, but in a way that would depend on the RunFunctions script, in order to have things consistent and simpler to maintain.

Furthermore, I found out recently that with SLURM (the most common/open-source job scheduler being used nowadays), I can use 'Allrun' scripts without problems, because Open-MPI will pick up on the nodes+cores to be used for a simulation, given that SLURM defines the necessary environment variables, therefore not requiring any additional+manually added arguments for 'mpirun'.

@paulmedwards: On Azure, any chance people can use a similar system to SLURM or SLURM itself? That way people can even schedule their jobs and not have to worry about additional MPI settings. And get emails when the job is launched/done as well ;)

paulmedwards

2019-07-12 12:49

reporter   ~0010557

@wyldckat: Thanks - and SLURM is definitely an option on Azure. I was actually using PBS pro but I do need to add to mpirun parameters there.

Issue History

Date Modified Username Field Change
2019-07-02 08:15 paulmedwards New Issue
2019-07-02 09:30 henry Note Added: 0010540
2019-07-02 09:50 paulmedwards Note Added: 0010542
2019-07-02 09:59 henry Note Added: 0010544
2019-07-05 19:16 wyldckat Note Added: 0010552
2019-07-12 12:49 paulmedwards Note Added: 0010557