View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0000231 | OpenFOAM | Bug | public | 2011-06-28 22:50 | 2015-02-06 10:17 |
Reporter | wyldckat | Assigned To | henry | ||
Priority | normal | Severity | minor | Reproducibility | always |
Status | resolved | Resolution | fixed | ||
Summary | 0000231: foamJob doesn't take advantage of sending FOAM_SETTINGS to foamExec | ||||
Description | When using foamJob, the very useful "FOAM_SETTINGS" environment variable is not visible to "foamExec" on the remote nodes. For it to be visible, "mpirun" must have the additional option "-x FOAM_SETTINGS". | ||||
Steps To Reproduce | On a multi-architecture environment of OpenFOAM builds: * Start the OpenFOAM environment with a dependency on a special "prefs-debug.sh" environment script or direct option definitions, such as "WM_ARCH_OPTION=32" and/or "WM_PRECISION_OPTION=SP", which aren't the default parameters. In other words, each new terminal has a clean environment and by calling the desired alias, the respective environment is activated... something like this: alias of20x-dbg32SP='source $HOME/OpenFOAM/OpenFOAM-2.0.x/etc/bashrc etc/prefs-debugSP.sh WM_ARCH_OPTION=32' * Then run in this environment a parallel case using "foamJob". On the remote processes, no OpenFOAM environment is started by default because "~/.bashrc" only has aliases. So, only "foamJob" relies 100% on "foamExec". "mpirun" is not told to send the variable FOAM_SETTINGS to the remote processes, therefore they will use the default options and/or "prefs.sh" options available at select folders. * Given this set-up, the parallel case might not even run, since the default options might not be applicable in the remote machines... | ||||
Additional Information | Attached is the patch for fixing this: patch4foamJob_send_settings_var.gz | ||||
Tags | No tags attached. | ||||
|
|
|
|
|
I'll use this report for a request to add a "-wait" argument to the "foamJob" script. The patch is attached with the name: patch4foamJob_wait_in_parallel.gz This is because sometimes it's useful to wait for a parallel run to complete, without having to see the log on screen or redirected elsewhere... |
|
patch_aliases_refresh (864 bytes)
diff --git a/etc/config/aliases.csh b/etc/config/aliases.csh index 6573c91..82c3e05 100644 --- a/etc/config/aliases.csh +++ b/etc/config/aliases.csh @@ -38,6 +38,9 @@ alias wm32 'wmSET WM_ARCH_OPTION=32' alias wmSP 'wmSET WM_PRECISION_OPTION=SP' alias wmDP 'wmSET WM_PRECISION_OPTION=DP' +# refresh the environment +alias wmREFRESH 'wmSET $FOAM_SETTINGS' + # clear env alias wmUNSET='source $WM_PROJECT_DIR/etc/config/unset.csh' diff --git a/etc/config/aliases.sh b/etc/config/aliases.sh index 1881597..011958a 100644 --- a/etc/config/aliases.sh +++ b/etc/config/aliases.sh @@ -38,6 +38,9 @@ alias wm32='wmSET WM_ARCH_OPTION=32' alias wmSP='wmSET WM_PRECISION_OPTION=SP' alias wmDP='wmSET WM_PRECISION_OPTION=DP' +# refresh the environment +alias wmREFRESH='wmSET $FOAM_SETTINGS' + # clear env alias wmUNSET='. $WM_PROJECT_DIR/etc/config/unset.sh' |
|
patch4foamJob_send_settings_var (376 bytes)
diff --git a/bin/foamJob b/bin/foamJob index 804976e..f487e2e 100755 --- a/bin/foamJob +++ b/bin/foamJob @@ -214,6 +214,12 @@ then break fi done + + # + # Send FOAM_SETTINGS to parallel processes. so the proper definitions are sent as well. + # + mpiopts="$mpiopts -x FOAM_SETTINGS" + ;; esac |
|
patch4foamJob_wait_in_parallel (1,314 bytes)
diff --git a/bin/foamJob b/bin/foamJob index 70e2c39..f487e2e 100755 --- a/bin/foamJob +++ b/bin/foamJob @@ -40,6 +40,7 @@ options: -case <dir> specify alternative case directory, default is the cwd -parallel parallel run of processors -screen also sends output to screen + -wait wait for execution to complete -version <ver> specify an alternative OpenFOAM version -help print the usage @@ -86,7 +87,7 @@ findExec() { # MAIN SCRIPT #~~~~~~~~~~~~ -unset parallelOpt screenOpt +unset parallelOpt screenOpt waitOpt # parse options @@ -109,6 +110,10 @@ do screenOpt=true shift ;; + -w | -wait) + waitOpt=true + shift + ;; -v | -version) [ "$#" -ge 2 ] || usage "'$1' option requires an argument" version="$2" @@ -222,6 +233,8 @@ then else echo "Executing: $mpirun $mpiopts $APPLICATION $@ -parallel > log 2>&1" $mpirun $mpiopts $APPLICATION $@ -parallel > log 2>&1 & + pid=$! + if [ "$waitOpt" = true ]; then wait $pid; fi fi else @@ -236,6 +249,8 @@ else else echo "Executing: $APPLICATION $@ > log 2>&1 &" $APPLICATION $@ > log 2>&1 & + pid=$! + if [ "$waitOpt" = true ]; then wait $pid; fi fi fi |
|
I re-uploaded the 2 initial patch files as normal text files, since this way it's easier to analyse them directly. I uploaded another patch named "patch_aliases_refresh" for adding the alias wmREFRESH. This probably could be done directly to wmSET, but since there might be some previous dependencies to do a clean sourcing, it seemed better to add a new alias. wmREFRESH will refresh the current options used to start the environment, based on the FOAM_SETTINGS environment variable. |
|
FYI: I've tested the patch "patch4foamJob_send_settings_var" with OpenFOAM 2.1.1 and Debian 6.0.3 x86_64. But "foamJob" didn't work as intended, i.e. by sending "FOAM_SETTINGS" to the remote shells. After isolating the problem, it looks like Debian 6.0.3 is using "dash" as default for "sh". The version in particular: 0.5.5.1-7.4 Changing the default from "sh" to "bash" in "foamExec" fixed the problem. Anyway, the problem is that this version of "dash" seems to be unable to handle additional arguments when sourcing a script. |
|
It would be useful if these scripts worked with bash and dash but if this isn't possible then hard-coding them to operate with bash is not a disaster. Could you provide updated versions of the scripts (not patches) for OpenFOAM-2.3.x? |
|
|
|
|
|
foamExec (4,093 bytes)
#!/bin/bash #------------------------------------------------------------------------------ # ========= | # \\ / F ield | OpenFOAM: The Open Source CFD Toolbox # \\ / O peration | # \\ / A nd | Copyright (C) 2011 OpenFOAM Foundation # \\/ M anipulation | #------------------------------------------------------------------------------- # License # This file is part of OpenFOAM. # # OpenFOAM is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # OpenFOAM is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License # for more details. # # You should have received a copy of the GNU General Public License # along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>. # # Script # foamExec # # Description # Usage: foamExec [-v foamVersion] <foamCommand> ... # # Runs the <foamVersion> version of executable <foamCommand> # with the rest of the arguments. # # Can also be used for parallel runs. For example, # \code # mpirun -np <nProcs> \ # foamExec -version <foamVersion> <foamCommand> ... -parallel # \endcode # # Note # This script must exist in <foamInstall>/OpenFOAM-<VERSION>/bin/ # or <foamInstall>/openfoam<VERSION>/bin/ (for the debian version) # # foamEtcFile must be found in the same directory as this script # # SeeAlso # foamEtcFile # #------------------------------------------------------------------------------ usage() { exec 1>&2 while [ "$#" -ge 1 ]; do echo "$1"; shift; done cat<<USAGE Usage: ${0##*/} [OPTION] <application> ... options: -prefix <dir> specify an alternative installation prefix pass through to foamEtcFile and set as FOAM_INST_DIR -version <ver> specify an alternative OpenFOAM version pass through to foamEtcFile -help print the usage * run a particular OpenFOAM version of <application> USAGE exit 1 } #------------------------------------------------------------------------------- # the bin dir: binDir="${0%/*}" # the project dir: projectDir="${binDir%/bin}" # the prefix dir (same as foamInstall): prefixDir="${projectDir%/*}" # # the name used for the project directory # projectDirName="${projectDir##*/}" unset etcOpts version # parse options while [ "$#" -gt 0 ] do case "$1" in -h | -help) usage ;; -m | -mode) [ "$#" -ge 2 ] || usage "'$1' option requires an argument" etcOpts="$etcOpts $1 $2" # pass-thru to foamEtcFile shift ;; -p | -prefix) [ "$#" -ge 2 ] || usage "'$1' option requires an argument" etcOpts="$etcOpts $1 $2" # pass-thru to foamEtcFile prefixDir="$2" shift ;; -v | -version) [ "$#" -ge 2 ] || usage "'$1' option requires an argument" etcOpts="$etcOpts $1 $2" # pass-thru to foamEtcFile version="$2" shift ;; --) shift break ;; -*) usage "invalid option '$1'" ;; *) break ;; esac shift done # # Find and source OpenFOAM settings (bashrc) # placed in function to preserve command-line arguments # sourceRc() { foamDotFile="$($binDir/foamEtcFile $etcOpts bashrc)" || { echo "Error : bashrc file could not be found for OpenFOAM-${version:-${WM_PROJECT_VERSION:-???}}" 1>&2 exit 1 } # set to consistent value before sourcing the bashrc export FOAM_INST_DIR="$prefixDir" . $foamDotFile $FOAM_SETTINGS } [ "$#" -ge 1 ] || usage "no application specified" sourceRc exec "$@" #------------------------------------------------------------------------------ |
|
foamJob (6,979 bytes)
#!/bin/sh #------------------------------------------------------------------------------ # ========= | # \\ / F ield | OpenFOAM: The Open Source CFD Toolbox # \\ / O peration | # \\ / A nd | Copyright (C) 2011-2013 OpenFOAM Foundation # \\/ M anipulation | #------------------------------------------------------------------------------- # License # This file is part of OpenFOAM. # # OpenFOAM is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # OpenFOAM is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License # for more details. # # You should have received a copy of the GNU General Public License # along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>. # # Script # foamJob # # Description # Run an OpenFOAM job in background. # Redirects the output to 'log' in the case directory. # #------------------------------------------------------------------------------ usage() { exec 1>&2 while [ "$#" -ge 1 ]; do echo "$1"; shift; done cat<<USAGE Usage: ${0##*/} [OPTION] <application> ... options: -case <dir> specify alternative case directory, default is the cwd -parallel parallel run of processors -screen also sends output to screen -wait wait for execution to complete (when not using -screen) -version <ver> specify an alternative OpenFOAM version -help print the usage * run an OpenFOAM job in background. Redirects the output to 'log' in the case directory USAGE exit 1 } #for being able to echo strings that have single quotes echoArgs() { addSpace="" for stringItem in "$@"; do echo -n "${addSpace}" if [ "${stringItem##* }" = "$stringItem" ] then echo -n "$stringItem" addSpace=" " else echo -n "'$stringItem'" addSpace=" " fi done unset stringItem addSpace } unset version # replacement for possibly buggy 'which' findExec() { case "$1" in */*) if [ -x "$1" ] then echo "$1" return 0 fi ;; esac oldIFS=$IFS IFS=':' for d in $PATH do # echo "testing: $d/$1" 1>&2 if [ -x "$d/$1" -a ! -d "$d/$1" ] then # echo "Found exec: $d/$1" 1>&2 IFS=$oldIFS echo "$d/$1" return 0 fi done IFS=$oldIFS echo "" return 1 } # MAIN SCRIPT #~~~~~~~~~~~~ unset parallelOpt screenOpt waitOpt # parse options while [ "$#" -gt 0 ] do case "$1" in -h | -help) usage ;; -case) [ "$#" -ge 2 ] || usage "'$1' option requires an argument" cd "$2" 2>/dev/null || usage "directory does not exist: '$2'" shift 2 ;; -p | -parallel) parallelOpt=true shift ;; -s | -screen) screenOpt=true shift ;; -w | -wait) waitOpt=true shift ;; -v | -version) [ "$#" -ge 2 ] || usage "'$1' option requires an argument" version="$2" shift 2 ;; --) shift break ;; -*) usage "invalid option '$1'" ;; *) break ;; esac done [ "$#" -ge 1 ] || usage "No application specified" # use foamExec for a specified version # also need foamExec for remote (parallel) runs if [ -n "$version" -o "$parallelOpt" = true ] then # when possible, determine if application even exists if [ -z "$version" ] then findExec $1 >/dev/null || usage "Application '$1' not found" fi # use foamExec for dispatching APPLICATION=`findExec foamExec` || usage "'foamExec' not found" [ -n "$version" ] && APPLICATION="$APPLICATION -version $version" # attempt to preserve the installation directory 'FOAM_INST_DIR' if [ -d "$FOAM_INST_DIR" ] then APPLICATION="$APPLICATION -prefix $FOAM_INST_DIR" fi else APPLICATION=`findExec $1` || usage "Application '$1' not found" echo "Application : $1" shift fi if [ "$parallelOpt" = true ] then # parallel # ~~~~~~~~ # # is the case decomposed? # if [ -r "processor0" ] then NPROCS="`/bin/ls -1d processor* | wc -l`" else echo "Case is not currently decomposed" if [ -r system/decomposeParDict ] then echo "system/decomposeParDict exists" echo "Try decomposing with \"foamJob decomposePar\"" exit 1 else echo "Cannot find system/decomposeParDict file required to decompose the case for parallel running." echo "Please consult the User Guide for details of parallel running" exit 1 fi fi # # locate mpirun # mpirun=`findExec mpirun` || usage "'mpirun' not found" mpiopts="-np $NPROCS" # # is the machine ready to run parallel? # echo "Parallel processing using $WM_MPLIB with $NPROCS processors" case "$WM_MPLIB" in *OPENMPI) # add hostfile info for hostfile in \ hostfile \ machines \ system/hostfile \ system/machines \ ; do if [ -r $hostfile ] then mpiopts="$mpiopts -hostfile $hostfile" break fi done # # Send FOAM_SETTINGS to parallel processes, so that the proper # definitions are sent as well. # mpiopts="$mpiopts -x FOAM_SETTINGS" ;; esac # # run (in parallel) # if [ "$screenOpt" = true ] then echo "Executing: $mpirun $mpiopts $APPLICATION $(echoArgs "$@") -parallel | tee log" $mpirun $mpiopts $APPLICATION "$@" -parallel | tee log else echo "Executing: $mpirun $mpiopts $APPLICATION $(echoArgs "$@") -parallel > log 2>&1" $mpirun $mpiopts $APPLICATION "$@" -parallel > log 2>&1 & pid=$! if [ "$waitOpt" = true ] then wait $pid fi fi else # # run (on single processor) # if [ "$screenOpt" = true ] then echo "Executing: $APPLICATION $(echoArgs "$@") | tee log &" $APPLICATION "$@" | tee log & wait $! else echo "Executing: $APPLICATION $(echoArgs "$@") > log 2>&1 &" $APPLICATION "$@" > log 2>&1 & pid=$! if [ "$waitOpt" = true ] then wait $pid fi fi fi #------------------------------------------------------------------------------ |
|
Attached are the following files (well, without the respective paths): - "bin/foamExec" - this is the file that really needs to be "bash", so that FOAM_SETTINGS is properly passed over the pipeline, otherwise it's lost. This was tested on Ubuntu 12.04, which has dash "0.5.7-2ubuntu2", therefore it's not something only related to older dash versions. - "bin/foamJob" - has the suggested new "-wait" option and the FOAM_SETTINGS export for Open-MPI. - "etc/config/aliases.csh" - has the suggested new alias "wmREFRESH" and has what I believe is a bug fix for "wmUNSET". - "etc/config/aliases.sh" - has the suggested new alias "wmREFRESH". As mentioned, this was tested on Ubuntu 12.04, although I didn't test with a "csh" shell. Now that I think about it, perhaps it the "wmREFRESH" option should have been a copy-paste-change of "wmSET", to avoid a small alias recursion... edit: forgot to mention that these files are based on recent 2.3.x commit 993cd4ee12. |
|
Thanks for the updated files Resolved by commit 369605c630c34a8e14c6b24866fca643f91e0222 |
Date Modified | Username | Field | Change |
---|---|---|---|
2011-06-28 22:50 | wyldckat | New Issue | |
2011-06-28 22:50 | wyldckat | File Added: patch4foamJob_send_settings_var.gz | |
2011-06-28 22:50 | wyldckat | File Added: patch4foamJob_wait_in_parallel.gz | |
2011-06-28 22:52 | wyldckat | Note Added: 0000497 | |
2011-07-10 12:12 | wyldckat | File Added: patch_aliases_refresh | |
2011-07-10 12:14 | wyldckat | File Added: patch4foamJob_send_settings_var | |
2011-07-10 12:14 | wyldckat | File Added: patch4foamJob_wait_in_parallel | |
2011-07-10 12:18 | wyldckat | Note Added: 0000531 | |
2012-12-24 14:52 | wyldckat | Note Added: 0001846 | |
2015-02-05 09:35 | henry | Note Added: 0003683 | |
2015-02-05 21:37 | wyldckat | File Added: aliases.csh | |
2015-02-05 21:37 | wyldckat | File Added: aliases.sh | |
2015-02-05 21:37 | wyldckat | File Added: foamExec | |
2015-02-05 21:37 | wyldckat | File Added: foamJob | |
2015-02-05 21:45 | wyldckat | Note Added: 0003702 | |
2015-02-05 22:00 | wyldckat | Note Edited: 0003702 | |
2015-02-06 10:17 | henry | Note Added: 0003709 | |
2015-02-06 10:17 | henry | Status | new => resolved |
2015-02-06 10:17 | henry | Resolution | open => fixed |
2015-02-06 10:17 | henry | Assigned To | => henry |