View Issue Details

IDProjectCategoryView StatusLast Update
0000231OpenFOAMBugpublic2015-02-06 10:17
Reporterwyldckat Assigned Tohenry  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Summary0000231: foamJob doesn't take advantage of sending FOAM_SETTINGS to foamExec
DescriptionWhen using foamJob, the very useful "FOAM_SETTINGS" environment variable is not visible to "foamExec" on the remote nodes. For it to be visible, "mpirun" must have the additional option "-x FOAM_SETTINGS".
Steps To ReproduceOn a multi-architecture environment of OpenFOAM builds:
* Start the OpenFOAM environment with a dependency on a special "prefs-debug.sh" environment script or direct option definitions, such as "WM_ARCH_OPTION=32" and/or "WM_PRECISION_OPTION=SP", which aren't the default parameters. In other words, each new terminal has a clean environment and by calling the desired alias, the respective environment is activated... something like this:
   alias of20x-dbg32SP='source $HOME/OpenFOAM/OpenFOAM-2.0.x/etc/bashrc etc/prefs-debugSP.sh WM_ARCH_OPTION=32'

* Then run in this environment a parallel case using "foamJob". On the remote processes, no OpenFOAM environment is started by default because "~/.bashrc" only has aliases. So, only "foamJob" relies 100% on "foamExec". "mpirun" is not told to send the variable FOAM_SETTINGS to the remote processes, therefore they will use the default options and/or "prefs.sh" options available at select folders.

* Given this set-up, the parallel case might not even run, since the default options might not be applicable in the remote machines...
Additional InformationAttached is the patch for fixing this: patch4foamJob_send_settings_var.gz
TagsNo tags attached.

Activities

wyldckat

2011-06-28 22:50

updater  

wyldckat

2011-06-28 22:50

updater  

wyldckat

2011-06-28 22:52

updater   ~0000497

I'll use this report for a request to add a "-wait" argument to the "foamJob" script. The patch is attached with the name: patch4foamJob_wait_in_parallel.gz

This is because sometimes it's useful to wait for a parallel run to complete, without having to see the log on screen or redirected elsewhere...

wyldckat

2011-07-10 12:12

updater  

patch_aliases_refresh (864 bytes)   
diff --git a/etc/config/aliases.csh b/etc/config/aliases.csh
index 6573c91..82c3e05 100644
--- a/etc/config/aliases.csh
+++ b/etc/config/aliases.csh
@@ -38,6 +38,9 @@ alias wm32 'wmSET WM_ARCH_OPTION=32'
 alias wmSP 'wmSET WM_PRECISION_OPTION=SP'
 alias wmDP 'wmSET WM_PRECISION_OPTION=DP'
 
+# refresh the environment
+alias wmREFRESH 'wmSET $FOAM_SETTINGS'
+
 # clear env
 alias wmUNSET='source $WM_PROJECT_DIR/etc/config/unset.csh'
 
diff --git a/etc/config/aliases.sh b/etc/config/aliases.sh
index 1881597..011958a 100644
--- a/etc/config/aliases.sh
+++ b/etc/config/aliases.sh
@@ -38,6 +38,9 @@ alias wm32='wmSET WM_ARCH_OPTION=32'
 alias wmSP='wmSET WM_PRECISION_OPTION=SP'
 alias wmDP='wmSET WM_PRECISION_OPTION=DP'
 
+# refresh the environment
+alias wmREFRESH='wmSET $FOAM_SETTINGS'
+
 # clear env
 alias wmUNSET='. $WM_PROJECT_DIR/etc/config/unset.sh'
 
patch_aliases_refresh (864 bytes)   

wyldckat

2011-07-10 12:14

updater  

patch4foamJob_send_settings_var (376 bytes)   
diff --git a/bin/foamJob b/bin/foamJob
index 804976e..f487e2e 100755
--- a/bin/foamJob
+++ b/bin/foamJob
@@ -214,6 +214,12 @@ then
                 break
             fi
         done
+
+        #
+        # Send FOAM_SETTINGS to parallel processes. so the proper definitions are sent as well.
+        #
+        mpiopts="$mpiopts -x FOAM_SETTINGS"
+
         ;;
     esac
 

wyldckat

2011-07-10 12:14

updater  

patch4foamJob_wait_in_parallel (1,314 bytes)   
diff --git a/bin/foamJob b/bin/foamJob
index 70e2c39..f487e2e 100755
--- a/bin/foamJob
+++ b/bin/foamJob
@@ -40,6 +40,7 @@ options:
   -case <dir>       specify alternative case directory, default is the cwd
   -parallel         parallel run of processors
   -screen           also sends output to screen
+  -wait             wait for execution to complete
   -version <ver>    specify an alternative OpenFOAM version
   -help             print the usage

@@ -86,7 +87,7 @@ findExec() {

 # MAIN SCRIPT
 #~~~~~~~~~~~~
-unset parallelOpt screenOpt
+unset parallelOpt screenOpt waitOpt


 # parse options
@@ -109,6 +110,10 @@ do
       screenOpt=true
       shift
       ;;
+   -w | -wait)
+      waitOpt=true
+      shift
+      ;;
    -v | -version)
       [ "$#" -ge 2 ] || usage "'$1' option requires an argument"
       version="$2"
@@ -222,6 +233,8 @@ then
     else
         echo "Executing: $mpirun $mpiopts $APPLICATION $@ -parallel > log 2>&1"
         $mpirun $mpiopts $APPLICATION $@ -parallel > log 2>&1 &
+        pid=$!
+        if [ "$waitOpt" = true ]; then wait $pid; fi
     fi

 else
@@ -236,6 +249,8 @@ else
     else
         echo "Executing: $APPLICATION $@ > log 2>&1 &"
         $APPLICATION $@ > log 2>&1 &
+        pid=$!
+        if [ "$waitOpt" = true ]; then wait $pid; fi
     fi
 fi

patch4foamJob_wait_in_parallel (1,314 bytes)   

wyldckat

2011-07-10 12:18

updater   ~0000531

I re-uploaded the 2 initial patch files as normal text files, since this way it's easier to analyse them directly.

I uploaded another patch named "patch_aliases_refresh" for adding the alias wmREFRESH. This probably could be done directly to wmSET, but since there might be some previous dependencies to do a clean sourcing, it seemed better to add a new alias.
wmREFRESH will refresh the current options used to start the environment, based on the FOAM_SETTINGS environment variable.

wyldckat

2012-12-24 14:52

updater   ~0001846

FYI: I've tested the patch "patch4foamJob_send_settings_var" with OpenFOAM 2.1.1 and Debian 6.0.3 x86_64. But "foamJob" didn't work as intended, i.e. by sending "FOAM_SETTINGS" to the remote shells.

After isolating the problem, it looks like Debian 6.0.3 is using "dash" as default for "sh". The version in particular: 0.5.5.1-7.4
Changing the default from "sh" to "bash" in "foamExec" fixed the problem.

Anyway, the problem is that this version of "dash" seems to be unable to handle additional arguments when sourcing a script.

henry

2015-02-05 09:35

manager   ~0003683

It would be useful if these scripts worked with bash and dash but if this isn't possible then hard-coding them to operate with bash is not a disaster.

Could you provide updated versions of the scripts (not patches) for OpenFOAM-2.3.x?

wyldckat

2015-02-05 21:37

updater  

aliases.csh (2,892 bytes)

wyldckat

2015-02-05 21:37

updater  

aliases.sh (2,923 bytes)

wyldckat

2015-02-05 21:37

updater  

foamExec (4,093 bytes)   
#!/bin/bash
#------------------------------------------------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | Copyright (C) 2011 OpenFOAM Foundation
#    \\/     M anipulation  |
#-------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM.
#
#     OpenFOAM is free software: you can redistribute it and/or modify it
#     under the terms of the GNU General Public License as published by
#     the Free Software Foundation, either version 3 of the License, or
#     (at your option) any later version.
#
#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
#     for more details.
#
#     You should have received a copy of the GNU General Public License
#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
#
# Script
#     foamExec
#
# Description
#     Usage: foamExec [-v foamVersion] <foamCommand> ...
#
#     Runs the <foamVersion> version of executable <foamCommand>
#     with the rest of the arguments.
#
#     Can also be used for parallel runs. For example,
#     \code
#         mpirun -np <nProcs> \
#             foamExec -version <foamVersion> <foamCommand> ... -parallel
#     \endcode
#
# Note
#     This script must exist in <foamInstall>/OpenFOAM-<VERSION>/bin/
#     or <foamInstall>/openfoam<VERSION>/bin/ (for the debian version)
#
#     foamEtcFile must be found in the same directory as this script
#
# SeeAlso
#    foamEtcFile
#
#------------------------------------------------------------------------------
usage() {
    exec 1>&2
    while [ "$#" -ge 1 ]; do echo "$1"; shift; done
    cat<<USAGE

Usage: ${0##*/} [OPTION] <application> ...

options:
  -prefix <dir>     specify an alternative installation prefix
                    pass through to foamEtcFile and set as FOAM_INST_DIR
  -version <ver>    specify an alternative OpenFOAM version
                    pass through to foamEtcFile
  -help             print the usage

* run a particular OpenFOAM version of <application>

USAGE
    exit 1
}

#-------------------------------------------------------------------------------

# the bin dir:
binDir="${0%/*}"

# the project dir:
projectDir="${binDir%/bin}"

# the prefix dir (same as foamInstall):
prefixDir="${projectDir%/*}"

# # the name used for the project directory
# projectDirName="${projectDir##*/}"


unset etcOpts version
# parse options
while [ "$#" -gt 0 ]
do
    case "$1" in
    -h | -help)
        usage
        ;;
    -m | -mode)
        [ "$#" -ge 2 ] || usage "'$1' option requires an argument"
        etcOpts="$etcOpts $1 $2"    # pass-thru to foamEtcFile
        shift
        ;;
    -p | -prefix)
        [ "$#" -ge 2 ] || usage "'$1' option requires an argument"
        etcOpts="$etcOpts $1 $2"    # pass-thru to foamEtcFile
        prefixDir="$2"
        shift
        ;;
    -v | -version)
        [ "$#" -ge 2 ] || usage "'$1' option requires an argument"
        etcOpts="$etcOpts $1 $2"    # pass-thru to foamEtcFile
        version="$2"
        shift
        ;;
    --)
        shift
        break
        ;;
    -*)
        usage "invalid option '$1'"
        ;;
    *)
        break
        ;;
    esac
    shift
done

#
# Find and source OpenFOAM settings (bashrc)
# placed in function to preserve command-line arguments
#
sourceRc()
{
    foamDotFile="$($binDir/foamEtcFile $etcOpts bashrc)" || {
        echo "Error : bashrc file could not be found for OpenFOAM-${version:-${WM_PROJECT_VERSION:-???}}" 1>&2
        exit 1
    }

    # set to consistent value before sourcing the bashrc
    export FOAM_INST_DIR="$prefixDir"

    . $foamDotFile $FOAM_SETTINGS
}


[ "$#" -ge 1 ] || usage "no application specified"

sourceRc
exec "$@"

#------------------------------------------------------------------------------
foamExec (4,093 bytes)   

wyldckat

2015-02-05 21:37

updater  

foamJob (6,979 bytes)   
#!/bin/sh
#------------------------------------------------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | Copyright (C) 2011-2013 OpenFOAM Foundation
#    \\/     M anipulation  |
#-------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM.
#
#     OpenFOAM is free software: you can redistribute it and/or modify it
#     under the terms of the GNU General Public License as published by
#     the Free Software Foundation, either version 3 of the License, or
#     (at your option) any later version.
#
#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
#     for more details.
#
#     You should have received a copy of the GNU General Public License
#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
#
# Script
#     foamJob
#
# Description
#     Run an OpenFOAM job in background.
#     Redirects the output to 'log' in the case directory.
#
#------------------------------------------------------------------------------
usage() {
    exec 1>&2
    while [ "$#" -ge 1 ]; do echo "$1"; shift; done
    cat<<USAGE

Usage: ${0##*/} [OPTION] <application> ...
options:
  -case <dir>       specify alternative case directory, default is the cwd
  -parallel         parallel run of processors
  -screen           also sends output to screen
  -wait             wait for execution to complete (when not using -screen)
  -version <ver>    specify an alternative OpenFOAM version
  -help             print the usage

* run an OpenFOAM job in background.
  Redirects the output to 'log' in the case directory

USAGE
    exit 1
}

#for being able to echo strings that have single quotes
echoArgs() {
    addSpace=""

    for stringItem in "$@"; do

        echo -n "${addSpace}"

        if [ "${stringItem##* }" = "$stringItem" ]
        then
            echo -n "$stringItem"
            addSpace=" "
        else
            echo -n "'$stringItem'"
            addSpace=" "
        fi

    done

    unset stringItem addSpace
}

unset version

# replacement for possibly buggy 'which'
findExec() {
    case "$1" in
    */*)
        if [ -x "$1" ]
        then
            echo "$1"
            return 0
        fi
        ;;
    esac

    oldIFS=$IFS
    IFS=':'
    for d in $PATH
    do
        # echo "testing: $d/$1" 1>&2
        if [ -x "$d/$1" -a ! -d "$d/$1" ]
        then
            # echo "Found exec: $d/$1" 1>&2
            IFS=$oldIFS
            echo "$d/$1"
            return 0
        fi
     done
     IFS=$oldIFS
     echo ""
     return 1
}



# MAIN SCRIPT
#~~~~~~~~~~~~
unset parallelOpt screenOpt waitOpt


# parse options
while [ "$#" -gt 0 ]
do
   case "$1" in
   -h | -help)
      usage
      ;;
   -case)
      [ "$#" -ge 2 ] || usage "'$1' option requires an argument"
      cd "$2" 2>/dev/null || usage "directory does not exist:  '$2'"
      shift 2
      ;;
   -p | -parallel)
      parallelOpt=true
      shift
      ;;
   -s | -screen)
      screenOpt=true
      shift
      ;;
   -w | -wait)
      waitOpt=true
      shift
      ;;
   -v | -version)
      [ "$#" -ge 2 ] || usage "'$1' option requires an argument"
      version="$2"
      shift 2
      ;;
   --)
      shift
      break
      ;;
   -*)
      usage "invalid option '$1'"
      ;;
   *)
      break
      ;;
   esac
done

[ "$#" -ge 1 ] || usage "No application specified"


# use foamExec for a specified version
# also need foamExec for remote (parallel) runs
if [ -n "$version" -o "$parallelOpt" = true ]
then
    # when possible, determine if application even exists
    if [ -z "$version" ]
    then
        findExec $1 >/dev/null || usage "Application '$1' not found"
    fi

    # use foamExec for dispatching
    APPLICATION=`findExec foamExec` || usage "'foamExec' not found"

    [ -n "$version" ] && APPLICATION="$APPLICATION -version $version"

    # attempt to preserve the installation directory 'FOAM_INST_DIR'
    if [ -d "$FOAM_INST_DIR" ]
    then
        APPLICATION="$APPLICATION -prefix $FOAM_INST_DIR"
    fi

else
    APPLICATION=`findExec $1` || usage "Application '$1' not found"
    echo "Application : $1"
    shift
fi


if [ "$parallelOpt" = true ]
then
    # parallel
    # ~~~~~~~~

    #
    # is the case decomposed?
    #
    if [ -r "processor0" ]
    then
        NPROCS="`/bin/ls -1d processor* | wc -l`"
    else
        echo "Case is not currently decomposed"
        if [ -r system/decomposeParDict ]
        then
            echo "system/decomposeParDict exists"
            echo "Try decomposing with \"foamJob decomposePar\""
            exit 1
        else
            echo "Cannot find system/decomposeParDict file required to decompose the case for parallel running."
            echo "Please consult the User Guide for details of parallel running"
            exit 1
        fi
    fi

    #
    # locate mpirun
    #
    mpirun=`findExec mpirun` || usage "'mpirun' not found"
    mpiopts="-np $NPROCS"

    #
    # is the machine ready to run parallel?
    #
    echo "Parallel processing using $WM_MPLIB with $NPROCS processors"
    case "$WM_MPLIB" in
    *OPENMPI)
        # add hostfile info
        for hostfile in \
            hostfile \
            machines \
            system/hostfile \
            system/machines \
            ;
        do
            if [ -r $hostfile ]
            then
                mpiopts="$mpiopts -hostfile $hostfile"
                break
            fi
        done

        #
        # Send FOAM_SETTINGS to parallel processes, so that the proper
        # definitions are sent as well.
        #
        mpiopts="$mpiopts -x FOAM_SETTINGS"

        ;;
    esac

    #
    # run (in parallel)
    #
    if [ "$screenOpt" = true ]
    then
        echo "Executing: $mpirun $mpiopts $APPLICATION $(echoArgs "$@") -parallel | tee log"
        $mpirun $mpiopts $APPLICATION "$@" -parallel | tee log
    else
        echo "Executing: $mpirun $mpiopts $APPLICATION $(echoArgs "$@") -parallel > log 2>&1"
        $mpirun $mpiopts $APPLICATION "$@" -parallel > log 2>&1 &

        pid=$!
        if [ "$waitOpt" = true ]
        then
            wait $pid
        fi
    fi

else
    #
    # run (on single processor)
    #
    if [ "$screenOpt" = true ]
    then
        echo "Executing: $APPLICATION $(echoArgs "$@") | tee log &"
        $APPLICATION "$@" | tee log &
        wait $!
    else
        echo "Executing: $APPLICATION $(echoArgs "$@") > log 2>&1 &"
        $APPLICATION "$@" > log 2>&1 &

        pid=$!
        if [ "$waitOpt" = true ]
        then
            wait $pid
        fi
    fi
fi

#------------------------------------------------------------------------------
foamJob (6,979 bytes)   

wyldckat

2015-02-05 21:45

updater   ~0003702

Last edited: 2015-02-05 22:00

Attached are the following files (well, without the respective paths):

- "bin/foamExec" - this is the file that really needs to be "bash", so that FOAM_SETTINGS is properly passed over the pipeline, otherwise it's lost. This was tested on Ubuntu 12.04, which has dash "0.5.7-2ubuntu2", therefore it's not something only related to older dash versions.

- "bin/foamJob" - has the suggested new "-wait" option and the FOAM_SETTINGS export for Open-MPI.

- "etc/config/aliases.csh" - has the suggested new alias "wmREFRESH" and has what I believe is a bug fix for "wmUNSET".

- "etc/config/aliases.sh" - has the suggested new alias "wmREFRESH".


As mentioned, this was tested on Ubuntu 12.04, although I didn't test with a "csh" shell.


Now that I think about it, perhaps it the "wmREFRESH" option should have been a copy-paste-change of "wmSET", to avoid a small alias recursion...


edit: forgot to mention that these files are based on recent 2.3.x commit 993cd4ee12.

henry

2015-02-06 10:17

manager   ~0003709

Thanks for the updated files
Resolved by commit 369605c630c34a8e14c6b24866fca643f91e0222

Issue History

Date Modified Username Field Change
2011-06-28 22:50 wyldckat New Issue
2011-06-28 22:50 wyldckat File Added: patch4foamJob_send_settings_var.gz
2011-06-28 22:50 wyldckat File Added: patch4foamJob_wait_in_parallel.gz
2011-06-28 22:52 wyldckat Note Added: 0000497
2011-07-10 12:12 wyldckat File Added: patch_aliases_refresh
2011-07-10 12:14 wyldckat File Added: patch4foamJob_send_settings_var
2011-07-10 12:14 wyldckat File Added: patch4foamJob_wait_in_parallel
2011-07-10 12:18 wyldckat Note Added: 0000531
2012-12-24 14:52 wyldckat Note Added: 0001846
2015-02-05 09:35 henry Note Added: 0003683
2015-02-05 21:37 wyldckat File Added: aliases.csh
2015-02-05 21:37 wyldckat File Added: aliases.sh
2015-02-05 21:37 wyldckat File Added: foamExec
2015-02-05 21:37 wyldckat File Added: foamJob
2015-02-05 21:45 wyldckat Note Added: 0003702
2015-02-05 22:00 wyldckat Note Edited: 0003702
2015-02-06 10:17 henry Note Added: 0003709
2015-02-06 10:17 henry Status new => resolved
2015-02-06 10:17 henry Resolution open => fixed
2015-02-06 10:17 henry Assigned To => henry