View Issue Details

IDProjectCategoryView StatusLast Update
0000211OpenFOAMBugpublic2014-12-14 21:46
Reporterwyldckat Assigned Tohenry  
PrioritylowSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
PlatformAny 
Summary0000211: Update the work flow when building with wmake for modern machines, by updating aliases
DescriptionToday's machines have up to 8 cores/threads in a single CPU, where some of them can probably lap around OpenCFD's good old "dm" and "noisy" a few times ;)

So my suggestion is to add to the portfolio of etc/aliases.{sh,csh} the following (or similar) entries (this is for bash/sh, untested with korn shell):

#wmSC - single core machine
#wmMC - multi-core machine
alias wmSC='unset WM_NCOMPPROCS && echo "Building enabled for a single core"'
alias wmMC='export WM_NCOMPPROCS=1; test -r /proc/cpuinfo && export WM_NCOMPPROCS=$(egrep "^processor" /proc/cpuinfo | wc -l) && echo "Building enabled for $WM_NCOMPPROCS cores"'

( Based on "ThirdParty/tools/makeThirdPartyFunctions:getNCOMPPROCS", so I assume it will work with korn shell. )


Of course this is for a single machine, multi-core set-up.
As for the work flow, this alias could be referenced on the chapter "Building the Sources": http://www.openfoam.com/download/source.php#x5-25000
Something like: «Go to the top-level source directory $WM_PROJECT_DIR, run wmMC and execute the top-level build script ./Allwmake.»
Or refer to it after that paragraph.
Additional Information"wmOpt", "wmDbg" and "wmProf" would also be a nice thing to add... but I suppose you'll want to leave that to the people who are more comfortable with OpenFOAM's building system and let them tweak the environment on their own :)

But multi-core is "sort-of a must" nowadays! ;)
TagsNo tags attached.

Activities

user19

2011-06-05 10:31

  ~0000421

Or, even better, create a command-line parsing shell script à la git-sh-setup which is sourced in all Allwmake scripts and in the wmake script. For one, it should know the -j flag. If it is missing, WM_NCOMPPROCS is left alone. If it is passed without an argument, WM_NCOMPPROCS is set to the value of $(($(getconf _NPROCESSORS_ONLN)+1)), otherwise it is set to the value passed as the argument. This would allow one to do

$ ./Allwmake -j

to get a parallel build with the optimal number of threads (_NPROCESSORS_ONLN+1 is a good idea because it makes sure that there is still some work to be done while the cores are waiting for the disk).

wyldckat

2011-07-10 14:28

updater   ~0000532

Attached is the patch "patch_multi_core_aliases" that adds the two proposed aliases, along with some more aliases that can simplify the usage of multi-core single/multi-machine. I'll document their usage on the next note(s).

The patch "patch_paraview_csh" is a fix for "etc/config/paraview.csh", because with csh (C shell) in Ubuntu gave me the error "Invalid null command." on that line. And by what I've read, that was the quickest and most reliable fix.

@mwild: over-scheduling by 1 can be a bit annoying to those people who wish to continue using the machine for browsing and other stuff :(

wyldckat

2011-07-10 14:29

updater  

patch_multi_core_aliases (2,890 bytes)   
diff --git a/etc/config/aliases.csh b/etc/config/aliases.csh
index 6573c91..c259bac 100644
--- a/etc/config/aliases.csh
+++ b/etc/config/aliases.csh
@@ -39,7 +39,7 @@ alias wmSP 'wmSET WM_PRECISION_OPTION=SP'
 alias wmDP 'wmSET WM_PRECISION_OPTION=DP'
 
 # clear env
-alias wmUNSET='source $WM_PROJECT_DIR/etc/config/unset.csh'
+alias wmUNSET 'source $WM_PROJECT_DIR/etc/config/unset.csh'
 
 # Toggle wmakeScheduler on/off
 #  - also need to set WM_HOSTS
@@ -47,6 +47,22 @@ alias wmUNSET='source $WM_PROJECT_DIR/etc/config/unset.csh'
 alias wmSchedON 'setenv WM_SCHEDULER $WM_PROJECT_DIR/wmake/wmakeScheduler'
 alias wmSchedOFF 'unsetenv WM_SCHEDULER'
 
+# Set hosts for scheduler
+#  - if none defined, single machine is assumed automatically
+alias wmHosts 'setenv WM_HOSTS'
+
+# Set default colours for scheduler
+alias wmColours 'black blue green cyan red magenta yellow'
+
+# Fast switching between multi-core and single-core building
+#   wmSCS - single-core single-machine
+#   wmMCS - multi-core single-machine
+#   wmMCM - multi-core multi-machine
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+alias wmSCS 'unsetenv WM_NCOMPPROCS && echo "Building enabled for a single core"'
+alias wmMCS 'setenv WM_NCOMPPROCS 1; test -r /proc/cpuinfo && setenv WM_NCOMPPROCS `egrep "^processor" /proc/cpuinfo | wc -l` && echo "Building enabled for $WM_NCOMPPROCS cores"'
+alias wmMCM 'setenv WM_NCOMPPROCS `wmSchedON; $WM_SCHEDULER -count` && echo "Building enabled for $WM_NCOMPPROCS cores"'
+
 # Change ParaView version
 # ~~~~~~~~~~~~~~~~~~~~~~~
 alias foamPV 'source `$WM_PROJECT_DIR/etc/config/paraview.csh` ParaView_VERSION=\!*; echo paraview-$ParaView_VERSION'
diff --git a/etc/config/aliases.sh b/etc/config/aliases.sh
index 1881597..d5b654a 100644
--- a/etc/config/aliases.sh
+++ b/etc/config/aliases.sh
@@ -47,6 +47,25 @@ alias wmUNSET='. $WM_PROJECT_DIR/etc/config/unset.sh'
 alias wmSchedON='export WM_SCHEDULER=$WM_PROJECT_DIR/wmake/wmakeScheduler'
 alias wmSchedOFF='unset WM_SCHEDULER'
 
+# Set hosts for scheduler
+#  - if none defined, single machine is assumed by wmakeScheduler
+wmHosts()
+{
+    export WM_HOSTS="$*"
+}
+
+# Set default colours for scheduler
+alias wmColours='black blue green cyan red magenta yellow'
+
+# Fast switching between multi-core and single-core building
+#   wmSCS - single-core single-machine
+#   wmMCS - multi-core single-machine
+#   wmMCM - multi-core multi-machine
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+alias wmSCS='unset WM_NCOMPPROCS && echo "Building enabled for a single core"'
+alias wmMCS='export WM_NCOMPPROCS=1; test -r /proc/cpuinfo && export WM_NCOMPPROCS=$(egrep "^processor" /proc/cpuinfo | wc -l) && echo "Building enabled for $WM_NCOMPPROCS cores"'
+alias wmMCM='export WM_NCOMPPROCS=$(wmSchedON; $WM_SCHEDULER -count) && echo "Building enabled for $WM_NCOMPPROCS core(s)"'
+
 # Change ParaView version
 # ~~~~~~~~~~~~~~~~~~~~~~~
 unset foamPV

patch_multi_core_aliases (2,890 bytes)   

wyldckat

2011-07-10 14:29

updater  

patch_paraview_csh (492 bytes)   
diff --git a/etc/config/paraview.csh b/etc/config/paraview.csh
index 86a5287..7a09f1b 100644
--- a/etc/config/paraview.csh
+++ b/etc/config/paraview.csh
@@ -74,8 +74,7 @@ case "$ParaView_MAJOR".*:
 
 case [0-9]*:
     # extract major from the version
-    setenv ParaView_MAJOR `echo ${ParaView_VERSION} | \
-        sed -e 's/^\([0-9][0-9]*\.[0-9][0-9]*\).*$/\1/'`
+    setenv ParaView_MAJOR `echo ${ParaView_VERSION} | sed -e 's/^\([0-9][0-9]*\.[0-9][0-9]*\).*$/\1/'`
     breaksw
 endsw
 
patch_paraview_csh (492 bytes)   

wyldckat

2011-07-10 14:37

updater   ~0000533

Last edited: 2011-07-10 14:39

On BASH/SH:
  wmHosts #initialize an empty WM_HOSTS. wmakeScheduler will use single machine by default.
  wmHosts aaa:1 bbb:4 #same as 'export WM_HOSTS="aaa:1 bbb:4"'
  
  wmColours #start up with the default colours, just like 'export WM_COLOURS="black blue green cyan red magenta yellow"'

  wmSCS # enable building with single-machine, single-core, but doesn't deactivate the scheduler
  wmMCS # enable building with single-machine, multi-core.
  wmMCM # activates the scheduler and building with multi-core. Implies proper previous usage of wmHosts


Scenarios:
 Scenario 1 - single machine, multi-core - run:
   wmMCS

 Scenario 2 - multiple machines, multi-core - run:
   wmColours
   wmHosts aaa:1 bbb:4
   wmMCM

 Scenario 3 - deactivate multi-core:
   wmSCS

wyldckat

2011-07-10 14:38

updater   ~0000534

On CSH/TCSH:
  wmHosts #initialize an empty WM_HOSTS. wmakeScheduler will use single machine by default.
  wmHosts 'aaa:1 bbb:4' #same as 'setenv WM_HOSTS "aaa:1 bbb:4"'
  
  wmColours #start up with the default colours, just like 'setenv WM_COLOURS "black blue green cyan red magenta yellow"'

  wmSCS # enable building with single-machine, single-core, but doesn't deactivate the scheduler
  wmMCS # enable building with single-machine, multi-core.
  wmMCM # activates the scheduler and building with multi-core. Implies proper previous usage of wmHosts


Scenarios:
 Scenario 1 - single machine, multi-core - run:
   wmMCS

 Scenario 2 - multiple machines, multi-core - run:
   wmColours
   wmHosts "aaa:1 bbb:4"
   wmMCM

 Scenario 3 - deactivate multi-core:
   wmSCS

henry

2014-12-14 21:46

manager   ~0003316

Resolved by commit bc5134a55ec35950a302608c81b47b8efe84798b in
https://github.com/OpenFOAM/OpenFOAM-dev

Issue History

Date Modified Username Field Change
2011-06-05 00:18 wyldckat New Issue
2011-06-05 10:31 user19 Note Added: 0000421
2011-07-10 14:28 wyldckat Note Added: 0000532
2011-07-10 14:29 wyldckat File Added: patch_multi_core_aliases
2011-07-10 14:29 wyldckat File Added: patch_paraview_csh
2011-07-10 14:37 wyldckat Note Added: 0000533
2011-07-10 14:38 wyldckat Note Added: 0000534
2011-07-10 14:39 wyldckat Note Edited: 0000533
2013-11-27 13:08 user803 File Added: bcl_HTML4.html
2013-11-27 22:22 henry File Deleted: bcl_HTML4.html
2014-12-14 21:46 henry Note Added: 0003316
2014-12-14 21:46 henry Status new => resolved
2014-12-14 21:46 henry Resolution open => fixed
2014-12-14 21:46 henry Assigned To => henry