View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0002032OpenFOAMBugpublic2016-03-23 13:392016-05-04 20:38
Reporterraviojha23@gmail.com Assigned Tohenry  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
OSLinux 
Fixed in Versiondev 
Summary0002032: The strategy string for scotch decomposition is throwing errors while decomposition of Mesh
DescriptionSome of the parameters of the strategy string that we define for scotch decomposition are not being recognized.The strategy string that I am using is the default one with values of bal parameter tweaked a little.
"r{job=t,map=t,poli=S,sep=(m{asc=b{bnd=(d{pass=40,dif=1,rem=1}|)f{move=80,pass=-1,bal=0.002},org=f{move=80,pass=-1,bal=0.002},width=3},low=h{pass=10}f{move=80,pass=-1,bal=0.002491},type=h,vert=80,rat=0.8}| m{asc=b{bnd=(d{pass=40,dif=1,rem=1}|)f{move=80,pass=-1,bal=0.002},org=f{move=80,pass=-1,bal=0.002},width=3},low=h{pass=10}f{move=80,pass=-1,bal=0.002491},type=h,vert=80,rat=0.8})}";

The parameter diff is not recognized when I try to decompose. Here is the error I get

ERROR: stratParserParse: invalid method parameter name "dif"
Steps To ReproduceI am using motorbike tutorial workload. Just update the file system/decomposeParDict with a parameter called strategy inside the scotchCoeffs and pass the above string as its value.
TagsNo tags attached.

Activities

raviojha23@gmail.com

2016-03-23 13:39

reporter  

decomposeParDict (1,601 bytes)    
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 64;

method          scotch;

scotchCoeffs
{
//strategy "b{sep=m{vert=100,low=h,asc=f}x}";
strategy "r{job=t,map=t,poli=S,sep=(m{asc=b{bnd=(d{pass=40,dif=1,rem=1}|)f{move=80,pass=-1,bal=0.002491},org=f{move=80,pass=-1,bal=0.002491},width=3},low=h{pass=10}f{move=80,pass=-1,bal=0.002491},type=h,vert=80,rat=0.8}| m{asc=b{bnd=(d{pass=40,dif=1,rem=1}|)f{move=80,pass=-1,bal=0.002491},org=f{move=80,pass=-1,bal=0.002491},width=3},low=h{pass=10}f{move=80,pass=-1,bal=0.002491},type=h,vert=80,rat=0.8})}";
}
simpleCoeffs
{
    n               (4 1 1);
    delta           0.001;
}

hierarchicalCoeffs
{
    n               (3 2 1);
    delta           0.001;
    order           xyz;
}

manualCoeffs
{
    dataFile        "cellDecomposition";
}


// ************************************************************************* //
decomposeParDict (1,601 bytes)   

MattijsJ

2016-03-24 10:10

reporter   ~0006056

The strategy string in scotchDecomp.H was extracted using 'gmap -vs'. The strategy string is passed straight in to scotch and obviously scotch thinks it is incorrect.

- maybe your string is incorrect. It could well be that the strategy string in scotchDecomp.H is out of data since extracted with an old version of gmap. Please let me know if we need to upgrade it.

- otherwise there might be a bug in scotch and it would be great if you could submit the bug report in https://gforge.inria.fr/projects/scotch/

Apart from this do you have any documentation on the strategy string? Are you finding any benefit from changing it?

wyldckat

2016-03-25 21:12

updater   ~0006058

I'm going to do this in two parts, since I'm still testing how Doxygen will render the documentation updates I've made to the classes "scotchDecomp" and "ptscotchDecomp". I'll post another comment and the updated files as soon as they're ready.

The problem with raviojha23's strategy string is that it seems to be using the one from the initial description block from the file "src/parallel/decompose/scotchDecomp/scotchDecomp.C". Which by the way, refers to one of the posts made here: http://gforge.inria.fr/forum/forum.php?thread_id=24518&forum_id=806&group_id=248
This string was designed for a rather old version of Scotch (maybe 5.10? I didn't check) and it will not work with Scotch 6.0.0 nor 6.0.3.

Given that the instructions aren't completely clear on how to get the latest strategy, I've managed to figure out it's done based on the bits and pieces made available and I'm going to list the steps here (they will be in the proposed changes as well):


    1. Edit the file system/decomposeParDict and use the following settings:

      method scotch;

      scotchCoeffs
      {
          writeGraph true;
      }

    2. Run decomposePar. For example, it will write a file named region0.grf.

    3. Now, instead of using gmap, run gpart with the following command structure to get the default strategy:

      gpart \<nProcs\> -vs \<grfFile\>

    where:

      - <grfFile> is the file that was obtained with the option writeGraph=true, namely region0.grf.
      - <nProcs> is the numberOfSubdomains defined in the dictionary file.

    4. At the end of the execution will be shown a long string, similar to the following example (complete line was cropped at [...]):

        S Strat=m{asc=b{width=3,bnd=d{pass=40,dif=1,rem=0}[...],type=h}

    5. Edit the file system/decomposeParDict once again and add the strategy entry as exemplified:

      method scotch;

      scotchCoeffs
      {
          //writeGraph true;
          strategy "m{asc=b{width=3,bnd=d{pass=40,dif=1,rem=0}[...],type=h}";
      }

    6. Finally, run decomposePar once again, to at least test if it works as intended.


Note: gpart can be found in the current search path by adding the respective bin folder from the Scotch installation, namely by running the following commands:

   source $(foamEtcFile config.sh/scotch)
   export PATH=$PATH:$SCOTCH_ARCH_PATH/bin

------------

@raviojha23: Please do test this and let us know if this works. And also please answer the questions that Mattijs made, since we're all very curious/interested on this topic :)

wyldckat

2016-03-25 22:17

updater  

updatesForScotchDecomp_v1.tar.gz (11,008 bytes)

wyldckat

2016-03-25 22:26

updater   ~0006059

Last edited: 2016-03-25 22:28

 @Henry: Attached is the package "updatesForScotchDecomp_v1.tar.gz", indexed to commit 5879080b3 in OpenFOAM-dev, which provides the following proposed changes:

  - doc/Doxygen/css/doxyMod.css
    - Added style entry for numbered lists, based on the ones for unnumbered lists.

  - src/parallel/decompose/scotchDecomp/scotchDecomp.C
  - src/parallel/decompose/ptscotchDecomp/ptscotchDecomp.C
    - Removed the lengthy description at the top of the two files, to the respective header files. More details below.

  - src/parallel/decompose/scotchDecomp/scotchDecomp.H
    - Added the information that was removed from the respective source file.
    - Reformatted the information to fit the same description convention.
    - Added detailed steps on how to retrieve the strategy default for the current Scotch version; although it was necessary to rely on regular HTML code to ensure the lists were properly formatted as needed.

  - src/parallel/decompose/ptscotchDecomp/ptscotchDecomp.H
    - Added the non-repeated information from the respective source file.
    - In place of the repeated information, pointed out to check the scotchDecomp class for more details.
    - Reformatted the remaining information to make it cleared on how to use the ".dgr" files.
    - No further details were added in this case, given that it relies on the new information that has been provided in "scotchDecomp.H".
    

@Mattjis: I don't know if you want to add more or rectify details for this proposed update?

henry

2016-03-25 22:38

manager   ~0006060

Thanks Bruno, I have applied your changes to OpenFOAM-dev:
commit 2c2c6efe6393b3ff2f59b29cde3b19da5207fa68

MattijsJ

2016-03-26 19:49

reporter   ~0006061

@Bruno: nice one. And as you mentioned it'd be really nice to have some (more) documentation on the use of the strategy strings ...

wyldckat

2016-03-27 20:54

updater   ~0006062

After another quick search, in the Scotch document section: http://gforge.inria.fr/docman/?group_id=248&view=listfile&dirid=326 - there we can find the file "scotch_user6.0.pdf".
And in subsection "8.3.2 Mapping strategy strings" there is already a lot of information about the strategy strings... so much information, although no examples, that it feels like it's like we're doing video and audio encoding...

Section "7.4 Strategy strings" in the file "ptscotch_user6.0.pdf" also has a lot of information... although no examples either :(

raviojha23@gmail.com

2016-03-30 12:54

reporter   ~0006065

Please excuse me for the delayed response.Had server access issues :(
@MattijsJ : Yes there is scotch manual 6.0 which I am referring as wyldckat also pointed out.The reason for this exercise is to find out whether I could get any performance improvements by tweaking some of the parameters in strategy string.

There is definitely some issue with the strategy string mentioned in scotchDecomp.C. The strategy string that I generated using wyldckat's method worked fine.

@wyldckat : Thanks a lot for such a detailed explanation. Even the manual falls short in covering the topic judiciously(no examples as you pointed out).

Coming to the topic at hand, I did try the steps you mentioned and it worked :). I am able to run decomposePar with the generated strategy string.I will keep on changing the strategy string unless I get a balanced partition of faces and get some performance improvements.

Amy pointers on the topic would be highly appreciated.

MattijsJ

2016-04-01 16:11

reporter   ~0006070

@raviojha23 : for OpenFOAM lowering the number of inter-processor faces (i.e. 'cuts' in scotch graph speak) is probably more important than
- the balance of the number of cells
- the number of neighbouring processors
So if there is a way of manipulating the cost function inside scotch that would be where to start your investigation.

Issue History

Date Modified Username Field Change
2016-03-23 13:39 raviojha23@gmail.com New Issue
2016-03-23 13:39 raviojha23@gmail.com File Added: decomposeParDict
2016-03-24 10:10 MattijsJ Note Added: 0006056
2016-03-25 21:12 wyldckat Note Added: 0006058
2016-03-25 22:17 wyldckat File Added: updatesForScotchDecomp_v1.tar.gz
2016-03-25 22:26 wyldckat Note Added: 0006059
2016-03-25 22:28 wyldckat Note Edited: 0006059
2016-03-25 22:38 henry Note Added: 0006060
2016-03-26 19:49 MattijsJ Note Added: 0006061
2016-03-27 20:28 wyldckat Project ThirdParty => OpenFOAM
2016-03-27 20:54 wyldckat Note Added: 0006062
2016-03-30 12:54 raviojha23@gmail.com Note Added: 0006065
2016-04-01 16:11 MattijsJ Note Added: 0006070
2016-05-04 20:38 henry Status new => resolved
2016-05-04 20:38 henry Fixed in Version => dev
2016-05-04 20:38 henry Resolution open => fixed
2016-05-04 20:38 henry Assigned To => henry