View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000465 | OpenFOAM | Bug | public | 2012-03-16 22:05 | 2012-06-01 14:40 |
| Reporter | Assigned To | ||||
| Priority | normal | Severity | major | Reproducibility | sometimes |
| Status | closed | Resolution | no change required | ||
| Platform | x86 64 | OS | Red Hat Enterprise Linux Server | OS Version | 5.7 (Tikanga) |
| Summary | 0000465: Problems with refineMesh and refineHexMesh in parallel | ||||
| Description | I am using OpenFOAM 2.0.x. I am trying to make a very large uniform block mesh with regions of local refinement (all hexahedral). The process I use is to make a coarse uniform mesh with blockMesh, do the local refinement in serial with refineMesh or refineHexMesh, run decomposePar, then use refineMesh or refineHexMesh in parallel to refine the entire mesh to the level of desired resolution. Running refineHexMesh in serial followed by refineHexMesh in parallel, just doesn't work, but that's not the main issue here, so disregard that case. What I've found works, sometimes, is to run refineMesh in serial, followed by refineMesh in parallel. It works for some refinement region shapes, but not others, which is the main issue I'm reporting here. Same with running refineHexMesh in serial, followed by refineMesh in parallel. When it doesn't work, I get an error like this: [8] --> FOAM FATAL ERROR: [8] in patch:procBoundary8to2 : Local size of patch is 644 (faces). Received from neighbour 646 faceCentres! | ||||
| Steps To Reproduce | I've attached two tarred/zipped run directories. In each, there are some grid generation scripts. Try running Allrun.refineMesh_refineMesh to create the mesh with blockMesh, locally refine in serial with refineMesh, decompose, and globally refine in parallel with refineMesh. The run_1_array case is an example where the refinement region shape causes no problems. The run_2_array case is just like run_1_array, but has a slightly different refinement region shape, and has the above noted problem. See log.refineMesh.1.b for the errors. | ||||
| Additional Information | If you have questions, feel free to contact me at matt.churchfield@nrel.gov. Thank you. | ||||
| Tags | No tags attached. | ||||
|
2012-03-16 22:05
|
|
|
|
You should try to blockMesh, decomposePar and then run refineHexMesh in parallel. (decomposePar does not know about the extra pointLevel,cellLevel structures needed to restart refineHexMesh so you cannot first refine in serial and then in parallel) |
|
|
The method outlined above, running blockMesh, refineMesh (serial), decomposePar, refineMesh (parallel), does work sometimes, probably half the time. Note, this is refineMesh, not refineHexMesh. Yes, I have seen that using refineHexMesh to do what I am trying to do does not work. It doesn't make sense to run blockMesh, decomposePar, then refine(Hex)Mesh in parallel if there are areas of local refinement because one will end up with a load unbalanced mesh. That is why I run refineMesh in serial to do the local refinement before running decomposePar. My parallel use of refineMesh after decomposing refines the entire mesh, so things remain load balanced. Note, that I've tried blockMesh, decomposePar, refineHexMesh in parallel, and then redistributeMeshPar, but for large meshes (300M cells) on our cluster where each 8-core node has 8GB memory, the redistributeMeshPar will just run for hours (I've tried up to 24 hours) without any resulting redistributed mesh. It seems like the method I've outlined should work--well, it does work for certain refinement regions. It seems like a bug when refineMesh tells me that the number of faces on corresponding processor boundaries are off by one or two faces. Thanks for looking into this. |
|
|
I've found what causes this problem. For example, say I use blockMesh to build a coarse background mesh, and then use refineMesh in serial to create two refinement zones. Say those two zones are rectangular prisms located near one another, but not overlapping. If only one layer of cells of the coarsest resolution lie between the two refinement regions, then I get the problem outlined above after running decomposePar and globally refining with refineMesh in parallel. However, there is no problem when there are at least two layers of the coarsest cells between the refinement regions. |
|
|
Thanks. refineMesh will work reliably for coordinate axis aligned hexes (it is a geometric cutter) and does not synchronise across processor boundaries so that is why a second refinement iteration will fail if it tries to refine. On the other hand refineHexMesh is parallel consistent but needs additional mesh information (cellLevel, pointLevel) which does not get decomposed. The alternative is to run refinHexMesh in parallel and doing load-balancing by hand: - estimate cells you are refining and how many cells that results in - use the processorWeights entry for ptscotch to weight the decomposition - redistributePar - refineHexMesh and now you should hopefully have a (better) balanced mesh. I'm closing this thread now. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2012-03-16 22:05 |
|
New Issue | |
| 2012-03-16 22:05 |
|
File Added: run_array.tar.gz | |
| 2012-03-19 08:50 |
|
Note Added: 0001140 | |
| 2012-03-19 13:10 |
|
Note Added: 0001144 | |
| 2012-05-31 14:55 |
|
Note Added: 0001350 | |
| 2012-06-01 14:40 |
|
Note Added: 0001354 | |
| 2012-06-01 14:40 |
|
Status | new => closed |
| 2012-06-01 14:40 |
|
Assigned To | => user4 |
| 2012-06-01 14:40 |
|
Resolution | open => no change required |
