View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000223 | OpenFOAM | Bug | public | 2011-06-20 16:41 | 2012-02-28 20:04 |
| Reporter | Assigned To | ||||
| Priority | urgent | Severity | major | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Platform | Linux | OS | Arch Linux | OS Version | x86_64 |
| Summary | 0000223: Multi processor runs return different results than single processor runs | ||||
| Description | Running multiprocessor simulations with cyclic boundary conditions return different results than the same case run on a single processor. The problem occurs when two cyclics linked together are decomposed to run on different processor. For example, taking the channel flow test case joined as attachment, when inout1 has its "inlet" residing on one processor, and its "outlet" on an other one. So this could be a simple (2 1 1) decomposition. Taking a simple (1 2 1) decomposition, so having corresponding cyclics on the same processor, yields much more accurate results. The variation (U_1 - U_2) between the two velocity fields at the same time steps are quite significant: - 0.2% of Umax for cases where corresponding cyclics are on different processors - compared to 0.0002% for the cases where corresponding cyclics are on the same processor. It's interesting to note that the mean Courant number returned from the very first timeStep (and onwards of course) is different between parallel and single processor runs. | ||||
| Steps To Reproduce | - Extract test case - Run blockMesh - Compare parallel results to single processor results | ||||
| Additional Information | This problem was discussed and acknowledged by Hrvoje a bit less than a month ago when he was on visit at our university. On his advice, the tests were run using the "diagonal" preconditioner, because it was the only preconditioner which could guarantee identical solution between single and parallel runs. I haven't be able to join him since then, so you might want to contact him to know whether he has made any progress on the issue. | ||||
| Tags | No tags attached. | ||||
|
2011-06-20 16:41
|
|
|
|
Can you reproduce the same behavior with OpenFOAM-2.0.x? If so send the test case and we will investigate. |
|
|
Dear Henry, Thank you for your swift reply! I will give it a try, but haven't installed OpenFOAM-2.0.x yet, so it might take a day or two. I'll come back to you as soon as possible. Kind regards, Francois. |
|
|
I've just tried Francois's test case with OpenFoam 2.0.x using DIC preconditioner (i've uncommented corresponding lines in fvSolution) on one and two processors and got the relative difference in velocity up to 0.37% at time 1 (mag(U-Ua)/mag(U)) I am very interested in this issue too. Kind regards, Ilya. |
|
|
Dear Henry, After compilation and runs with OpenFOAM-2.0.x, the errors returned are exactly the same. This said, I used foamUpgradeCyclics to convert my 1.7.x test cases to 2.0.x cases, and I haven't had time to investigate whether that was enough to make full use of the new cyclic implementation in 2.0.x. Just for information, here are the difference max( mag( U1 - U2)/mag(U1) ) for different decompositions on 2 processors, compared to a single processor run, after a 1 second run: - simple( 2 1 1 ) : 0.538 % - simple( 1 2 1 ) : 0.0009 % - simple( 1 1 2 ) : 0.1858 % Kind regards, Francois. |
|
|
Dear Henry, Further changing the Cyclics definition to the pure 2.0.x standard using matchTolerance 0.0001 instead of featureCos 0.9 yields almost exactly the same results: - simple( 2 1 1 ) : 0.5345 % - simple( 1 2 1 ) : 0.0009 % - simple( 1 1 2 ) : 0.1855 % I was therefore wondering whether any progress was made towards solving this issue? Kind regards, Francois. |
|
|
The case you are running is chaotic at the small scales and the evolution of the turbulent structures is dependent on the details of the numerics, round-off errors etc. and indeed the way in which the domain is decomposed. What is important is that the statistics generated are independent of the decomposition, i.e. the means and variances. However, the case as provided runs for far too short at time to produce reliable means; have you tried to run for much longer and compared the mean velocity field? |
|
|
orphaned report - closing |
|
|
Mm.. I had almost forgotten I finally got feedback om my bug post.. Sorry about that. But as a reply, I should say that yes, I did take longer time averages, anticipating that the problem might come from slight differences in initial conditions.. However, after 30 seconds averaging and 150'000 iterations with a CFLmax of 0.1, there were still differences between single and multi-processor runs. See attachment. Kind regards, Francois. |
|
2012-02-28 14:35
|
|
|
|
How big are the differences? Are they getting smaller with averaging? Where in the domain are they? |
|
|
The differences are in the order of a few percent. Somewhere between 0 and 4% away from the wall, and around 10% close to the wall. The differences to tend to get smaller with averaging, in the order of 0.1 to 0.2% per 5 seconds averaging window, although I don't know what the trend becomes after 30 seconds averaging. This because it already took me more then a month of computation time to get that far on 1 processor, so I stopped the computation at that point. Kind regards, Francois. |
|
|
It would appear that the difference tends to zero with averaging as expected. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2011-06-20 16:41 |
|
New Issue | |
| 2011-06-20 16:41 |
|
File Added: testCase.tgz | |
| 2011-06-20 16:47 | henry | Note Added: 0000464 | |
| 2011-06-20 22:17 |
|
Note Added: 0000465 | |
| 2011-06-20 22:18 |
|
Note Edited: 0000465 | |
| 2011-06-21 12:43 |
|
Note Added: 0000470 | |
| 2011-06-21 23:11 |
|
Note Added: 0000472 | |
| 2011-07-02 16:14 |
|
Note Added: 0000504 | |
| 2012-01-03 18:45 | henry | Note Added: 0000894 | |
| 2012-02-28 12:01 |
|
Note Added: 0001079 | |
| 2012-02-28 12:01 |
|
Status | new => closed |
| 2012-02-28 12:01 |
|
Assigned To | => user2 |
| 2012-02-28 12:01 |
|
Resolution | open => no change required |
| 2012-02-28 14:35 |
|
Note Added: 0001084 | |
| 2012-02-28 14:35 |
|
Status | closed => feedback |
| 2012-02-28 14:35 |
|
Resolution | no change required => reopened |
| 2012-02-28 14:35 |
|
File Added: update-28-02-2012.pdf | |
| 2012-02-28 14:36 |
|
Note Edited: 0001084 | |
| 2012-02-28 15:40 | henry | Note Added: 0001087 | |
| 2012-02-28 18:50 |
|
Note Added: 0001094 | |
| 2012-02-28 18:50 |
|
Status | feedback => assigned |
| 2012-02-28 20:04 | henry | Note Added: 0001095 | |
| 2012-02-28 20:04 | henry | Status | assigned => resolved |
| 2012-02-28 20:04 | henry | Resolution | reopened => fixed |
