View Issue Details

IDProjectCategoryView StatusLast Update
0000747OpenFOAMBugpublic2013-03-14 15:29
Reporterdhora Assigned Tohenry  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Summary0000747: Difference between parallel and serial runs
DescriptionI got different results in a SRFSimpleFoam case. A parallel run on two cores and a serial run give very similar results, on 4 cores the results are significantly different. The problems can mainly be observed at extreme off-design operating points, nevertheless I think that there could be a problem somewhere in the communication between the different processor patches.


Steps To ReproduceI start all the simulations from time step 6000, do one iteration and write out different fields. The serial and the parallel run give different UrelEqn().H1() and UrelEqn().A() values at the processor patch. fvc::div(phi, Urel) and fvc::grad(p) seem to be correct. The problem can be reproduced with the SFCDV and the linearUpwindV schemes, I didn't test further schemes.
Additional InformationCase could be provided.
TagsNo tags attached.

Activities

dhora

2013-02-13 14:37

reporter  

pEqn.H (1,410 bytes)   

dhora

2013-02-13 14:37

reporter  

UrelEqn.H (711 bytes)   

dhora

2013-02-13 14:40

reporter  

png.tar.gz (906,211 bytes)

dhora

2013-02-13 14:41

reporter  

system.tar.gz (2,295 bytes)

dhora

2013-02-13 14:42

reporter  

boundary.gz (664 bytes)

wyldckat

2013-02-13 20:34

updater   ~0001911

Two questions:
1. How did you post-process the results?
2. What did the residuals look like or what values did they have for each run?

dhora

2013-02-13 22:16

reporter  

SRFSimpleFoamTest_p.log (4,389 bytes)   

dhora

2013-02-13 22:16

reporter  

SRFSimpleFoamTest_s.log (4,177 bytes)   

dhora

2013-02-13 22:22

reporter   ~0001914

The procedure was as follows for the parallel run:

decomposePar
SRFSimpleFoam: 1 Iteration
reconstructPar
Post-processing with ParaView

and the serial run:

SRFSimpleFoam: 1 Iteration
Post-processing with ParaView

dhora

2013-02-14 20:37

reporter   ~0001917

Last edited: 2013-02-14 20:49

turbulence->divDevReff(Urel) is responsible for the differences. And there is no difference between parallel and serial runs if I switch off the turbulence in constant/RASProperties.

user19

2013-02-16 10:25

  ~0001918

Hi David

Have you compared the results of the steady state solution?

dhora

2013-02-19 14:35

reporter   ~0001930

Hi Michael

Do you mean after the single iteration or if I continue the simulation? It's difficult to get a completely converged solution for such an operating point. The impeller efficiency fluctuates around a mean value which is 2% higher in the parallel run. I didn't expect such a difference.

user4

2013-02-19 17:29

  ~0001932

A possible cause for any differences is the different behaviour of the linear solvers in parallel and serial. They are only guaranteed to give the same result up to the residual tolerance you've set. This can easily accumulate.

Does the difference get less if you tighten the tolerance?

dhora

2013-02-20 10:24

reporter   ~0001935

Last edited: 2013-02-21 12:18

I have to correct my statement concerning the turbulence. The difference was there but very small due to the laminar/eddy viscosity ratio. However, there is a problem in the laplacian term. I wrote the following 3 terms into volVectorFields directly before the Urel equation system is being assembled:

fvc::div(phi, Urel)
fvc::laplacian(turbulence->nuEff(), Urel)
fvc::div(turbulence->nuEff()*dev(T(fvc::grad(Urel))))

fvc::laplacian(turbulence->nuEff(), Urel) gives different values in serial and parallel runs. That was done with the laminar RASModel. I think that excludes any problems with tolerances.

dhora

2013-03-06 11:48

reporter   ~0001959

The problem can be reproduced with the airFoil2D case:

1) Do 2000 iterations with simpleFoam (serial run)
2) Execute the uploaded divDevReff tool (serial run and parallel run on 16 cores decomposed with scotch decomposition)

dhora

2013-03-06 11:48

reporter  

airFoil2D.tar.gz (303,359 bytes)

dhora

2013-03-07 08:22

reporter   ~0001961

fvc::snGrad is affected as well

dhora

2013-03-12 12:58

reporter   ~0001981

Last edited: 2013-03-12 13:14

internal faces use nonOrthDeltaCoeffs in snGrad, processorBoundary faces use deltaCoeffs.

I think that's wrong and should be changed if there's no special reason for this inconsistent implementation.

henry

2013-03-12 13:35

manager   ~0001982

Thank you for spending the time to analyse this issue. The problem arose when we introduced the distinction between orthogonal and non-orthogonal deltaCoeffs which was needed to avoid the growing complexity in converting between the two as needed. I am now working on the best way forward, possible introducing the particular deltaCoeffs to be used in the boundary snGrad as an argument.

Will report back when this work is complete.

Thanks again for reporting and analysing this issue.

henry

2013-03-14 15:29

manager   ~0002008

Resolved in OpenFOAM-2.2.x by commit 1573905e175b5f879468eb57931ff2b22e857e01

Issue History

Date Modified Username Field Change
2013-02-13 14:37 dhora New Issue
2013-02-13 14:37 dhora File Added: pEqn.H
2013-02-13 14:37 dhora File Added: UrelEqn.H
2013-02-13 14:40 dhora File Added: png.tar.gz
2013-02-13 14:41 dhora File Added: system.tar.gz
2013-02-13 14:42 dhora File Added: boundary.gz
2013-02-13 20:34 wyldckat Note Added: 0001911
2013-02-13 22:16 dhora File Added: SRFSimpleFoamTest_p.log
2013-02-13 22:16 dhora File Added: SRFSimpleFoamTest_s.log
2013-02-13 22:22 dhora Note Added: 0001914
2013-02-14 20:37 dhora Note Added: 0001917
2013-02-14 20:48 dhora Note Edited: 0001917
2013-02-14 20:49 dhora Note Edited: 0001917
2013-02-16 10:25 user19 Note Added: 0001918
2013-02-19 14:35 dhora Note Added: 0001930
2013-02-19 17:29 user4 Note Added: 0001932
2013-02-20 10:24 dhora Note Added: 0001935
2013-02-21 12:18 dhora Note Edited: 0001935
2013-03-06 11:48 dhora Note Added: 0001959
2013-03-06 11:48 dhora File Added: airFoil2D.tar.gz
2013-03-07 08:22 dhora Note Added: 0001961
2013-03-12 12:58 dhora Note Added: 0001981
2013-03-12 13:13 dhora Note Edited: 0001981
2013-03-12 13:13 dhora Note Edited: 0001981
2013-03-12 13:14 dhora Note Edited: 0001981
2013-03-12 13:35 henry Note Added: 0001982
2013-03-14 15:29 henry Note Added: 0002008
2013-03-14 15:29 henry Status new => resolved
2013-03-14 15:29 henry Resolution open => fixed
2013-03-14 15:29 henry Assigned To => henry