View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0002516OpenFOAMBugpublic2017-03-29 14:522019-04-04 19:16
ReporterShorty Assigned Tohenry  
PrioritynormalSeveritytweakReproducibilityalways
Status resolvedResolutionfixed 
PlatformGNU/LinuxOSUbuntuOS Version14.04
Fixed in Versiondev 
Summary0002516: The more outer loops, the slower the outer loop calculation gets
DescriptionHey everybody,

I am not sure if this is a bug or something that can be explained in a mathematical point of view. However, based on the studies that I did the last few days I will just let you know that there might be a big problem for solid mechanics calculation in particular the speed of the outer loops. In fact, I build my own solver in which I realized this problem but I can reproduce it with the normal solidDisplacementFoam too (the simple case is attached).

What did I realized:
-----------------------------------------------------------------------------
I realized that the more outer loops I have, the slower the outer loop calculation gets. In particular, I found that solving the system is the point where the solver needs more effort from outer loop to outer loop. To analyze it, I turned off the solver performance and added the cpu clock time before and after the DEqn.solve() function and output the result for each 50 outer loop. I performed more than 25000 outer loops. The time to calculate 50 outer loops increased linear. This lead to a parabolic profile in the whole calculation duration. However, if we would stop after 100 outer loops and go on in time, this behavior cannot be analyzed/realized well.

Example given:
-----------------------------------------------------------------------------
The first 50 outer loops of one particular time step are performed with 0.1s
Further 50 outer loops (after 20000 finished outer loops) took almost 6.8s.

I would expect that 50 outer loops at the beginning should take the same amount than 50 outer loops after 10000 outer loops. Of course, if the linear system would need more and more inner iterations then the time would increase to a special threshold after we reach 1000 inner iterations. I could understand this based on the fact that the matrix gets (maybe) poorer and thus more inner iterations are needed. However this is not the case.

In order to prevent this, I limited the solver to 100 inner iterations (maxIter). I could not find the particular code in which we need more time.

I would be very interested what causes this behavior.
Tobi
Steps To Reproduce- insert the following line into the solidDisplacementFoam:

                initialResidual = DEqn.solve().max().initialResidual();

                if (show == 50)
                {
                    Info<< iCorr << " "
                        << runTime.elapsedCpuTime() << " "
                        << runTime.elapsedCpuTime() - oldCpu << " "
                        << initialResidual
                        << endl;
                    show = 0;
                    oldCpu = runTime.elapsedCpuTime();
                }
                  
                show++;


- Recompile
- Run the attached solidDisplacementFoam case
Additional InformationI think I realized this behavior of outer loop slow down even in normal solver.
TagsNo tags attached.

Activities

Shorty

2017-03-29 14:52

reporter  

2DBeam.tar.gz (228,550 bytes)

Shorty

2017-03-29 14:53

reporter  

outerCorrectiondt.eps (37,949 bytes)   
outerCorrectiondt.eps (37,949 bytes)   

Shorty

2017-03-29 14:53

reporter  

outerCorrectionCPUTime.png (5,300 bytes)   
outerCorrectionCPUTime.png (5,300 bytes)   

Shorty

2017-03-29 14:55

reporter   ~0007992

Maybe a few words to the two pictures:

outerCorrectiondt.eps
-----------------------------------------------------------------------
It shows the time needed to solve 50 outer loops. It increases linearly
The more outer loops the slower is the outer loop calculation

outerCorrectionCPUTime.png
-----------------------------------------------------------------------
It is the accumulated time for all outer loops. It is clear that if the
single outer loop calculation increases linearly that the global time
increase parabolic

Shorty

2017-04-01 10:28

reporter  

PimpleFoamOuterCorrections.pdf (20,166 bytes)

Shorty

2017-04-01 10:28

reporter  

PimpleFoamOuterCorrectionsPressure.pdf (1,741,938 bytes)

Shorty

2017-04-01 10:30

reporter  

pitzDaily.tar.gz (187,315 bytes)

Shorty

2017-04-01 10:33

reporter   ~0008012

Hey together,

I made the same analyze now with the pimpleFoam case and got the same behavior. Okay I might check it with a bit under-relaxation but I guess it will give me the same linear increase of CPU time for the outer corrections. The residuals should be better I guess. Any glue why this problem exist?

henry

2017-04-01 18:40

manager   ~0008013

I am unable to reproduce this behaviour: when I run pimlpeFoam on the pitzDaily case and change the number of outer correctors from 1 to 128 the time taken increases slightly less than linearly as expected.

This is a user-support request.

Shorty

2017-04-01 19:19

reporter   ~0008014

If you want to reproduce it you have to use 50000 outer loops. You may ask why we need 50.000 ... not for fluid mechanics but for solid mechanics it can happen. I refer to Philip Cardiff and his work.

If you are not interested, you can close it again.

Shorty

2017-04-03 20:35

reporter   ~0008016

Hey henry,

I got the point. Compiled FOAM for profiling and the result is: (just the interesting thing):

 26.38 9.20 9.20 20010 0.46 0.46 Foam::List<Foam::token>::~List()
 19.59 16.03 6.83 60028 0.11 0.11 Foam::List<Foam::SolverPerformance<Foam::Vector<double> > >::setSize(int)


Based on the fact that we store each iteration in the solverPerformanceDict (dataTemplates.C) the function that appends the new result (List.C -> setSize(int)) is very slow and extreme limiting (no USEMEMCPY activated). I will check how it behaves with usememcpy activated.

Shorty

2017-04-03 21:00

reporter   ~0008017

Okay,

after activating USEMEMCPY, everything is very fast! Thats what I expected. So no bug, no tweak, nothing needed.

Sorry for bothering you.

chris

2017-06-30 12:43

manager   ~0008312

Reporter says there is no issue

henry

2019-04-04 19:16

manager   ~0010408

Resolved by commit 2f3d47ad7e54a537758666a561809a02946d34f2

Issue History

Date Modified Username Field Change
2017-03-29 14:52 Shorty New Issue
2017-03-29 14:52 Shorty File Added: 2DBeam.tar.gz
2017-03-29 14:52 Shorty Tag Attached: outer loops slow down
2017-03-29 14:53 Shorty File Added: outerCorrectiondt.eps
2017-03-29 14:53 Shorty File Added: outerCorrectionCPUTime.png
2017-03-29 14:55 Shorty Note Added: 0007992
2017-04-01 10:28 Shorty File Added: PimpleFoamOuterCorrections.pdf
2017-04-01 10:28 Shorty File Added: PimpleFoamOuterCorrectionsPressure.pdf
2017-04-01 10:30 Shorty File Added: pitzDaily.tar.gz
2017-04-01 10:33 Shorty Note Added: 0008012
2017-04-01 18:40 henry Assigned To => henry
2017-04-01 18:40 henry Status new => closed
2017-04-01 18:40 henry Resolution open => no change required
2017-04-01 18:40 henry Note Added: 0008013
2017-04-01 19:19 Shorty Status closed => feedback
2017-04-01 19:19 Shorty Resolution no change required => reopened
2017-04-01 19:19 Shorty Note Added: 0008014
2017-04-03 20:35 Shorty Note Added: 0008016
2017-04-03 20:35 Shorty Status feedback => assigned
2017-04-03 21:00 Shorty Note Added: 0008017
2017-06-30 12:43 chris Status assigned => closed
2017-06-30 12:43 chris Resolution reopened => fixed
2017-06-30 12:43 chris Note Added: 0008312
2018-07-10 11:20 administrator Tag Detached: outer loops slow down
2019-04-04 19:16 henry Status closed => resolved
2019-04-04 19:16 henry Fixed in Version => dev
2019-04-04 19:16 henry Note Added: 0010408