View Issue Details

IDProjectCategoryView StatusLast Update
0002962OpenFOAMBugpublic2018-06-28 09:06
Reportershildenbrand Assigned Towill  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformHP Z640OSDebianOS Version9.4
Product Versiondev 
Fixed in Versiondev 
Summary0002962: Refresh button in Paraview (using paraFoam) does not work
DescriptionWhen postprocessing with Paraview (using paraFoam with OpenFOAM reader) pressing the refresh button does not load newly created timesteps.
This functionality existed in a prior version of the OpenFOAM reader and is very useful when postprocessing a big transient case.
Without this funtionality paraview has to be closed an re-opened which can be very time-consuming.
Steps To ReproduceRun tutorial case, e.g., ./tutorials/incompressible/simpleFoam/windAroundBuildings, start paraFoam after time=50 has been written and press reload after time=100 has been written. The new timestep is not read in.
TagsNo tags attached.

Activities

henry

2018-05-29 11:17

manager   ~0009665

I use the "Refresh Times" button several time a day and just used it now without any problems.

chris

2018-05-29 11:21

manager   ~0009666

I just ran the example with windAroundBuildings using:
- openfoam-dev pack, version 20180527
- paraviewopenfoam54, version 0-3

It worked fine, I cannot reproduce the issue.

chris

2018-05-29 11:44

manager   ~0009667

I can reproduce the issue if I start paraFoam BEFORE time=50 has been written and press reload after time=50 has been written.

henry

2018-05-29 11:48

manager   ~0009668

I cannot reproduce the problem even if I start paraFoam BEFORE time=50 has been written and press reload after time=50 has been written; time=50 loads fine.

shildenbrand

2018-05-29 12:02

reporter   ~0009669

I use locally compiled version of both. I'll check the compilations again to see if there are any errors.

will

2018-05-29 18:19

manager   ~0009670

Go to "View > Animation View", and set "Mode" in the new panel to "Snap To TimeSteps". Does that fix it?

shildenbrand

2018-05-30 07:28

reporter   ~0009671

My compilation of the PVReader shows many warnings (see file attached) but I am not sure if this is connected to my problem. If I provide "stdbool.h" manually the problem of not being able to refresh persists.
In "View > Animation View", "Mode" was already set to "Snap To TimeSteps", so this doesn't fix it.
compile_PVReader (576,639 bytes)

shildenbrand

2018-05-30 07:33

reporter   ~0009672

Additional information:
If I leave Paraview open, close the loaded case, and re-open it (when there are more timesteps available), I still only see the number of timesteps available at the first start of Paraview.

When using the builtin-reader (.foam extension), refresh works fine

will

2018-06-14 16:23

manager   ~0009766

The warnings generated when compiling the reader module are a result of paraview's code. They are irritating, but they do not affect functionality.

I have built on debian 9.4 in a virtual machine, and I cannot reproduce your issue.

shildenbrand

2018-06-15 06:44

reporter   ~0009771

Thank you for testing this in my environment. I made some more tests this morning. Could you please try to reproduce the issue when you set

fileHandler collated;

in ./OpenFOAM-dev/etc/controlDict and do a wmRefresh?

We have set collated as standard, I tried this morning the uncollated filehandler and now the refresh button works again.

will

2018-06-15 10:19

manager   ~0009775

OK, the fact that you are using collated file handling was the crucial bit of information. I can reproduce this now. The file handler is keeping a cache of the times and this cache is not being rebuilt when you hit refresh. If the first half of the if statement in masterUncollatedFileOperation::findTimes is removed then the problem goes away. That's not a solution, though. @MattijsJ could you please advise how this should be fixed?

MattijsJ

2018-06-19 11:52

reporter   ~0009795

This is not easily fixed. The masterUncollated and collated file handlers cache times to avoid excessive directory reading. They internally keep track of any additional time directories being added. The paraFoam is external to this so does not know that time directories have been added.

For now as a workaround run paraFoam with uncollated. In bash:

FOAM_FILEHANDLER=uncollated paraFoam

shildenbrand

2018-06-19 12:15

reporter   ~0009796

The workaround works for me. Thank you.

will

2018-06-20 12:46

manager   ~0009805

We can't expect users to know of this limitation and modify their execution of paraFoam accordingly. I would have thought that we need to either (a) allow paraFoam/vtkPVFoam to disable the caching and live with the additional read-times, or (b) add a method so that paraFoam/vtkPVFoam can clear the cache when the user hits refresh. @Mattijs, what is your preference, or do you have an alternative?

MattijsJ

2018-06-22 10:43

reporter   ~0009810

A similar (but different) issue lives inside e.g. decomposePar (and that thermo field writing business) where we need to switch off the threading. In fact we don't need to switch off threading - we just need to be able to wait for everything to be written.

Similar to your suggestion I would add a 'clearOut' or 'flush' method which would clear the cache and guarantee any files to be written. This would cover both the thread business and time caching issues.

MattijsJ

2018-06-26 20:49

reporter  

flush.patch (15,350 bytes)   
diff --git a/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C b/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C
index d2a3ca2c2..df79d50c0 100644
--- a/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C
+++ b/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C
@@ -50,12 +50,6 @@ Foam::twoPhaseMixtureThermo::twoPhaseMixtureThermo
     thermo1_(nullptr),
     thermo2_(nullptr)
 {
-    // Note: we're writing files to be read in immediately afterwards.
-    //       Avoid any thread-writing problems.
-    float bufSz =
-        fileOperations::collatedFileOperation::maxThreadFileBufferSize;
-    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
-
     {
         volScalarField T1
         (
@@ -86,9 +80,9 @@ Foam::twoPhaseMixtureThermo::twoPhaseMixtureThermo
         T2.write();
     }
 
-    fileOperations::collatedFileOperation::maxThreadFileBufferSize =
-        bufSz;
-
+    // Note: we're writing files to be read in immediately afterwards.
+    //       Avoid any thread-writing problems.
+    fileHandler().flush();
 
     thermo1_ = rhoThermo::New(U.mesh(), phase1Name());
     thermo2_ = rhoThermo::New(U.mesh(), phase2Name());
diff --git a/applications/utilities/mesh/manipulation/setSet/setSet.C b/applications/utilities/mesh/manipulation/setSet/setSet.C
index f425a201f..70d8d3bc8 100644
--- a/applications/utilities/mesh/manipulation/setSet/setSet.C
+++ b/applications/utilities/mesh/manipulation/setSet/setSet.C
@@ -48,7 +48,6 @@ Description
 #include "faceZoneSet.H"
 #include "pointZoneSet.H"
 #include "timeSelector.H"
-#include "collatedFileOperation.H"
 
 #include <stdio.h>
 
@@ -349,6 +348,8 @@ void removeZone
         zones.setSize(zones.size()-1);
         zones.clearAddressing();
         zones.write();
+        // Force flushing so we know it has finished writing
+        fileHandler().flush();
     }
 }
 
@@ -603,6 +604,8 @@ bool doCommand
                     currentSet.instance() = mesh.time().timeName();
                 }
                 currentSet.write();
+                // Make sure writing is finished
+                fileHandler().flush();
             }
         }
     }
@@ -809,8 +812,6 @@ int main(int argc, char *argv[])
     // Specific to topoSet/setSet: quite often we want to block upon writing
     // a set so we can immediately re-read it. So avoid use of threading
     // for set writing.
-    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
-
     timeSelector::addOptions(true, false);
     #include "addRegionOption.H"
     argList::addBoolOption("noVTK", "do not write VTK files");
diff --git a/applications/utilities/mesh/manipulation/topoSet/topoSet.C b/applications/utilities/mesh/manipulation/topoSet/topoSet.C
index e4aa0eb77..10e1718fb 100644
--- a/applications/utilities/mesh/manipulation/topoSet/topoSet.C
+++ b/applications/utilities/mesh/manipulation/topoSet/topoSet.C
@@ -40,7 +40,6 @@ Description
 #include "faceZoneSet.H"
 #include "pointZoneSet.H"
 #include "IOdictionary.H"
-#include "collatedFileOperation.H"
 
 using namespace Foam;
 
@@ -85,6 +84,7 @@ void removeZone
         zones.setSize(zones.size()-1);
         zones.clearAddressing();
         zones.write();
+        fileHandler().flush();
     }
 }
 
@@ -193,11 +193,6 @@ polyMesh::readUpdateState meshReadUpdate(polyMesh& mesh)
 
 int main(int argc, char *argv[])
 {
-    // Specific to topoSet/setSet: quite often we want to block upon writing
-    // a set so we can immediately re-read it. So avoid use of threading
-    // for set writing.
-    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
-
     timeSelector::addOptions(true, false);
     #include "addDictOption.H"
     #include "addRegionOption.H"
@@ -302,6 +297,7 @@ int main(int argc, char *argv[])
                     // Synchronize for coupled patches.
                     if (!noSync) currentSet().sync(mesh);
                     currentSet().write();
+                    fileHandler().flush();
                 }
                 break;
 
@@ -336,6 +332,7 @@ int main(int argc, char *argv[])
                     // Synchronize for coupled patches.
                     if (!noSync) currentSet().sync(mesh);
                     currentSet().write();
+                    fileHandler().flush();
                 }
                 break;
 
@@ -343,12 +340,14 @@ int main(int argc, char *argv[])
                     Info<< "    Clearing " << currentSet().type() << endl;
                     currentSet().clear();
                     currentSet().write();
+                    fileHandler().flush();
                 break;
 
                 case topoSetSource::INVERT:
                     Info<< "    Inverting " << currentSet().type() << endl;
                     currentSet().invert(currentSet().maxSize(mesh));
                     currentSet().write();
+                    fileHandler().flush();
                 break;
 
                 case topoSetSource::REMOVE:
diff --git a/applications/utilities/parallelProcessing/decomposePar/decomposePar.C b/applications/utilities/parallelProcessing/decomposePar/decomposePar.C
index 92e2950f5..642e8ccb8 100644
--- a/applications/utilities/parallelProcessing/decomposePar/decomposePar.C
+++ b/applications/utilities/parallelProcessing/decomposePar/decomposePar.C
@@ -104,7 +104,6 @@ Usage
 #include "pointFieldDecomposer.H"
 #include "lagrangianFieldDecomposer.H"
 #include "decompositionModel.H"
-#include "collatedFileOperation.H"
 
 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
 
@@ -459,12 +458,6 @@ int main(int argc, char *argv[])
         // Decompose the mesh
         if (!decomposeFieldsOnly)
         {
-            // Disable buffering when writing mesh since we need to read
-            // it later on when decomposing the fields
-            float bufSz =
-                fileOperations::collatedFileOperation::maxThreadFileBufferSize;
-            fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
-
             mesh.decomposeMesh(dictIO.objectPath());
 
             mesh.writeDecomposition(decomposeSets);
@@ -521,8 +514,7 @@ int main(int argc, char *argv[])
                     << endl;
             }
 
-            fileOperations::collatedFileOperation::maxThreadFileBufferSize =
-                bufSz;
+            fileHandler().flush();
         }
 
 
diff --git a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C
index 6a24a189c..684fab66d 100644
--- a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C
+++ b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C
@@ -97,7 +97,8 @@ int Foam::vtkPVFoam::setTime(int nRequest, const double requestTimes[])
 {
     Time& runTime = dbPtr_();
 
-    // Get times list
+    // Get times list. Flush first to force refresh.
+    fileHandler().flush();
     instantList Times = runTime.times();
 
     int nearestIndex = timeIndex_;
@@ -248,11 +249,6 @@ Foam::vtkPVFoam::vtkPVFoam
 
     fileName FileName(vtkFileName);
 
-    // Make sure not to use the threaded version - it does not like
-    // being loaded as a shared library - static cleanup order is problematic.
-    // For now just disable the threaded writer.
-    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
-
     // avoid argList and get rootPath/caseName directly from the file
     fileName fullCasePath(FileName.path());
 
@@ -575,6 +571,8 @@ double* Foam::vtkPVFoam::findTimes(int& nTimeSteps)
     if (dbPtr_.valid())
     {
         Time& runTime = dbPtr_();
+        // Get times list. Flush first to force refresh.
+        fileHandler().flush();
         instantList timeLst = runTime.times();
 
         // find the first time for which this mesh appears to exist
diff --git a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C
index 2f1c0add6..dba80d9a0 100644
--- a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C
+++ b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C
@@ -192,6 +192,9 @@ void Foam::vtkPVFoam::updateInfoLagrangian
 
     // Generate a list of lagrangian clouds across all times
     HashSet<fileName> cloudDirs;
+
+    // Get times list. Flush first to force refresh.
+    fileHandler().flush();
     instantList times = dbPtr_().times();
     forAll(times, timei)
     {
@@ -706,6 +709,8 @@ void Foam::vtkPVFoam::updateInfoLagrangianFields()
     // set. ParaView will display "(partial)" after field names that only apply
     // to some of the clouds.
     const arrayRange& range = arrayRangeLagrangian_;
+
+    fileHandler().flush();
     for (label partId = range.start(); partId < range.end(); ++ partId)
     {
         const instantList times = dbPtr_().times();
diff --git a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C
index d79398601..69b76444b 100644
--- a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C
+++ b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C
@@ -31,7 +31,6 @@ License
 #include "Time.H"
 #include "patchZones.H"
 #include "OStringStream.H"
-#include "collatedFileOperation.H"
 
 // VTK includes
 #include "vtkDataArraySelection.h"
@@ -171,12 +170,6 @@ Foam::vtkPVblockMesh::vtkPVblockMesh
             << FileName << endl;
     }
 
-    // Make sure not to use the threaded version - it does not like
-    // being loaded as a shared library - static cleanup order is problematic.
-    // For now just disable the threaded writer.
-    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
-
-
     // avoid argList and get rootPath/caseName directly from the file
     fileName fullCasePath(fileName(FileName).path());
 
diff --git a/etc/controlDict b/etc/controlDict
index d927ea413..4179091ae 100644
--- a/etc/controlDict
+++ b/etc/controlDict
@@ -58,7 +58,7 @@ OptimisationSwitches
 
     //- Parallel IO file handler
     //  uncollated (default), collated or masterUncollated
-    fileHandler uncollated;
+    fileHandler collated;
 
     //- collated: thread buffer size for queued file writes.
     //  If set to 0 or not sufficient for the file size threading is not used.
diff --git a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C
index 614e846af..5abaf3c70 100644
--- a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C
+++ b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C
@@ -608,6 +608,18 @@ bool Foam::fileOperations::collatedFileOperation::writeObject
     }
 }
 
+void Foam::fileOperations::collatedFileOperation::flush() const
+{
+    if (debug)
+    {
+        Pout<< "collatedFileOperation::flush : clearing and waiting for thread"
+            << endl;
+    }
+    masterUncollatedFileOperation::flush();
+    // Wait for thread to finish (note: also removes thread)
+    writer_.waitAll();
+}
+
 
 Foam::word Foam::fileOperations::collatedFileOperation::processorsDir
 (
diff --git a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H
index f1acda477..4e65454ce 100644
--- a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H
+++ b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H
@@ -155,6 +155,9 @@ public:
 
         // Other
 
+            //- Forcibly wait until all output done. Flush any cached data
+            virtual void flush() const;
+
             //- Actual name of processors dir
             virtual word processorsDir(const IOobject&) const;
 
diff --git a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C
index 8c8640ff0..65b39843d 100644
--- a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C
+++ b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C
@@ -982,6 +982,17 @@ Foam::label Foam::fileOperation::nProcs
 }
 
 
+void Foam::fileOperation::flush() const
+{
+    if (debug)
+    {
+        Pout<< "fileOperation::flush : clearing processor directories cache"
+            << endl;
+    }
+    procsDirs_.clear();
+}
+
+
 Foam::fileName Foam::fileOperation::processorsCasePath
 (
     const IOobject& io,
diff --git a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H
index 2ea2832a7..84c928b57 100644
--- a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H
+++ b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H
@@ -485,6 +485,9 @@ public:
             virtual void setTime(const Time&) const
             {}
 
+            //- Forcibly wait until all output done. Flush any cached data
+            virtual void flush() const;
+
             //- Generate path (like io.path) from root+casename with any
             //  'processorXXX' replaced by procDir (usually 'processsors')
             fileName processorsCasePath
diff --git a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C
index 25fc3c811..0f0675e30 100644
--- a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C
+++ b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C
@@ -2532,6 +2532,13 @@ Foam::fileOperations::masterUncollatedFileOperation::NewOFstream
 }
 
 
+void Foam::fileOperations::masterUncollatedFileOperation::flush() const
+{
+    fileOperation::flush();
+    times_.clear();
+}
+
+
 Foam::label Foam::fileOperations::masterUncollatedFileOperation::addWatch
 (
     const fileName& fName
diff --git a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H
index 15ee260a0..93862fc2a 100644
--- a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H
+++ b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H
@@ -752,6 +752,9 @@ public:
             //- Callback for time change
             virtual void setTime(const Time&) const;
 
+            //- Forcibly wait until all output done. Flush any cached data
+            virtual void flush() const;
+
             //- Return cached times
             const HashPtrTable<instantList>& times() const
             {
flush.patch (15,350 bytes)   

MattijsJ

2018-06-26 20:49

reporter   ~0009832

Attached a patch for dev which adds a 'flush' method to all fileOperations. It calls this from the PV* readers and replaces all instances of maxThreadXXX manipulation. The only testing I've done so far is the collated+RefreshTimes button so it needs some more testing.

will

2018-06-28 09:06

manager   ~0009840

Thanks for the patch @MattijsJ. The file handler tests run fine with this, the whole test loop runs collated, and I've run a couple of additional cases successfully, too, so I'm happy it's not causing additional issues. Pushed as commit 025e4837.

Issue History

Date Modified Username Field Change
2018-05-29 10:26 shildenbrand New Issue
2018-05-29 11:17 henry Note Added: 0009665
2018-05-29 11:21 chris Note Added: 0009666
2018-05-29 11:44 chris Note Added: 0009667
2018-05-29 11:48 henry Note Added: 0009668
2018-05-29 12:02 shildenbrand Note Added: 0009669
2018-05-29 18:19 will Note Added: 0009670
2018-05-30 07:28 shildenbrand File Added: compile_PVReader
2018-05-30 07:28 shildenbrand Note Added: 0009671
2018-05-30 07:33 shildenbrand Note Added: 0009672
2018-06-14 16:23 will Note Added: 0009766
2018-06-15 06:44 shildenbrand Note Added: 0009771
2018-06-15 10:19 will Note Added: 0009775
2018-06-19 11:52 MattijsJ Note Added: 0009795
2018-06-19 12:15 shildenbrand Note Added: 0009796
2018-06-20 12:46 will Note Added: 0009805
2018-06-22 10:43 MattijsJ Note Added: 0009810
2018-06-26 20:49 MattijsJ File Added: flush.patch
2018-06-26 20:49 MattijsJ Note Added: 0009832
2018-06-28 09:06 will Assigned To => will
2018-06-28 09:06 will Status new => resolved
2018-06-28 09:06 will Resolution open => fixed
2018-06-28 09:06 will Fixed in Version => dev
2018-06-28 09:06 will Note Added: 0009840