2018-08-17 14:23 BST

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0002962OpenFOAM[All Projects] Bugpublic2018-06-28 09:06
Reportershildenbrand 
Assigned Towill 
PrioritynormalSeverityminorReproducibilityalways
StatusresolvedResolutionfixed 
PlatformHP Z640OSDebianOS Version9.4
Product Versiondev 
Target VersionFixed in Versiondev 
Summary0002962: Refresh button in Paraview (using paraFoam) does not work
DescriptionWhen postprocessing with Paraview (using paraFoam with OpenFOAM reader) pressing the refresh button does not load newly created timesteps.
This functionality existed in a prior version of the OpenFOAM reader and is very useful when postprocessing a big transient case.
Without this funtionality paraview has to be closed an re-opened which can be very time-consuming.
Steps To ReproduceRun tutorial case, e.g., ./tutorials/incompressible/simpleFoam/windAroundBuildings, start paraFoam after time=50 has been written and press reload after time=100 has been written. The new timestep is not read in.
TagsNo tags attached.
Attached Files
  • ? file icon compile_PVReader (576,639 bytes) 2018-05-30 07:28
  • patch file icon flush.patch (15,350 bytes) 2018-06-26 20:49 -
    diff --git a/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C b/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C
    index d2a3ca2c2..df79d50c0 100644
    --- a/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C
    +++ b/applications/solvers/multiphase/compressibleInterFoam/twoPhaseMixtureThermo/twoPhaseMixtureThermo.C
    @@ -50,12 +50,6 @@ Foam::twoPhaseMixtureThermo::twoPhaseMixtureThermo
         thermo1_(nullptr),
         thermo2_(nullptr)
     {
    -    // Note: we're writing files to be read in immediately afterwards.
    -    //       Avoid any thread-writing problems.
    -    float bufSz =
    -        fileOperations::collatedFileOperation::maxThreadFileBufferSize;
    -    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
    -
         {
             volScalarField T1
             (
    @@ -86,9 +80,9 @@ Foam::twoPhaseMixtureThermo::twoPhaseMixtureThermo
             T2.write();
         }
     
    -    fileOperations::collatedFileOperation::maxThreadFileBufferSize =
    -        bufSz;
    -
    +    // Note: we're writing files to be read in immediately afterwards.
    +    //       Avoid any thread-writing problems.
    +    fileHandler().flush();
     
         thermo1_ = rhoThermo::New(U.mesh(), phase1Name());
         thermo2_ = rhoThermo::New(U.mesh(), phase2Name());
    diff --git a/applications/utilities/mesh/manipulation/setSet/setSet.C b/applications/utilities/mesh/manipulation/setSet/setSet.C
    index f425a201f..70d8d3bc8 100644
    --- a/applications/utilities/mesh/manipulation/setSet/setSet.C
    +++ b/applications/utilities/mesh/manipulation/setSet/setSet.C
    @@ -48,7 +48,6 @@ Description
     #include "faceZoneSet.H"
     #include "pointZoneSet.H"
     #include "timeSelector.H"
    -#include "collatedFileOperation.H"
     
     #include <stdio.h>
     
    @@ -349,6 +348,8 @@ void removeZone
             zones.setSize(zones.size()-1);
             zones.clearAddressing();
             zones.write();
    +        // Force flushing so we know it has finished writing
    +        fileHandler().flush();
         }
     }
     
    @@ -603,6 +604,8 @@ bool doCommand
                         currentSet.instance() = mesh.time().timeName();
                     }
                     currentSet.write();
    +                // Make sure writing is finished
    +                fileHandler().flush();
                 }
             }
         }
    @@ -809,8 +812,6 @@ int main(int argc, char *argv[])
         // Specific to topoSet/setSet: quite often we want to block upon writing
         // a set so we can immediately re-read it. So avoid use of threading
         // for set writing.
    -    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
    -
         timeSelector::addOptions(true, false);
         #include "addRegionOption.H"
         argList::addBoolOption("noVTK", "do not write VTK files");
    diff --git a/applications/utilities/mesh/manipulation/topoSet/topoSet.C b/applications/utilities/mesh/manipulation/topoSet/topoSet.C
    index e4aa0eb77..10e1718fb 100644
    --- a/applications/utilities/mesh/manipulation/topoSet/topoSet.C
    +++ b/applications/utilities/mesh/manipulation/topoSet/topoSet.C
    @@ -40,7 +40,6 @@ Description
     #include "faceZoneSet.H"
     #include "pointZoneSet.H"
     #include "IOdictionary.H"
    -#include "collatedFileOperation.H"
     
     using namespace Foam;
     
    @@ -85,6 +84,7 @@ void removeZone
             zones.setSize(zones.size()-1);
             zones.clearAddressing();
             zones.write();
    +        fileHandler().flush();
         }
     }
     
    @@ -193,11 +193,6 @@ polyMesh::readUpdateState meshReadUpdate(polyMesh& mesh)
     
     int main(int argc, char *argv[])
     {
    -    // Specific to topoSet/setSet: quite often we want to block upon writing
    -    // a set so we can immediately re-read it. So avoid use of threading
    -    // for set writing.
    -    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
    -
         timeSelector::addOptions(true, false);
         #include "addDictOption.H"
         #include "addRegionOption.H"
    @@ -302,6 +297,7 @@ int main(int argc, char *argv[])
                         // Synchronize for coupled patches.
                         if (!noSync) currentSet().sync(mesh);
                         currentSet().write();
    +                    fileHandler().flush();
                     }
                     break;
     
    @@ -336,6 +332,7 @@ int main(int argc, char *argv[])
                         // Synchronize for coupled patches.
                         if (!noSync) currentSet().sync(mesh);
                         currentSet().write();
    +                    fileHandler().flush();
                     }
                     break;
     
    @@ -343,12 +340,14 @@ int main(int argc, char *argv[])
                         Info<< "    Clearing " << currentSet().type() << endl;
                         currentSet().clear();
                         currentSet().write();
    +                    fileHandler().flush();
                     break;
     
                     case topoSetSource::INVERT:
                         Info<< "    Inverting " << currentSet().type() << endl;
                         currentSet().invert(currentSet().maxSize(mesh));
                         currentSet().write();
    +                    fileHandler().flush();
                     break;
     
                     case topoSetSource::REMOVE:
    diff --git a/applications/utilities/parallelProcessing/decomposePar/decomposePar.C b/applications/utilities/parallelProcessing/decomposePar/decomposePar.C
    index 92e2950f5..642e8ccb8 100644
    --- a/applications/utilities/parallelProcessing/decomposePar/decomposePar.C
    +++ b/applications/utilities/parallelProcessing/decomposePar/decomposePar.C
    @@ -104,7 +104,6 @@ Usage
     #include "pointFieldDecomposer.H"
     #include "lagrangianFieldDecomposer.H"
     #include "decompositionModel.H"
    -#include "collatedFileOperation.H"
     
     // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
     
    @@ -459,12 +458,6 @@ int main(int argc, char *argv[])
             // Decompose the mesh
             if (!decomposeFieldsOnly)
             {
    -            // Disable buffering when writing mesh since we need to read
    -            // it later on when decomposing the fields
    -            float bufSz =
    -                fileOperations::collatedFileOperation::maxThreadFileBufferSize;
    -            fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
    -
                 mesh.decomposeMesh(dictIO.objectPath());
     
                 mesh.writeDecomposition(decomposeSets);
    @@ -521,8 +514,7 @@ int main(int argc, char *argv[])
                         << endl;
                 }
     
    -            fileOperations::collatedFileOperation::maxThreadFileBufferSize =
    -                bufSz;
    +            fileHandler().flush();
             }
     
     
    diff --git a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C
    index 6a24a189c..684fab66d 100644
    --- a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C
    +++ b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoam.C
    @@ -97,7 +97,8 @@ int Foam::vtkPVFoam::setTime(int nRequest, const double requestTimes[])
     {
         Time& runTime = dbPtr_();
     
    -    // Get times list
    +    // Get times list. Flush first to force refresh.
    +    fileHandler().flush();
         instantList Times = runTime.times();
     
         int nearestIndex = timeIndex_;
    @@ -248,11 +249,6 @@ Foam::vtkPVFoam::vtkPVFoam
     
         fileName FileName(vtkFileName);
     
    -    // Make sure not to use the threaded version - it does not like
    -    // being loaded as a shared library - static cleanup order is problematic.
    -    // For now just disable the threaded writer.
    -    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
    -
         // avoid argList and get rootPath/caseName directly from the file
         fileName fullCasePath(FileName.path());
     
    @@ -575,6 +571,8 @@ double* Foam::vtkPVFoam::findTimes(int& nTimeSteps)
         if (dbPtr_.valid())
         {
             Time& runTime = dbPtr_();
    +        // Get times list. Flush first to force refresh.
    +        fileHandler().flush();
             instantList timeLst = runTime.times();
     
             // find the first time for which this mesh appears to exist
    diff --git a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C
    index 2f1c0add6..dba80d9a0 100644
    --- a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C
    +++ b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVFoam/vtkPVFoamUpdateInfo.C
    @@ -192,6 +192,9 @@ void Foam::vtkPVFoam::updateInfoLagrangian
     
         // Generate a list of lagrangian clouds across all times
         HashSet<fileName> cloudDirs;
    +
    +    // Get times list. Flush first to force refresh.
    +    fileHandler().flush();
         instantList times = dbPtr_().times();
         forAll(times, timei)
         {
    @@ -706,6 +709,8 @@ void Foam::vtkPVFoam::updateInfoLagrangianFields()
         // set. ParaView will display "(partial)" after field names that only apply
         // to some of the clouds.
         const arrayRange& range = arrayRangeLagrangian_;
    +
    +    fileHandler().flush();
         for (label partId = range.start(); partId < range.end(); ++ partId)
         {
             const instantList times = dbPtr_().times();
    diff --git a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C
    index d79398601..69b76444b 100644
    --- a/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C
    +++ b/applications/utilities/postProcessing/graphics/PVReaders/vtkPVblockMesh/vtkPVblockMesh.C
    @@ -31,7 +31,6 @@ License
     #include "Time.H"
     #include "patchZones.H"
     #include "OStringStream.H"
    -#include "collatedFileOperation.H"
     
     // VTK includes
     #include "vtkDataArraySelection.h"
    @@ -171,12 +170,6 @@ Foam::vtkPVblockMesh::vtkPVblockMesh
                 << FileName << endl;
         }
     
    -    // Make sure not to use the threaded version - it does not like
    -    // being loaded as a shared library - static cleanup order is problematic.
    -    // For now just disable the threaded writer.
    -    fileOperations::collatedFileOperation::maxThreadFileBufferSize = 0;
    -
    -
         // avoid argList and get rootPath/caseName directly from the file
         fileName fullCasePath(fileName(FileName).path());
     
    diff --git a/etc/controlDict b/etc/controlDict
    index d927ea413..4179091ae 100644
    --- a/etc/controlDict
    +++ b/etc/controlDict
    @@ -58,7 +58,7 @@ OptimisationSwitches
     
         //- Parallel IO file handler
         //  uncollated (default), collated or masterUncollated
    -    fileHandler uncollated;
    +    fileHandler collated;
     
         //- collated: thread buffer size for queued file writes.
         //  If set to 0 or not sufficient for the file size threading is not used.
    diff --git a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C
    index 614e846af..5abaf3c70 100644
    --- a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C
    +++ b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.C
    @@ -608,6 +608,18 @@ bool Foam::fileOperations::collatedFileOperation::writeObject
         }
     }
     
    +void Foam::fileOperations::collatedFileOperation::flush() const
    +{
    +    if (debug)
    +    {
    +        Pout<< "collatedFileOperation::flush : clearing and waiting for thread"
    +            << endl;
    +    }
    +    masterUncollatedFileOperation::flush();
    +    // Wait for thread to finish (note: also removes thread)
    +    writer_.waitAll();
    +}
    +
     
     Foam::word Foam::fileOperations::collatedFileOperation::processorsDir
     (
    diff --git a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H
    index f1acda477..4e65454ce 100644
    --- a/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H
    +++ b/src/OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H
    @@ -155,6 +155,9 @@ public:
     
             // Other
     
    +            //- Forcibly wait until all output done. Flush any cached data
    +            virtual void flush() const;
    +
                 //- Actual name of processors dir
                 virtual word processorsDir(const IOobject&) const;
     
    diff --git a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C
    index 8c8640ff0..65b39843d 100644
    --- a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C
    +++ b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.C
    @@ -982,6 +982,17 @@ Foam::label Foam::fileOperation::nProcs
     }
     
     
    +void Foam::fileOperation::flush() const
    +{
    +    if (debug)
    +    {
    +        Pout<< "fileOperation::flush : clearing processor directories cache"
    +            << endl;
    +    }
    +    procsDirs_.clear();
    +}
    +
    +
     Foam::fileName Foam::fileOperation::processorsCasePath
     (
         const IOobject& io,
    diff --git a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H
    index 2ea2832a7..84c928b57 100644
    --- a/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H
    +++ b/src/OpenFOAM/global/fileOperations/fileOperation/fileOperation.H
    @@ -485,6 +485,9 @@ public:
                 virtual void setTime(const Time&) const
                 {}
     
    +            //- Forcibly wait until all output done. Flush any cached data
    +            virtual void flush() const;
    +
                 //- Generate path (like io.path) from root+casename with any
                 //  'processorXXX' replaced by procDir (usually 'processsors')
                 fileName processorsCasePath
    diff --git a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C
    index 25fc3c811..0f0675e30 100644
    --- a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C
    +++ b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.C
    @@ -2532,6 +2532,13 @@ Foam::fileOperations::masterUncollatedFileOperation::NewOFstream
     }
     
     
    +void Foam::fileOperations::masterUncollatedFileOperation::flush() const
    +{
    +    fileOperation::flush();
    +    times_.clear();
    +}
    +
    +
     Foam::label Foam::fileOperations::masterUncollatedFileOperation::addWatch
     (
         const fileName& fName
    diff --git a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H
    index 15ee260a0..93862fc2a 100644
    --- a/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H
    +++ b/src/OpenFOAM/global/fileOperations/masterUncollatedFileOperation/masterUncollatedFileOperation.H
    @@ -752,6 +752,9 @@ public:
                 //- Callback for time change
                 virtual void setTime(const Time&) const;
     
    +            //- Forcibly wait until all output done. Flush any cached data
    +            virtual void flush() const;
    +
                 //- Return cached times
                 const HashPtrTable<instantList>& times() const
                 {
    
    patch file icon flush.patch (15,350 bytes) 2018-06-26 20:49 +

-Relationships
+Relationships

-Notes

~0009665

henry (manager)

I use the "Refresh Times" button several time a day and just used it now without any problems.

~0009666

chris (manager)

I just ran the example with windAroundBuildings using:
- openfoam-dev pack, version 20180527
- paraviewopenfoam54, version 0-3

It worked fine, I cannot reproduce the issue.

~0009667

chris (manager)

I can reproduce the issue if I start paraFoam BEFORE time=50 has been written and press reload after time=50 has been written.

~0009668

henry (manager)

I cannot reproduce the problem even if I start paraFoam BEFORE time=50 has been written and press reload after time=50 has been written; time=50 loads fine.

~0009669

shildenbrand (reporter)

I use locally compiled version of both. I'll check the compilations again to see if there are any errors.

~0009670

will (manager)

Go to "View > Animation View", and set "Mode" in the new panel to "Snap To TimeSteps". Does that fix it?

~0009671

shildenbrand (reporter)

My compilation of the PVReader shows many warnings (see file attached) but I am not sure if this is connected to my problem. If I provide "stdbool.h" manually the problem of not being able to refresh persists.
In "View > Animation View", "Mode" was already set to "Snap To TimeSteps", so this doesn't fix it.

~0009672

shildenbrand (reporter)

Additional information:
If I leave Paraview open, close the loaded case, and re-open it (when there are more timesteps available), I still only see the number of timesteps available at the first start of Paraview.

When using the builtin-reader (.foam extension), refresh works fine

~0009766

will (manager)

The warnings generated when compiling the reader module are a result of paraview's code. They are irritating, but they do not affect functionality.

I have built on debian 9.4 in a virtual machine, and I cannot reproduce your issue.

~0009771

shildenbrand (reporter)

Thank you for testing this in my environment. I made some more tests this morning. Could you please try to reproduce the issue when you set

fileHandler collated;

in ./OpenFOAM-dev/etc/controlDict and do a wmRefresh?

We have set collated as standard, I tried this morning the uncollated filehandler and now the refresh button works again.

~0009775

will (manager)

OK, the fact that you are using collated file handling was the crucial bit of information. I can reproduce this now. The file handler is keeping a cache of the times and this cache is not being rebuilt when you hit refresh. If the first half of the if statement in masterUncollatedFileOperation::findTimes is removed then the problem goes away. That's not a solution, though. @MattijsJ could you please advise how this should be fixed?

~0009795

MattijsJ (reporter)

This is not easily fixed. The masterUncollated and collated file handlers cache times to avoid excessive directory reading. They internally keep track of any additional time directories being added. The paraFoam is external to this so does not know that time directories have been added.

For now as a workaround run paraFoam with uncollated. In bash:

FOAM_FILEHANDLER=uncollated paraFoam

~0009796

shildenbrand (reporter)

The workaround works for me. Thank you.

~0009805

will (manager)

We can't expect users to know of this limitation and modify their execution of paraFoam accordingly. I would have thought that we need to either (a) allow paraFoam/vtkPVFoam to disable the caching and live with the additional read-times, or (b) add a method so that paraFoam/vtkPVFoam can clear the cache when the user hits refresh. @Mattijs, what is your preference, or do you have an alternative?

~0009810

MattijsJ (reporter)

A similar (but different) issue lives inside e.g. decomposePar (and that thermo field writing business) where we need to switch off the threading. In fact we don't need to switch off threading - we just need to be able to wait for everything to be written.

Similar to your suggestion I would add a 'clearOut' or 'flush' method which would clear the cache and guarantee any files to be written. This would cover both the thread business and time caching issues.

~0009832

MattijsJ (reporter)

Attached a patch for dev which adds a 'flush' method to all fileOperations. It calls this from the PV* readers and replaces all instances of maxThreadXXX manipulation. The only testing I've done so far is the collated+RefreshTimes button so it needs some more testing.

~0009840

will (manager)

Thanks for the patch @MattijsJ. The file handler tests run fine with this, the whole test loop runs collated, and I've run a couple of additional cases successfully, too, so I'm happy it's not causing additional issues. Pushed as commit 025e4837.
+Notes

-Issue History
Date Modified Username Field Change
2018-05-29 10:26 shildenbrand New Issue
2018-05-29 11:17 henry Note Added: 0009665
2018-05-29 11:21 chris Note Added: 0009666
2018-05-29 11:44 chris Note Added: 0009667
2018-05-29 11:48 henry Note Added: 0009668
2018-05-29 12:02 shildenbrand Note Added: 0009669
2018-05-29 18:19 will Note Added: 0009670
2018-05-30 07:28 shildenbrand File Added: compile_PVReader
2018-05-30 07:28 shildenbrand Note Added: 0009671
2018-05-30 07:33 shildenbrand Note Added: 0009672
2018-06-14 16:23 will Note Added: 0009766
2018-06-15 06:44 shildenbrand Note Added: 0009771
2018-06-15 10:19 will Note Added: 0009775
2018-06-19 11:52 MattijsJ Note Added: 0009795
2018-06-19 12:15 shildenbrand Note Added: 0009796
2018-06-20 12:46 will Note Added: 0009805
2018-06-22 10:43 MattijsJ Note Added: 0009810
2018-06-26 20:49 MattijsJ File Added: flush.patch
2018-06-26 20:49 MattijsJ Note Added: 0009832
2018-06-28 09:06 will Assigned To => will
2018-06-28 09:06 will Status new => resolved
2018-06-28 09:06 will Resolution open => fixed
2018-06-28 09:06 will Fixed in Version => dev
2018-06-28 09:06 will Note Added: 0009840
+Issue History