View Issue Details

IDProjectCategoryView StatusLast Update
0002324OpenFOAMBugpublic2016-11-25 14:43
Reporterjoegi Assigned Tohenry  
PrioritynormalSeverityminorReproducibilityrandom
Status resolvedResolutionfixed 
PlatformcentosOScentosOS Versioncentos
Fixed in Versiondev 
Summary0002324: writeDictionary + timeActivatedFileUpdate
DescriptionUse of both function objects together will stalled the simulation, sometimes after the pc is in iddle for a long time the simulation will restart. Have seen this behavior specially when running with many cores.
Steps To ReproduceTry to use a combination of these function objects in parallel.
TagsNo tags attached.

Activities

wyldckat

2016-11-08 23:01

updater   ~0007127

@joegi: Can you please provide more specific details?
There are probably few dozens of possible combinations, given the number of possible dictionary files that can be written and replaced... maybe even if no dictionary file is given?


> will stalled the simulation, sometimes after the pc is in iddle for a long time the simulation will restart

This could potentially be linked to the use of NFS or another filesystem sharing mechanism. Did you make any changes to the settings in "etc/controlDict", specifically within the block "OptimisationSwitches"?

joegi

2016-11-08 23:27

viewer   ~0007129

I see this problem in distributed and shared memory computers.

With timeActivatedFileUpdate I am changing fvSchemes and fvSolution. A wild guess is that each function object is trying to access the same dictionary at the same time.

This issue can be sensitive when running in cluster as you are burning time while the simulation is stalled. In any case, the combination of writeDictionary + timeActivatedFileUpdate is something that not everybody will use.

wyldckat

2016-11-09 11:13

updater   ~0007130

I still have to ask you, to be certain of what needs to be tested:

  1. Which dictionaries are selected to be written by "writeDictionary"?

  2. Does it fail in either order, namely "writeDictionary + timeActivatedFileUpdate" and "timeActivatedFileUpdate + writeDictionary" ?

joegi

2016-11-10 08:53

viewer   ~0007131

The order is indifferent.

The dictionaries are fvSolution, fvSchemes, controlDict, thermophysicalPropertires, turbulenceProperties

MattijsJ

2016-11-17 13:36

reporter   ~0007224

If you use e.g. timeStampMaster then it will trigger a re-read if the new modification time is bigger than the old one + fileModificationSkew (in etc/controlDict). If your filing system is slower than that the reading might overlap with the writing.

This is a generic problem but I would imagine that this might happen more frequently with the timeActivatedFileUpdate functionObject since it is executed at the start of the time loop as if the file-modification checking.

You could try increasing the fileModificationSkew.

MattijsJ

2016-11-22 20:23

reporter   ~0007290

An additional problem is that the copy of the file can be quite slow. Attached a drop-in replacement for src/functionObjects/utilities/timeActivatedFileUpdate/timeActivatedFileUpdate.C that uses a move instead so much much less likely to see your problem.
timeActivatedFileUpdate.C (3,916 bytes)   
/*---------------------------------------------------------------------------*\
  =========                 |
  \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
   \\    /   O peration     |
    \\  /    A nd           | Copyright (C) 2011-2016 OpenFOAM Foundation
     \\/     M anipulation  |
-------------------------------------------------------------------------------
License
    This file is part of OpenFOAM.

    OpenFOAM is free software: you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
    ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
    FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
    for more details.

    You should have received a copy of the GNU General Public License
    along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.

\*---------------------------------------------------------------------------*/

#include "timeActivatedFileUpdate.H"
#include "Time.H"
#include "polyMesh.H"
#include "addToRunTimeSelectionTable.H"

// * * * * * * * * * * * * * * Static Data Members * * * * * * * * * * * * * //

namespace Foam
{
namespace functionObjects
{
    defineTypeNameAndDebug(timeActivatedFileUpdate, 0);

    addToRunTimeSelectionTable
    (
        functionObject,
        timeActivatedFileUpdate,
        dictionary
    );
}
}


// * * * * * * * * * * * * * Private Member Functions  * * * * * * * * * * * //

void Foam::functionObjects::timeActivatedFileUpdate::updateFile()
{
    label i = lastIndex_;
    while
    (
        i < timeVsFile_.size()-1
     && timeVsFile_[i+1].first() < time_.value()
    )
    {
        i++;
    }

    if (i > lastIndex_)
    {
        Info<< nl << type() << ": copying file" << nl << timeVsFile_[i].second()
            << nl << "to:" << nl << fileToUpdate_ << nl << endl;

        fileName destFile(fileToUpdate_ + Foam::name(pid()));
        cp(timeVsFile_[i].second(), destFile);
        mv(destFile, fileToUpdate_);
        lastIndex_ = i;
    }
}


// * * * * * * * * * * * * * * * * Constructors  * * * * * * * * * * * * * * //

Foam::functionObjects::timeActivatedFileUpdate::timeActivatedFileUpdate
(
    const word& name,
    const Time& runTime,
    const dictionary& dict
)
:
    functionObject(name),
    time_(runTime),
    fileToUpdate_(dict.lookup("fileToUpdate")),
    timeVsFile_(),
    lastIndex_(-1)
{
    read(dict);
}


// * * * * * * * * * * * * * * * * Destructor  * * * * * * * * * * * * * * * //

Foam::functionObjects::timeActivatedFileUpdate::~timeActivatedFileUpdate()
{}


// * * * * * * * * * * * * * * * Member Functions  * * * * * * * * * * * * * //

bool Foam::functionObjects::timeActivatedFileUpdate::read
(
    const dictionary& dict
)
{
    dict.lookup("fileToUpdate") >> fileToUpdate_;
    dict.lookup("timeVsFile") >> timeVsFile_;

    lastIndex_ = -1;
    fileToUpdate_.expand();

    Info<< type() << ": time vs file list:" << nl;
    forAll(timeVsFile_, i)
    {
        timeVsFile_[i].second() = timeVsFile_[i].second().expand();
        if (!isFile(timeVsFile_[i].second()))
        {
            FatalErrorInFunction
                << "File: " << timeVsFile_[i].second() << " not found"
                << nl << exit(FatalError);
        }

        Info<< "    " << timeVsFile_[i].first() << tab
            << timeVsFile_[i].second() << endl;
    }
    Info<< endl;

    updateFile();

    return true;
}


bool Foam::functionObjects::timeActivatedFileUpdate::execute()
{
    updateFile();

    return true;
}


bool Foam::functionObjects::timeActivatedFileUpdate::write()
{
    return true;
}


// ************************************************************************* //
timeActivatedFileUpdate.C (3,916 bytes)   

henry

2016-11-25 14:43

manager   ~0007328

Resolved in OpenFOAM-dev by commit b2d5dca4882e7e460146c092f32c384615ba55b8

Issue History

Date Modified Username Field Change
2016-11-08 22:38 joegi New Issue
2016-11-08 23:01 wyldckat Note Added: 0007127
2016-11-08 23:27 joegi Note Added: 0007129
2016-11-09 11:13 wyldckat Note Added: 0007130
2016-11-10 08:53 joegi Note Added: 0007131
2016-11-17 13:36 MattijsJ Note Added: 0007224
2016-11-22 20:23 MattijsJ File Added: timeActivatedFileUpdate.C
2016-11-22 20:23 MattijsJ Note Added: 0007290
2016-11-25 14:43 henry Assigned To => henry
2016-11-25 14:43 henry Status new => resolved
2016-11-25 14:43 henry Resolution open => fixed
2016-11-25 14:43 henry Fixed in Version => dev
2016-11-25 14:43 henry Note Added: 0007328