View Issue Details

IDProjectCategoryView StatusLast Update
0000236OpenFOAMBugpublic2014-07-07 10:49
Reporterwyldckat Assigned Tohenry  
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionfixed 
Summary0000236: Using functionObject systemCall has become a serious pain...
DescriptionGreetings!

I was analysing how to use the new abort function object and I stumbled on this issue, while I was trying to use systemCall to do a forced sleep call on each write, so I could touch the watched file in the mean time.

But like the summary says "using functionObject systemCall has become a serious pain". This is because it requires and I quote:
«
--> FOAM FATAL ERROR:
Executing user-supplied system calls is not enabled by default
because of security issues. If you trust the case you can enable this
facility be adding to the InfoSwitches setting in the system controlDict:

    allowSystemOperations 1

The system controlDict is either

    ~/.OpenFOAM/$WM_PROJECT_VERSION/controlDict

or

    $WM_PROJECT_DIR/etc/controlDict
»

And the pain? Well, although the message clearly states "If you trust the case", it then tells you to go edit the _global_ system controlDict!! Problem is local, so go global... so... this is just postponing a security flaw... although it's a somewhat understandable way to fix this problem, but still...

Now, here comes the crash information: I then tried filling a simple local-global controlDict, expecting that it would load my local-global controlDict after the global-global, but instead it crashed! See the "Steps to Reproduce" for more.
Steps To ReproduceCreate "~/.OpenFOAM/2.0.x/controlDict" with these contents:
«
InfoSwitches
{
  allowSystemOperations 1;
}

DebugSwitches
{

}

OptimisationSwitches
{

}
»

Run any OF binary on a valid case and watch it give you a Segmentation fault... funny ain't it? ;)

And filling at least one valid item of each empty list won't work either :(
Additional InformationSolution?! Simply copy the global-global controlDict to the local-global place, then _hack_ on the copy...

This is if we don't want to not have to "merge around" when we do the next git pull :(
TagsNo tags attached.

Activities

user19

2011-07-04 05:51

  ~0000506

Well, it is a good thing that this is a global setting, because that is under the user's control. If it was in system/controlDict, that would be entirely useless.

The global controlDict in the user-directory always had to be a complete controlDict. Also, if I remember correctly, you can't use the #include and #inputMode directives in the global controlDict. I agree that this is a pain and not very useful. It would be much better if first the global $WM_PROJECT_DIR/etc/controlDict was read, then merged with the one in ~/.OpenFOAM/controlDict, and finally merged with the one in ~/.OpenFOAM/$WM_PROJECT_VERSION/controlDict. This would allow for a default controlDict in the OpenFOAM sources with two possible levels of customisations.

user4

2011-07-04 10:07

  ~0000509

The idea behind all this is if you receive a case from an untrusted source it might have a system("rm -rf ."). Same for user-supplied c++ code (codeStream/codedFixedValue/codedFunctionObject). So the setting cannot be local to the case.

The etc/controlDict controls e.g. the operation of dictionary reading so it does not allow any fancy dictionary preprocessing - these use settings that are only set when reading the etc/controlDict itself.

user19

2011-07-04 10:11

  ~0000510

Last edited: 2011-07-04 10:11

IMHO it would be nice if OpenFOAM read all three global controlDicts and merged them. That would allow one to just override specific settings, without having to copy the whole thing.

wyldckat

2011-07-04 10:23

updater   ~0000511

But as soon as we activate it for our own usage and only after that do we then receive the "evil" case, we're done for just the same :(

Now, if there was some system like what SSH does when recognizing a new machine, asking if we want to connect to it or not since it's new or it has changed... now there's a security feature ;) Also annoying, but nonetheless it would be _paranoid_ just enough :D
The sad part is that this is a pain to implement :(

Either way, "Segmentation fault" just because of an incomplete global controlDict... it should check for sanity first and then load it if possible; if not, complain to the user about the bad file or even skip to the global-global if possible. Otherwise a newbie might just think he/she broke their OF installation... and reinstalling it won't fix it either...

user19

2011-07-04 14:34

 

0001-ENH-Merge-all-global-controlDict-s-when-reading.patch (10,381 bytes)   
From 438a83a5d191c8199417bc03753c4e549740addd Mon Sep 17 00:00:00 2001
From: Michael Wild <themiwi@users.sourceforge.net>
Date: Mon, 4 Jul 2011 15:30:46 +0200
Subject: [PATCH] ENH: Merge all global controlDict's when reading

Signed-off-by: Michael Wild <themiwi@users.sourceforge.net>
---
 .../foamDebugSwitches/foamDebugSwitches.C          |    8 ++++-
 .../adiabaticFlameT/adiabaticFlameT.C              |    2 +-
 .../equilibriumFlameT/equilibriumFlameT.C          |    2 +-
 .../mixtureAdiabaticFlameT.C                       |    2 +-
 src/OSspecific/POSIX/POSIX.C                       |   39 +++++++++++---------
 .../db/dynamicLibrary/dynamicCode/dynamicCode.C    |    2 +-
 src/OpenFOAM/global/debug/debug.C                  |   11 ++++--
 src/OpenFOAM/include/OSspecific.H                  |    2 +-
 .../meshShapes/cellModeller/globalCellModeller.C   |    2 +-
 .../primitives/strings/stringOps/stringOps.C       |    2 +-
 10 files changed, 43 insertions(+), 29 deletions(-)

diff --git a/applications/utilities/miscellaneous/foamDebugSwitches/foamDebugSwitches.C b/applications/utilities/miscellaneous/foamDebugSwitches/foamDebugSwitches.C
index 8f0e53f..69045a4 100644
--- a/applications/utilities/miscellaneous/foamDebugSwitches/foamDebugSwitches.C
+++ b/applications/utilities/miscellaneous/foamDebugSwitches/foamDebugSwitches.C
@@ -61,7 +61,13 @@ int main(int argc, char *argv[])
 
     if (args.optionFound("old") || args.optionFound("new"))
     {
-        dictionary controlDict(IFstream(findEtcFile("controlDict", true))());
+        fileNameList controlDictFiles = findEtcFile("controlDict", true);
+        dictionary controlDict;
+        forAllReverse(controlDictFiles, cI)
+        {
+            IFstream fs(controlDictFiles[cI]);
+            controlDict.merge(dictionary(fs));
+        }
 
         wordHashSet oldDebug
         (
diff --git a/applications/utilities/thermophysical/adiabaticFlameT/adiabaticFlameT.C b/applications/utilities/thermophysical/adiabaticFlameT/adiabaticFlameT.C
index ccf8e55..b23ef7a 100644
--- a/applications/utilities/thermophysical/adiabaticFlameT/adiabaticFlameT.C
+++ b/applications/utilities/thermophysical/adiabaticFlameT/adiabaticFlameT.C
@@ -76,7 +76,7 @@ int main(int argc, char *argv[])
 
     Info<< nl << "Reading Burcat data dictionary" << endl;
 
-    fileName BurcatCpDataFileName(findEtcFile("thermoData/BurcatCpData"));
+    fileName BurcatCpDataFileName(findEtcFile("thermoData/BurcatCpData")[0]);
 
     // Construct control dictionary
     IFstream BurcatCpDataFile(BurcatCpDataFileName);
diff --git a/applications/utilities/thermophysical/equilibriumFlameT/equilibriumFlameT.C b/applications/utilities/thermophysical/equilibriumFlameT/equilibriumFlameT.C
index e46c644..0813d3a 100644
--- a/applications/utilities/thermophysical/equilibriumFlameT/equilibriumFlameT.C
+++ b/applications/utilities/thermophysical/equilibriumFlameT/equilibriumFlameT.C
@@ -77,7 +77,7 @@ int main(int argc, char *argv[])
 
     Info<< nl << "Reading Burcat data dictionary" << endl;
 
-    fileName BurcatCpDataFileName(findEtcFile("thermoData/BurcatCpData"));
+    fileName BurcatCpDataFileName(findEtcFile("thermoData/BurcatCpData")[0]);
 
     // Construct control dictionary
     IFstream BurcatCpDataFile(BurcatCpDataFileName);
diff --git a/applications/utilities/thermophysical/mixtureAdiabaticFlameT/mixtureAdiabaticFlameT.C b/applications/utilities/thermophysical/mixtureAdiabaticFlameT/mixtureAdiabaticFlameT.C
index 2c8b783..330edb3 100644
--- a/applications/utilities/thermophysical/mixtureAdiabaticFlameT/mixtureAdiabaticFlameT.C
+++ b/applications/utilities/thermophysical/mixtureAdiabaticFlameT/mixtureAdiabaticFlameT.C
@@ -75,7 +75,7 @@ int main(int argc, char *argv[])
 
     Info<< nl << "Reading Burcat data dictionary" << endl;
 
-    fileName BurcatCpDataFileName(findEtcFile("thermoData/BurcatCpData"));
+    fileName BurcatCpDataFileName(findEtcFile("thermoData/BurcatCpData")[0]);
 
     // Construct control dictionary
     IFstream BurcatCpDataFile(BurcatCpDataFileName);
diff --git a/src/OSspecific/POSIX/POSIX.C b/src/OSspecific/POSIX/POSIX.C
index 48e4b3a..60a2684 100644
--- a/src/OSspecific/POSIX/POSIX.C
+++ b/src/OSspecific/POSIX/POSIX.C
@@ -253,8 +253,9 @@ bool Foam::chDir(const fileName& dir)
 }
 
 
-Foam::fileName Foam::findEtcFile(const fileName& name, bool mandatory)
+Foam::fileNameList Foam::findEtcFile(const fileName& name, bool mandatory)
 {
+    fileNameList results;
     //
     // search for user files in
     // * ~/.OpenFOAM/VERSION
@@ -266,13 +267,13 @@ Foam::fileName Foam::findEtcFile(const fileName& name, bool mandatory)
         fileName fullName = searchDir/FOAMversion/name;
         if (isFile(fullName))
         {
-            return fullName;
+            results.append(fullName);
         }
 
         fullName = searchDir/name;
         if (isFile(fullName))
         {
-            return fullName;
+            results.append(fullName);
         }
     }
 
@@ -290,13 +291,13 @@ Foam::fileName Foam::findEtcFile(const fileName& name, bool mandatory)
             fileName fullName = searchDir/FOAMversion/name;
             if (isFile(fullName))
             {
-                return fullName;
+                results.append(fullName);
             }
 
             fullName = searchDir/name;
             if (isFile(fullName))
             {
-                return fullName;
+                results.append(fullName);
             }
         }
     }
@@ -313,13 +314,13 @@ Foam::fileName Foam::findEtcFile(const fileName& name, bool mandatory)
             fileName fullName = searchDir/"site"/FOAMversion/name;
             if (isFile(fullName))
             {
-                return fullName;
+                results.append(fullName);
             }
 
             fullName = searchDir/"site"/name;
             if (isFile(fullName))
             {
-                return fullName;
+                results.append(fullName);
             }
         }
     }
@@ -335,24 +336,28 @@ Foam::fileName Foam::findEtcFile(const fileName& name, bool mandatory)
         fileName fullName = searchDir/"etc"/name;
         if (isFile(fullName))
         {
-            return fullName;
+            results.append(fullName);
         }
     }
 
     // Not found
     // abort if the file is mandatory, otherwise return null
-    if (mandatory)
+    if (results.empty())
     {
-        std::cerr
-            << "--> FOAM FATAL ERROR in Foam::findEtcFile() :"
-               " could not find mandatory file\n    '"
-            << name.c_str() << "'\n\n" << std::endl;
-        ::exit(1);
+        if (mandatory)
+        {
+            std::cerr
+                << "--> FOAM FATAL ERROR in Foam::findEtcFile() :"
+                   " could not find mandatory file\n    '"
+                << name.c_str() << "'\n\n" << std::endl;
+            ::exit(1);
+        }
+        // Return null-constructed fileName rather than fileName::null
+        // to avoid cyclic dependencies in the construction of globals
+        results.append(fileName());
     }
 
-    // Return null-constructed fileName rather than fileName::null
-    // to avoid cyclic dependencies in the construction of globals
-    return fileName();
+    return results;
 }
 
 
diff --git a/src/OpenFOAM/db/dynamicLibrary/dynamicCode/dynamicCode.C b/src/OpenFOAM/db/dynamicLibrary/dynamicCode/dynamicCode.C
index 327fd56..010c20f 100644
--- a/src/OpenFOAM/db/dynamicLibrary/dynamicCode/dynamicCode.C
+++ b/src/OpenFOAM/db/dynamicLibrary/dynamicCode/dynamicCode.C
@@ -177,7 +177,7 @@ bool Foam::dynamicCode::resolveTemplates
         // not found - fallback to ~OpenFOAM expansion
         if (file.empty())
         {
-            file = findEtcFile(codeTemplateDirName/templateName);
+            file = findEtcFile(codeTemplateDirName/templateName)[0];
         }
 
         if (file.empty())
diff --git a/src/OpenFOAM/global/debug/debug.C b/src/OpenFOAM/global/debug/debug.C
index b0c6969..f93bcc0 100644
--- a/src/OpenFOAM/global/debug/debug.C
+++ b/src/OpenFOAM/global/debug/debug.C
@@ -75,10 +75,13 @@ Foam::dictionary& Foam::debug::controlDict()
 {
     if (!controlDictPtr_)
     {
-        controlDictPtr_ = new dictionary
-        (
-            IFstream(findEtcFile("controlDict", true))()
-        );
+        fileNameList controlDictFiles = findEtcFile("controlDict", true);
+        controlDictPtr_ = new dictionary();
+        forAllReverse(controlDictFiles, cI)
+        {
+            IFstream fs(controlDictFiles[cI]);
+            controlDictPtr_->merge(dictionary(fs));
+        }
     }
 
     return *controlDictPtr_;
diff --git a/src/OpenFOAM/include/OSspecific.H b/src/OpenFOAM/include/OSspecific.H
index f01ad29..d9da89a 100644
--- a/src/OpenFOAM/include/OSspecific.H
+++ b/src/OpenFOAM/include/OSspecific.H
@@ -110,7 +110,7 @@ bool chDir(const fileName& dir);
 //
 //  \return The full path name or fileName() if the name cannot be found
 //  Optionally abort if the file cannot be found
-fileName findEtcFile(const fileName&, bool mandatory=false);
+fileNameList findEtcFile(const fileName&, bool mandatory=false);
 
 //- Make a directory and return an error if it could not be created
 //  and does not already exist
diff --git a/src/OpenFOAM/meshes/meshShapes/cellModeller/globalCellModeller.C b/src/OpenFOAM/meshes/meshShapes/cellModeller/globalCellModeller.C
index c545e73..10b4ae2 100644
--- a/src/OpenFOAM/meshes/meshShapes/cellModeller/globalCellModeller.C
+++ b/src/OpenFOAM/meshes/meshShapes/cellModeller/globalCellModeller.C
@@ -36,7 +36,7 @@ Description
 // PtrList of models
 Foam::PtrList<Foam::cellModel> Foam::cellModeller::models_
 (
-    IFstream(findEtcFile("cellModels", true))()
+    IFstream(findEtcFile("cellModels", true)[0])()
 );
 
 // List of model pointers
diff --git a/src/OpenFOAM/primitives/strings/stringOps/stringOps.C b/src/OpenFOAM/primitives/strings/stringOps/stringOps.C
index ebbc47c..6b43895 100644
--- a/src/OpenFOAM/primitives/strings/stringOps/stringOps.C
+++ b/src/OpenFOAM/primitives/strings/stringOps/stringOps.C
@@ -562,7 +562,7 @@ Foam::string& Foam::stringOps::inplaceExpand
             // otherwise add extra test
             if (user == "OpenFOAM")
             {
-                s = findEtcFile(file);
+                s = findEtcFile(file)[0];
             }
             else
             {
-- 
1.7.4.1

user19

2011-07-04 14:38

  ~0000514

Something like the attached patch 0001-ENH-Merge-all-global-controlDict-s-when-reading.patch works for me. Instead of findEtcFile() just reporting the first file found, it returns a list of all files that have been found (or if not, either errors out or returns a list with a single, null-constructed fileName). In debug::controlDict() this list is then used to merge the various controlDict's that have been found. The same happens in foamDebugSwitches.

This way it is possible to only override specific settings in the global controlDict file, allowing administrators to set the defaults, while the users can selectively override them.

henry

2011-07-05 10:44

manager   ~0000516

I have fixed the problem pertaining to the initial bug-report here which relates to global construction order of classes which read etc/controlDict:

commit 6a6e508a359c674f2d93b5daa23ae7e92ce83ce

We are also considering the proposal to support merging of the global controlDict
and investigating options for this.

user19

2011-07-05 15:12

  ~0000517

Great! Sorry for hijacking this report, I realised too late that my proposal probably was too OT and would have been better served by its own report.

henry

2011-07-06 12:25

manager   ~0000519

Thanks for the patch on which I have based the
commit 49af5b15c20054543a3d674ba949bedef5ab73ec

Note that these corrections and developments do not entirely solve the problem of possible seg-faults if there are errors in the controlDict. However the problem is now limited to parse errors because the error messaging system is not fully operational before the debug switches are read and set.

wyldckat

2011-08-20 21:55

updater   ~0000619

Greetings!

Attached is the file "patch_preemptive_error_message", which is a patch proposition for (almost) completely closing this problem of seg-faulting for good.
It doesn't feel right to simply do a segmentation fault with no error message... this could lead new/stressed users to go "bananas" over a message à lá Windows 95 :(

The attached proposition is also available here: https://github.com/wyldckat/OpenFOAM-2.0.x/commits/issue236 - I can make pull request from Github if you want, since this might speed things up...

Best regards,
Bruno

wyldckat

2011-08-20 21:55

updater  

patch_preemptive_error_message (5,029 bytes)   
From: https://github.com/wyldckat/OpenFOAM-2.0.x/commits/issue236

diff --git a/src/OpenFOAM/db/error/IOerror.C b/src/OpenFOAM/db/error/IOerror.C
index 150502a..5d7f9ac 100644
--- a/src/OpenFOAM/db/error/IOerror.C
+++ b/src/OpenFOAM/db/error/IOerror.C
@@ -31,6 +31,7 @@ License
 #include "Pstream.H"

 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
+bool Foam::FatalIOErrorIsInstantiated=false;

 Foam::IOerror::IOerror(const string& title)
 :
@@ -38,7 +39,9 @@ Foam::IOerror::IOerror(const string& title)
     ioFileName_("unknown"),
     ioStartLineNumber_(-1),
     ioEndLineNumber_(-1)
-{}
+{
+    FatalIOErrorIsInstantiated=true;
+}


 Foam::IOerror::IOerror(const dictionary& errDict)
@@ -47,11 +50,15 @@ Foam::IOerror::IOerror(const dictionary& errDict)
     ioFileName_(errDict.lookup("ioFileName")),
     ioStartLineNumber_(readLabel(errDict.lookup("ioStartLineNumber"))),
     ioEndLineNumber_(readLabel(errDict.lookup("ioEndLineNumber")))
-{}
+{
+    FatalIOErrorIsInstantiated=true;
+}


 Foam::IOerror::~IOerror() throw()
-{}
+{
+  FatalIOErrorIsInstantiated=false;
+}


 Foam::OSstream& Foam::IOerror::operator()
@@ -64,11 +71,28 @@ Foam::OSstream& Foam::IOerror::operator()
     const label ioEndLineNumber
 )
 {
-    error::operator()(functionName, sourceFileName, sourceFileLineNumber);
-    ioFileName_ = ioFileName;
-    ioStartLineNumber_ = ioStartLineNumber;
-    ioEndLineNumber_ = ioEndLineNumber;
-
+    if(FatalIOErrorIsInstantiated)
+    {
+        error::operator()(functionName, sourceFileName, sourceFileLineNumber);
+        ioFileName_ = ioFileName;
+        ioStartLineNumber_ = ioStartLineNumber;
+        ioEndLineNumber_ = ioEndLineNumber;
+    }
+    else
+    {
+        cerr<< "Foam::IOerror::operator():\n"
+            << "--- Emergency exit deployed ---\n"
+            << "Messages to be delivered:\n"
+            << "\t Function Name of the reporter: " << functionName << std::endl
+            << "\t Source Filename of the function: " << sourceFileName << std::endl
+            << "\t Source line number: " << sourceFileLineNumber << std::endl
+            << "\t Filename that triggered this error: " << ioFileName << std::endl
+            << "\t Start line number of the offense: " << ioStartLineNumber << std::endl
+            << "\t ending at: " << ioEndLineNumber << std::endl << std::endl
+            << "--- Next step will be a segfault. ---"
+            << std::endl << std::endl;
+        exit(1);
+    }
     return operator OSstream&();
 }

diff --git a/src/OpenFOAM/db/error/error.C b/src/OpenFOAM/db/error/error.C
index 56a617c..fa91761 100644
--- a/src/OpenFOAM/db/error/error.C
+++ b/src/OpenFOAM/db/error/error.C
@@ -33,6 +33,8 @@ License

 // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

+bool Foam::FatalErrorIsInstantiated=false;
+
 Foam::error::error(const string& title)
 :
     std::exception(),
@@ -51,6 +53,7 @@ Foam::error::error(const string& title)
             << endl;
         exit(1);
     }
+    FatalErrorIsInstantiated=true;
 }


@@ -73,6 +76,7 @@ Foam::error::error(const dictionary& errDict)
             << endl;
         exit(1);
     }
+    FatalErrorIsInstantiated=true;
 }


@@ -88,12 +92,14 @@ Foam::error::error(const error& err)
     messageStreamPtr_(new OStringStream(*err.messageStreamPtr_))
 {
     //*messageStreamPtr_ << err.message();
+    FatalErrorIsInstantiated=true;
 }


 Foam::error::~error() throw()
 {
     delete messageStreamPtr_;
+    FatalErrorIsInstantiated=false;
 }


@@ -104,9 +110,24 @@ Foam::OSstream& Foam::error::operator()
     const int sourceFileLineNumber
 )
 {
-    functionName_ = functionName;
-    sourceFileName_ = sourceFileName;
-    sourceFileLineNumber_ = sourceFileLineNumber;
+    if(FatalErrorIsInstantiated)
+    {
+      functionName_ = functionName;
+      sourceFileName_ = sourceFileName;
+      sourceFileLineNumber_ = sourceFileLineNumber;
+    }
+    else
+    {
+        cerr<< "Foam::error::operator():\n"
+            << "--- Emergency exit deployed ---\n"
+            << "Messages to be delivered:\n"
+            << "\t Function Name of the reporter: " << functionName << std::endl
+            << "\t Source Filename of the function: " << sourceFileName << std::endl
+            << "\t Source line number: " << sourceFileLineNumber << std::endl
+            << "--- Next step will be a segfault. ---"
+            << std::endl << std::endl;
+        exit(1);
+    }

     return operator OSstream&();
 }
diff --git a/src/OpenFOAM/db/error/error.H b/src/OpenFOAM/db/error/error.H
index 2105930..a582ebb 100644
--- a/src/OpenFOAM/db/error/error.H
+++ b/src/OpenFOAM/db/error/error.H
@@ -55,6 +55,10 @@ SourceFiles
 namespace Foam
 {

+// These two must be set when the respective classes have been instantiated
+extern bool FatalErrorIsInstantiated;
+extern bool FatalIOErrorIsInstantiated;
+
 // Forward declaration of friend functions and operators
 class error;
 Ostream& operator<<(Ostream&, const error&);
patch_preemptive_error_message (5,029 bytes)   

user4

2011-08-22 15:37

  ~0000620

Hi Bruno,

your patch fixes using member data of a non-existing object (FatalError not yet constructed) so you don't see a problem but the object is still non-existent.

Do you know why the object has not been constructed yet? The order in global.Cver is -construct error streams first, -load controlDict (debug.C) second.

wyldckat

2011-08-22 15:58

updater   ~0000621

Last edited: 2011-08-22 16:00

Hi Mattijs,

Mmm... I should have mentioned that I get a segfault whenever I have a bad "~/.OpenFOAM/2.0.x/controlDict", which is related to the last note from Henry. For example, creating a file with these contents:

  InfoSwitches
  {
    allowSystemOperations 1
  }

This will simply lead to a segfault, with no additional error message. Simply because no ";" was found in the file. If on the other hand, I have something like this:
  InfoSwitches
  {
    allowSystemOperations 1
  }

  DebugSwitches
  {
    empty 0;
  }

This loads well and does not segfault because the last ";" wraps things up for the token parser (if I'm not mistaken).

As far as I can understand, the order of instantiations/constructs are being done this way due to some flags that are needed for the FatalError related objects... otherwise I would have suggested changing the code order, so this wouldn't occur.

I'm defending the need for some error message simply because this is one of those things that can easily be mistaken for something worse. If it simply segfaults without anymore information, all tutorials and cases stop working... and the person doesn't know/remember the changes made to these system wide "controlDict" files, can lead to at least trying to reinstall OpenFOAM or even reinstall the whole system... which might not even fix the problem, if the /home partition isn't formatted. (edit: in some cases, this can be a whole work day shot out the window!)
Last but not least... this is one of the few times I found OpenFOAM crashing without any other message indicating what went' wrong... and on the ones that it didn't give any messages, it was simply case bound, not system wide :(

user4

2011-08-22 17:43

  ~0000622

I agree with the need for an error message but the solution isn't good. It might run through with the current compiler set / optimisation level but you are still using a member function of an object which has not been constructed yet. We need to find out why the object has not been constructed and if not how we can force it to be constructed.

Mattijs

user19

2011-08-22 19:41

  ~0000623

You can't, at least not the way things are currently implemented. The global stream objects depend on the global controlDict file. I can't remember how the dependency chain looked exactly, but that's what I found out some months ago.

wyldckat

2011-08-22 23:09

updater   ~0000624

_Arrrrggg_..... and I quote:
«
//- Define the debug information, lookup as \a Name
#define defineDebugSwitchWithName(Type, Name, DebugSwitch) \
    int Type::debug(::Foam::debug::debugSwitch(Name, DebugSwitch))
»

This means that every single library dynamically loaded will automatically want to update their own static information about the respective debug option in "controlDict". With me running "blockMesh", it started at "surfacePatchIOList" in "libtriSurface.so". Now I understand what mwild meant!


OK, two scenarios pop-up in my head...

SCENARIO 1:
Taking into account the given restrictions... and this practical example at global.Cver:

  #include "JobInfo.H"
  bool Foam::JobInfo::constructed(false);


Then would this be suitable?
* Use a dedicated static method to act as a wrapper for "::Foam::FatalError" inside "Foam::error";
* Likewise for "::Foam::FatalIOError" in "Foam::IOerror";
* These two wrappers would then work along with the "static bool constructed;" for each respective class, similarly to JobInfo's implementation.

I suppose there would be as many overloaded static wrappers as the respective operators they would wrap... but at least this way, any future leaks would be contained this way!? At least for any calls to "FatalIOErrorIn" and "FatalErrorIn"?



SCENARIO 2:
Force construction for "FatalIOError" and "FatalError" directly in "FatalIOErrorIn" and company ... in a nearly identical fashion to "Foam::debug::switchSet"-->"controlDictPtr_"/"Foam::debug::controlDict()".

This scenario makes more sense, since the "*Ptr_" pointers were designed for the very same reason: they wouldn't exist otherwise, were it not for pointer based construction on-the-fly!

user19

2011-08-23 10:23

  ~0000625

Thing is, you have this problem *everywhere*! All the global objects that get instantiated in global.Cver are subject to this issue. I once proposed solution 2 (using functions that instantiate the objects on first call) but Henry opposed it due to performance concerns (see http://is.gd/1YPWLG).

wyldckat

2011-08-23 22:26

updater   ~0000626

Mmm... So, to sum up:
* Re-engineering the current initialization system is out of the question... at least in the near future.
* Without re-engineering the system, the stuff at "global.Cver" will never be jump-started before everyone else... unless of course there is a way to hack the dynamic library loading mechanism to force a priority loading of the objects at "global.Cver" ("LD_PRELOAD" wouldn't work either, since "global.Cver" is deeply embed into "libOpenFOAM.so").


Therefore, the only solution left is a zero level error system!? By what I've seen in the _nearby_ code, "cerr" is often used as such. From my point of view, the only things left to figure out, are:
* Will "bool Foam::FatalIOError::constructed(false);" always be instantiated at compile time, no matter with which compiler settings? If so, then this can act as a _pre-emptive state flag_!
* The macro "FatalIOErrorIn" can do something like this:
    ZeroLevelError(::Foam::FatalIOError::constructed, ...); FatalIOError(...)
  without loss of cohesion, since it's the last one that matters. "void ZeroLevelError" can automatically return if the first argument is true. It can even be a class of static methods for this kind of handling.

If the previous two points can be implemented without loss of code efficiency/cohesion (I don't know enough yet to figure it out on my own :( ), then it's a good way to do this, at least for now...


The other way to go would be to have an independent binary application to always act as a pre-parser when _core_ dictionary files are changed, so that it checks them before calling the real binary!? This would actually work well with mwild's proposition for containing the OpenFOAM shell environment...

user4

2011-08-24 14:48

  ~0000628

- added a print stmt to debug::debugSwitch. Order of symbols seems to be:
surfacePatchIOList
surfacePatch
..
writeJobInfo

- this writeJobInfo is the one from global.Cver and this is the one we want first.

tried:
- adding -lOpenFOAM as first argument on all link lines
- adding to global.Cver compilation:
    -Xlinker -z -Xlinker initfirst
to force it to be loaded first
- adding to OpenFOAM/Make/options
    -Xlinker -z -Xlinker initfirst
None of this seemed to have changed the static initialiser construction - it is still surfacePatchIOList first. Is it actually possible to change the order?

Bruno's 'zero-level error system' seems to work. Hacked a
    if (JobInfo::constructed)
into primitiveEntryIO.C and that seems to do the trick.

user19

2011-08-24 15:41

  ~0000629

Running with "LD_DEBUG:all blockMesh" and grepping for "calling init:" gives me the following (yeah, still 1.7.x, I know...):

     24300: calling init: /lib/x86_64-linux-gnu/libpthread.so.0
     24300: calling init: /lib/x86_64-linux-gnu/libc.so.6
     24300: calling init: /lib/x86_64-linux-gnu/libdl.so.2
     24300: calling init: /lib/x86_64-linux-gnu/libm.so.6
     24300: calling init: /lib/x86_64-linux-gnu/libgcc_s.so.1
     24300: calling init: /usr/lib/x86_64-linux-gnu/libstdc++.so.6
     24300: calling init: /lib/x86_64-linux-gnu/libutil.so.1
     24300: calling init: /usr/lib/libopen-pal.so.0
     24300: calling init: /usr/lib/libopen-rte.so.0
     24300: calling init: /usr/lib/libmpi.so.0
     24300: calling init: /lib/x86_64-linux-gnu/libnsl.so.1
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/openmpi/libmetis-parmetis.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/openmpi/libparmetis.so
     24300: calling init: /usr/lib/libmetis.so.3.1
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libGKlib.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libmetis.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libscotcherrexit.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libscotch.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/openmpi/libparMetisDecomp.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libmetisDecomp.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libscotchDecomp.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libtriSurface.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libdecompositionMethods.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/liblagrangian.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libmeshTools.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libfiniteVolume.so
     24300: calling init: /lib/x86_64-linux-gnu/libz.so.1
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/openmpi/libPstream.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libdynamicMesh.so
     24300: calling init: /home/mwild/Software/OpenFOAM/OpenFOAM-1.7.x/lib/linux64GccDPDebug/libOpenFOAM.so
     24300: calling init: /lib/x86_64-linux-gnu/libnss_compat.so.2
     24300: calling init: /lib/x86_64-linux-gnu/libnss_files.so.2
     24300: calling init: /lib/x86_64-linux-gnu/libnss_nis.so.2

If I read this correctly, it looks like libOpenFOAM.so is initialized *last*. Playing around with LD_PRELOAD didn't help, because it only influenced the *loading* order, but not the *initialization* order.

wyldckat

2011-08-24 20:21

updater   ~0000630

Last edited: 2011-08-24 20:23

By the limited know-how I have on Assembly, this is does look a lot like the standard stack list LIFO - Last In, First Out. This makes sense if the dynamic library loading mechanism uses a stack for keeping track of the loaded libraries with objects to be initialized, then popping the top for the next one on the list.

But still, this looks a lot like it would be a dedicated compiler related hacking, which might work with Gcc, but not Icc. The other possibility are OS kernels working in different LIFO/FIFO points of view when it comes to loading and initializing, therefore limiting what a compiler can or cannot do.

Delayed loading is conceptually possible, but AFAIK, this requires a more advanced linking and dynamic loading mechanisms...

edit: I forgot to mention that "JobInfo::constructed(false)" might be a false positive. From my experience, it's either Linux or Gcc which seems to be prone to leaving the memory zeroed out and ready for consumption for applications.

user4

2011-08-26 16:46

  ~0000631

hi guys,

I've pushed that 'zero-level error system' hack to fc5421fbee44e8b1bb04f38eeaa248fa34be7765. I've only added it to primitiveEntryIO.C but there might be others where it is needed.

It checks JobInfo::constructed and uses cerr (instead of streams).

Please let me know if it needs to be added to other places or if we can control the library initialisation order after all. Saw that xlC (IBM's compiler) has a -qpriority flag to control this.

Mattijs

wyldckat

2011-08-28 13:23

updater   ~0000633

Greetings to all!

@Mattijs: Nice! Very nice indeed! Although hooking the system to JobInfo has me a bit concerned...
About that "false positive" I mentioned at the end of note #630: during my tests, these _normal_ values for "bool", "int" and "double" seemed to be hard-coded at build time, so it should pre-initialized.

Some more feedback:
1. I have yet to find more initialization issues like this one... only time will tell ;)

2. I've been (re)searching how to handle the whole ordeal of initialization order and have been (b)logging about it here: http://www.cfd-online.com/Forums/blogs/wyldckat/930-linking-dynamic-libraries-initializing-variables.html - this way I wouldn't clog this bug issue with a heavy load of links.

Copy-paste of the conclusions so far:
* Hacking the loading/initialization order is probably a must. This means that for each platform+compiler, there will be different solutions, if any.
* Using "namespace std" for the error management code seemed like a good solution, since there seems to be a careful selection on ld/gcc. But embedding it directly in libOpenFOAM doesn't seem to help things.
* What would help the initialization order would be to separate these more basic (debug) functions to a separate and somewhat independent library. This would make it easier to enforce the initialization order, since this library would have a higher probability of being put closer to libc, pthreads and Open-MPI, therefore being initialized on time.

user4

2011-09-01 10:15

  ~0000637

- I think primitives are always pre-initialised. JobInfo::constructed just happens to be called that way since we originally had the construction order problem when writing job files.

- the current solution is ok but a nicer one would be to have a FatalError like macro return a stream, either 'cerr' or the fatalerror stream.

- the only way I could force the initialisation order was to split off the global.Cver into a separate object/library (libglobal.o) and link this to the library which is initialised first (I think libtriSurface.o). In the end I just linked it to all libraries. Very messy.

Issue History

Date Modified Username Field Change
2011-07-03 22:12 wyldckat New Issue
2011-07-04 05:51 user19 Note Added: 0000506
2011-07-04 10:07 user4 Note Added: 0000509
2011-07-04 10:11 user19 Note Added: 0000510
2011-07-04 10:11 user19 Note Edited: 0000510
2011-07-04 10:23 wyldckat Note Added: 0000511
2011-07-04 14:34 user19 File Added: 0001-ENH-Merge-all-global-controlDict-s-when-reading.patch
2011-07-04 14:38 user19 Note Added: 0000514
2011-07-05 10:44 henry Note Added: 0000516
2011-07-05 15:12 user19 Note Added: 0000517
2011-07-06 12:25 henry Note Added: 0000519
2011-07-06 12:25 henry Status new => resolved
2011-07-06 12:25 henry Resolution open => fixed
2011-07-06 12:25 henry Assigned To => henry
2011-08-20 21:55 wyldckat Note Added: 0000619
2011-08-20 21:55 wyldckat Status resolved => feedback
2011-08-20 21:55 wyldckat Resolution fixed => reopened
2011-08-20 21:55 wyldckat File Added: patch_preemptive_error_message
2011-08-22 15:37 user4 Note Added: 0000620
2011-08-22 15:58 wyldckat Note Added: 0000621
2011-08-22 15:58 wyldckat Status feedback => assigned
2011-08-22 16:00 wyldckat Note Edited: 0000621
2011-08-22 17:43 user4 Note Added: 0000622
2011-08-22 19:41 user19 Note Added: 0000623
2011-08-22 23:09 wyldckat Note Added: 0000624
2011-08-23 10:23 user19 Note Added: 0000625
2011-08-23 22:26 wyldckat Note Added: 0000626
2011-08-24 14:48 user4 Note Added: 0000628
2011-08-24 15:41 user19 Note Added: 0000629
2011-08-24 20:21 wyldckat Note Added: 0000630
2011-08-24 20:23 wyldckat Note Edited: 0000630
2011-08-26 16:46 user4 Note Added: 0000631
2011-08-28 13:23 wyldckat Note Added: 0000633
2011-09-01 10:15 user4 Note Added: 0000637
2014-07-07 10:46 henry Status assigned => closed
2014-07-07 10:49 henry Resolution reopened => fixed