View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001021OpenFOAMBugpublic2013-09-19 13:272013-10-04 09:14
Reporteruser528Assigned Touser4 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
PlatformLinuxOSSUSE Linux Enterprise ServerOS Version11.1
Summary0001021: Infinite memory usage on exit(FatalError) when ran in background
DescriptionReporting an error via
FatalErrorIn() << exit(FatalError)
makes program run out of memory if launched in the background (with & sign in bash).
Steps To Reproduce1. Compile the following OpenFOAM program:
// infMemoryTest.C
#include <unistd.h>
#include "fvCFD.H"


// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //


int main(int argc, char *argv[])
{
    ::sleep(2);
    Info << "Test" << endl;
    FatalErrorIn("Test")
        << exit(FatalError);

    return 0;
}


// ************************************************************************* //

2. Prepare a bash script to launch it:
-------------
#!/bin/sh
 infMemoryTest >t &
# infMemoryTest >t 2>&1 &
-------------
3. Launch the script
4. Expected behavior is that program finished normally.
5. Instead, the program does not finish, but consumes more and more memory until it is killed manually
Additional InformationNote that if the error output is redirected to a file (as in the commented out line in shell script), the program finishes normally. The same is if the program is directly launched from command line, not from bash script, or if the program is not launched in the background (& sign removed).

Also the problem does not take place if the exit(FatalError) is removed from the source.

Also note that the following raw-C++ program does not demonstrate such a behavior:
-------------
#include <iostream>
#include <unistd.h>

int main() {
   sleep(2);
   std::cout << "Test" << std::endl;
   std::cerr << "Test-err" << std::endl;
   return 0;
}
-------------

I am not absolutely sure that the problem is in OpenFOAM, but it can always be reproduced here, both in a small example code above and in big my code. Also the fact that standard C++ example does not demonstrate the behavior makes me think that it is more a problem of OpenFOAM.
TagsNo tags attached.

Activities

wyldckat

2013-09-19 13:36

updater   ~0002501

A couple of quick questions that might come in handy for the developers:
 1- What's the shell "/bin/sh" you are using? You can check by running:
      ls -l /bin/sh

 2- What GCC version was used for building your local OpenFOAM installation?

user528

2013-09-19 13:46

  ~0002502

1. It is bash
2. g++ (GCC) 4.7.2

Forgot also to mention that the problem existed when I cloned OpenFOAM repo about a month ago, and exists today after today's fetch from github

user17

2013-09-19 16:53

  ~0002506

I cannot reproduce this. Can you attach gdb to it and see where it is, or reproduce it on another operating system?

user528

2013-09-20 11:57

  ~0002507

Firstly, I found that it happens only with stty tostop option set.

Secondly, I managed to attach to a running process using gdb, interrupting it in a random place while it was accumulating memory, and got the following backtrace:

(gdb) bt
                      
#0 0x00007ffff18f10f6 in memcpy () from /lib64/libc.so.6
#1 0x00007ffff7ee9dc0 in char* std::string::_S_construct<char*>(char*, char*, std::allocator<char> const&, std::forward_iterator_tag) ()
   from /p/dt/sde/tools/x86-64_linux26/boost/1.46.0_64/lib/libboost_filesystem.so.1.46.0
#2 0x00007ffff20f39ca in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<char*>(char*, char*, std::allocator<char> const&) ()
   from /usr/intel/pkgs/gcc/4.7.0/lib64/libstdc++.so.6
#3 0x00007ffff20d02cd in std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const () from /usr/intel/pkgs/gcc/4.7.0/lib64/libstdc++.so.6
#4 0x00007ffff40d2a0f in str (this=<optimized out>) at /nfs/inn/itools/em64t_SLES11/pkgs/gcc/4.7.2/.bin/../lib64/gcc/x86_64-suse-linux/4.7.2/../../../../include/c++/4.7.2/sstream:458
#5 Foam::OStringStream::str (this=<optimized out>) at lnInclude/OStringStream.H:113
#6 0x00007ffff40cbf5d in Foam::error::message (this=this@entry=0x7ffff4740060 <Foam::FatalIOError>) at lnInclude/error.C:164
#7 0x00007ffff40cc0da in Foam::operator<< (os=..., ioErr=...) at lnInclude/IOerror.C:260
#8 0x00007ffff40cfb7f in Foam::IOerror::exit (this=0x7ffff4740060 <Foam::FatalIOError>) at lnInclude/IOerror.C:202
#9 0x00007ffff4159d35 in operator<< <Foam::IOerror, int> (os=..., m=...) at lnInclude/errorManip.H:117
#10 Foam::IOstream::check (this=0x7ffff4740180 <Foam::Perr>, operation=0x7ffff442fbd0 "Ostream& operator<<(Ostream&, const char*)") at db/IOstreams/IOstreams/IOstream.C:100
#11 0x00007ffff40f2b3a in Foam::operator<< (os=..., s=<optimized out>) at primitives/chars/char/charIO.C:65
#12 0x00007ffff40cc0fa in Foam::operator<< (os=..., ioErr=...) at lnInclude/IOerror.C:259
#13 0x00007ffff40cfb7f in Foam::IOerror::exit (this=0x7ffff4740060 <Foam::FatalIOError>) at lnInclude/IOerror.C:202
#14 0x00007ffff4159d35 in operator<< <Foam::IOerror, int> (os=..., m=...) at lnInclude/errorManip.H:117
#15 Foam::IOstream::check (this=0x7ffff4740180 <Foam::Perr>, operation=0x7ffff442fbd0 "Ostream& operator<<(Ostream&, const char*)") at db/IOstreams/IOstreams/IOstream.C:100
#16 0x00007ffff40f2b3a in Foam::operator<< (os=..., s=<optimized out>) at primitives/chars/char/charIO.C:65
#17 0x00007ffff40cc0fa in Foam::operator<< (os=..., ioErr=...) at lnInclude/IOerror.C:259
#18 0x00007ffff40cfb7f in Foam::IOerror::exit (this=0x7ffff4740060 <Foam::FatalIOError>) at lnInclude/IOerror.C:202
#19 0x00007ffff4159d35 in operator<< <Foam::IOerror, int> (os=..., m=...) at lnInclude/errorManip.H:117
#20 Foam::IOstream::check (this=0x7ffff4740180 <Foam::Perr>, operation=0x7ffff442fbd0 "Ostream& operator<<(Ostream&, const char*)") at db/IOstreams/IOstreams/IOstream.C:100
...
the last five lines are repeated much more times.

Obviously for some reason the error output stream becomes bad, and IOstream::check launches FatalErrorIn() << exit(FatalError) recursively once again, and so on.

I've checked with my test pure C++ program:
-- test-prg.cpp ---
#include <iostream>
#include <unistd.h>

int main() {
   sleep(2);
   std::cout << "Test" << std::endl;
   std::cerr << "Test-err" << std::endl;
   std::cout << std::cerr.bad() << std::endl;
   return 0;
}
--------------
with the following launch script:
--------------
#!/bin/sh
./test-prg >log &
--------------
it prints 0 to log file with stty -tostop, and 1 with stty tostop.

=====================

So obviously that's it: if for any reason the error output stream becomes "bad", OpenFOAM goes into infinite recursion trying to report this.

user17

2013-09-20 16:00

  ~0002508

Why is the error stream bad but the standard output stream good? That doesn't make sense to me.

I've added a simple workaround for the memory leak to 2.2.x.

user528

2013-09-20 16:23

  ~0002509

Obviously good are those streams that were redirected to file in this scenario.

In my example above I started the program with

./test-prg >log &

so there was no reason for output stream to be bad, it was going to file anyway, but the error stream goes to terminal and becomes bad in case of stty tostop.

If I redirect it as follows:
./test-prg 2>log &
then the output stream will be bad. If I do not redirect any, both will be bad, and if I redirect both, both will be good. Tested this with
---------
#include<iostream>
#include<fstream>
using namespace std;
int main() {
    ofstream f("t");
    cout << "test" << endl;
    cerr << "test" << endl;
    f << cout.bad() << cerr.bad() << endl;
}
-----------

==============

However, for OpenFOAM only the non-redirected stderr leads to memory leak.

user4

2013-10-04 09:14

  ~0002533

Fixed in 4ff24f54a6d6df66630248c801dac3279c677a38

Thanks for hunting this one down!

Issue History

Date Modified Username Field Change
2013-09-19 13:27 user528 New Issue
2013-09-19 13:36 wyldckat Note Added: 0002501
2013-09-19 13:46 user528 Note Added: 0002502
2013-09-19 16:53 user17 Note Added: 0002506
2013-09-20 11:57 user528 Note Added: 0002507
2013-09-20 16:00 user17 Note Added: 0002508
2013-09-20 16:23 user528 Note Added: 0002509
2013-10-04 09:14 user4 Note Added: 0002533
2013-10-04 09:14 user4 Status new => resolved
2013-10-04 09:14 user4 Fixed in Version => 2.2.x
2013-10-04 09:14 user4 Resolution open => fixed
2013-10-04 09:14 user4 Assigned To => user4