View Issue Details

IDProjectCategoryView StatusLast Update
0001941OpenFOAMBugpublic2015-12-14 13:07
Reporterwyldckat Assigned Tohenry  
PrioritynormalSeverityfeatureReproducibilitysometimes
Status resolvedResolutionfixed 
Product Versiondev 
Summary0001941: Purging dep files that rely on outdated source code files - contribution attached
DescriptionThe latest fvOption revamp resulted in files changing places and/or being renamed. Although we already have the scripts "wmakeLnIncludeAll", "wrmdepold" and "wrmdep", one critical script was still missing (or at least I can't find it) and attached is a proposition for it: wcleanAllDep

This script is meant to be used after "wrmdepold" (if necessary) and before "wmakeLnIncludeAll", used at the top level of the project, which will take care of removing all ".dep" files that rely on dead symbolic links for "*.[CH]" files.

It's meant to be used in the following scenario for a full "git pull + Allwmake" update of an existing build:

  foam
  git pull
  wrmdepold -rmdir src applications
  wcleanAllDep
  wmakeLnIncludeAll
  ./Allwmake

This still needs some additional testing, since I'm not 100% certain yet if this will cover all broken/missing files.
If it works as intended, this list of steps can be added to the git page for the next OpenFOAM-dev based release: http://www.openfoam.org/download/git.php
Additional InformationThis issue has been bugging me for several years and I've finally managed to come around and propose a solution for it. At least since http://www.openfoam.org/mantisbt/view.php?id=321 , where Chris asked that I provide any suggestions about the last topic on that report, namely this item:

  Additional instructions would be welcome for when the latest git pull introduces new files that require a full (or partial) clean rebuild of the affected libraries and/or applications.
Tagscontribution

Activities

wyldckat

2015-12-07 20:00

updater  

wcleanAllDep (2,690 bytes)   
#!/bin/sh
#------------------------------------------------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | Copyright (C) 2011-2015 OpenFOAM Foundation
#    \\/     M anipulation  |
#-------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM.
#
#     OpenFOAM is free software: you can redistribute it and/or modify it
#     under the terms of the GNU General Public License as published by
#     the Free Software Foundation, either version 3 of the License, or
#     (at your option) any later version.
#
#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
#     for more details.
#
#     You should have received a copy of the GNU General Public License
#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
#
# Script
#     wcleanAllDep
#
# Description
#     search all the "src" and "application" directories of the project for
#     broken symbolic links for source code files and then remove all .dep
#     files that rely on those files that no longer exist.
#
#------------------------------------------------------------------------------
Script=${0##*/}

usage() {
    exec 1>&2
    while [ "$#" -ge 1 ]; do echo "$1"; shift; done
    cat<<USAGE
Usage: $Script

    Remove all .dep files that rely on missing files

USAGE
    exit 1
}


#------------------------------------------------------------------------------
# Parse arguments and options
#------------------------------------------------------------------------------

while [ "$#" -gt 0 ]
do
    case "$1" in
    -h | -help)
        usage
        ;;
    *)
        usage "unknown option/argument: '$*'"
        ;;
    esac
done

[ -d bin -a -d src ] || usage "not in the project top level directory"

echo "Purging all dep files that rely on files that no longer exist..."
fileNameList=$(find -L src applications -name '*.[CH]' -type l \
                    -exec basename {} \;)

for fileName in $fileNameList
do
    cd src
    wrmdep -a $fileName

    cd ../applications
    wrmdep -a $fileName

    cd ..
done

#------------------------------------------------------------------------------
# Cleanup local variables and functions
#------------------------------------------------------------------------------

unset Script usage


#------------------------------------------------------------------------------
wcleanAllDep (2,690 bytes)   

wyldckat

2015-12-07 20:04

updater   ~0005723

Last edited: 2015-12-07 20:14

I forgot to mention the following details:

 - I didn't assign this immediately, as I wasn't certain to whom I should assign it, since this affects both development and documentation.

 - Additional testing and suggestions by fellow enthusiasts is very welcome!

 - (edit:) The attached file "wcleanAllDep" is meant to be placed inside the folder "wmake". I didn't suggest updating/upgrading another file with this feature, since none of the other scripts seemed to be the right choice, given the way all of the others do their own jobs.

henry

2015-12-07 20:31

manager   ~0005724

The reason I did not implement a wrmdepall is that now that the dep files are in the platforms directory they are cleaned out with wcleanMachine. Given that removing all the dep files with cause a complete rebuild I would recommend using wcleanMachine instead of wrmdepall.

wyldckat

2015-12-07 20:44

updater   ~0005725

Mmm... sorry, then I made a bad choice of name for the script and I should have given the complete use scenario.

The idea for the new script is to remove only the ".dep" files that really need to be rebuilt.
With the latest update to fvOption, the following files changed location or were renamed:

  fvOptionList.C
  fvIOoptionList.H
  fvOptionIO.C
  fvOptionListTemplates.C
  fvOption.C
  createFvOptions.H
  makeFvOption.H
  fvOptionI.H
  fvOptionList.H
  fvIOoptionList.C
  fvOption.H

Because these files were listed in several ".dep" files, this results in "wmake" simply terminating stating that a particular ".o" file could not be built because dependencies were missing, i.e. the files listed in ".dep" were not found.
This is where the attached script tries to fill in this missing feature, namely to purge only the ".dep" files that need to be purged.

As for other choices in names, the following didn't seem right to me:

  - "wclean dep" - would take out all dep files

  - "wrmdepold" - seemed conflicting/confusing to overload this script with this feature.

  - "wmakeLnIncludeAll" - not exactly part of its job, but we depend on the broken links this script will eliminate.

  - "wpurgedep" - this is outside of the current nomenclature, but it's the closest name to the actual function of the new script.

henry

2015-12-07 20:57

manager   ~0005726

I see. What I did after moving fvOption.H was

wrmdep fvOption.H

which removed all associated dep files and the build worked fine afterwards.

It strikes me that your script is like wrmdep but instead of having to specify the files for which the associated dep files should be removed it is worked out automatically. So I think it should be an option of wrmdep. wrmdepold sholud probably also be an option of wrmdep.

wyldckat

2015-12-07 21:29

updater   ~0005727

Mmm... OK, it didn't feel right to merge the two existing scripts, since operate in very different ways, but it would definitely consolidate the operations.

Then should perhaps "wrmdep" change from simply this:

  Usage: wrmdep [-a | -all | all] [file]

To the following options?

  Uses:
     wrmdep [-a | -all | all] [file]
     wrmdep [-o | -old] [-rmdir] [dir1 .. dirN]
     wrmdep -auto

Where the second is re-adapted from "wrmdepold" and the third is re-adapted from the attached "wcleanAllDep".
If you're not working on this already, I can propose such a re-adaptation of the script.

henry

2015-12-07 21:36

manager   ~0005728

I think this would be fine. The only option which doesn't really fit is -rmdir which doesn't relate to dep files at all and should probably be moved to wclean.

I am not working on this so please go ahead and reorganize these scripts.

wyldckat

2015-12-13 18:28

updater  

wclean (5,757 bytes)   
#!/bin/sh
#------------------------------------------------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | Copyright (C) 2011-2015 OpenFOAM Foundation
#    \\/     M anipulation  |
#-------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM.
#
#     OpenFOAM is free software: you can redistribute it and/or modify it
#     under the terms of the GNU General Public License as published by
#     the Free Software Foundation, either version 3 of the License, or
#     (at your option) any later version.
#
#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
#     for more details.
#
#     You should have received a copy of the GNU General Public License
#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
#
# Script
#     wclean
#
# Description
#     Clean up the wmake control directory Make/\$WM_OPTIONS and remove the
#     lnInclude directories generated for libraries.
#
#------------------------------------------------------------------------------
Script=${0##*/}

usage() {
    while [ "$#" -ge 1 ]; do echo "$1"; shift; done
    cat<<USAGE

Usage: $Script [OPTION] [dir]
       $Script [OPTION] target [dir [MakeDir]]

options:
  -s | -silent      ignored - for compatibility with wmake
  -help             print the usage

Clean up the wmake control directory Make/\$WM_OPTIONS and remove the
lnInclude directories generated for libraries.

The targets correspond to a subset of the 'wmake' special targets:
  all               all subdirectories, uses any Allwclean or Allclean
                    files if they exist
  exe | lib | libo | libso
                    clean Make, any *.dep files and lnInclude directories
  rmdir
                    remove only empty directories for the requested dir

USAGE
    exit 1
}


#------------------------------------------------------------------------------
# Parse arguments and options
#------------------------------------------------------------------------------

while [ "$#" -gt 0 ]
do
    case "$1" in
    -h | -help)
        usage
        ;;
    -s | -silent)    # ignored - for compatibility with wmake
        shift
        ;;
    -*)
        usage "unknown option: '$*'"
        ;;
    *)
        break
        ;;
    esac
done


#------------------------------------------------------------------------------
# check arguments and change to the directory in which to run wclean
#------------------------------------------------------------------------------

unset dir targetType
MakeDir=Make

if [ $# -ge 1 ]
then

    if [ -d "$1" ]
    then
        dir=$1
    else
        targetType=$1
    fi

    # specified directory name:
    [ $# -ge 2 ] && dir=$2

    # specified alternative name for the Make sub-directory:
    [ $# -ge 3 ] && MakeDir=$3

    if [ "$dir" ]
    then
        cd $dir 2>/dev/null || {
            echo "$Script error: could not change to directory '$dir'" 1>&2
            exit 1
        }
    fi

    # provide some feedback
    echo "$Script ${dir:-./}"
fi


#------------------------------------------------------------------------------
# If target=rmdir, then only remove empty folders and exit
#------------------------------------------------------------------------------

if [ "$targetType" = rmdir ]
then
    # get subdirs ourselves so we can avoid particular directories
    for dir in $(find . -mindepth 1 -maxdepth 1 \
                           -type d \( -name .git -prune -o -print \) )
    do
        echo "check dir: $dir"
        find $dir -depth -type d -empty -exec rmdir {} \; -print
    done

    exit 0
fi

#------------------------------------------------------------------------------
# Recurse the directories tree
#------------------------------------------------------------------------------

if [ "$targetType" = all ]
then
    if [ -e Allwclean ]       # consistent with Allwmake
    then
        ./Allwclean
        exit $?
    elif [ -e Allclean ]      # often used for tutorial cases
    then
        ./Allclean
        exit $?
    else
        # For all the sub-directories containing a 'Make' directory
        for dir in `find . \( -type d -a -name Make \)`
        do
            echo $dir
            $0 ${dir%/Make}   # parent directory - trim /Make from the end
        done
    fi
fi

# targetType is not needed beyond this point
unset targetType


#------------------------------------------------------------------------------
# Clean the 'Make' directory if present
#------------------------------------------------------------------------------

if [ -d $MakeDir ]
then
    objectsDir=$MakeDir/$WM_OPTIONS
    if echo $PWD | grep "$WM_PROJECT_DIR"
    then
        platformPath=$WM_PROJECT_DIR/platforms/${WM_OPTIONS}
        objectsDir=$platformPath$(echo $PWD | sed s%$WM_PROJECT_DIR%% )
    fi
    rm -rf $objectsDir 2>/dev/null
fi


#------------------------------------------------------------------------------
# Remove the lnInclude directory if present
#------------------------------------------------------------------------------

if [ -d lnInclude ]
then
    rm -rf lnInclude 2>/dev/null
fi


#------------------------------------------------------------------------------
# Cleanup local variables and functions
#------------------------------------------------------------------------------

unset Script usage


#------------------------------------------------------------------------------
wclean (5,757 bytes)   

wyldckat

2015-12-13 18:28

updater  

wrmdep (6,391 bytes)   
#!/bin/sh
#------------------------------------------------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | Copyright (C) 2015 OpenFOAM Foundation
#    \\/     M anipulation  |
#-------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM.
#
#     OpenFOAM is free software: you can redistribute it and/or modify it
#     under the terms of the GNU General Public License as published by
#     the Free Software Foundation, either version 3 of the License, or
#     (at your option) any later version.
#
#     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
#     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
#     FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
#     for more details.
#
#     You should have received a copy of the GNU General Public License
#     along with OpenFOAM.  If not, see <http://www.gnu.org/licenses/>.
#
# Script
#     wrmdep [-a | -all | all] [file]
#     wrmdep [-o | -old] [dir1 .. dirN]
#     wrmdep -auto
#
# Description
#     This is a catch-all script for pruning .dep files, depending on the
#     provided arguments.
#
#     [-a | -all | all] [file]:
#     Remove all .dep files from the object directory tree corresponding to the
#     current source derectory or remove only the .dep files referring to the
#     optionally specified [file].  With the -a/-all/all option the .dep files
#     are removed for all platforms rather than just the current platform.
#
#     [-o | -old] [dir1 .. dirN]:
#     Remove *.dep files that are without a corresponding .C or .L source file.
#     This often occurs when a directory has been moved.
#         - print questionable directory and the *.dep file
#
#     -auto:
#     Search all the "src" and "application" directories of the project for
#     broken symbolic links for source code files and then remove all .dep
#     files that rely on those files that no longer exist.
#
#------------------------------------------------------------------------------
Script=${0##*/}

# Source the wmake functions
. ${0%/*}/scripts/wmakeFunctions

usage() {
    exec 1>&2
    while [ "$#" -ge 1 ]; do echo "$1"; shift; done
    cat<<USAGE
Usage:

    $Script [-a | -all | all] [file]

        Remove all .dep files or remove .dep files referring to <file>
        With the -a/-all/all option the .dep files are removed for all
        platform rather than just the current platform.

    $Script [-o | -old] [dir1 .. dirN]

        Remove *.dep files that are without a corresponding .C or .L file.
        This often occurs when a directory has been moved.
          - print questionable directory and file

        Note: For removing empty source code folders, run: wclean rmdir

    $Script -auto

        Search all the "src" and "application" directories of the project for
        broken symbolic links for source code files and then remove all .dep
        files that rely on those files that no longer exist.
        Must be executed in main project source code folder: $WM_PROJECT_DIR

USAGE
    exit 1
}


#------------------------------------------------------------------------------
# Parse arguments and options
#------------------------------------------------------------------------------

# Default is for removing all .dep files in the current directory
rmdepMode="files"

# Default to processing only the current platform
all=

while [ "$#" -gt 0 ]
do
    case "$1" in
    # Print help
    -h | -help)
        usage
        ;;
    # Non-stop compilation, ignoring errors
    -a | -all | all)
        all="all"
        shift
        ;;
    -o | -old)
        rmdepMode="oldFolders"
        shift
        ;;
    -auto)
        rmdepMode="autoMode"
        shift
        ;;
    -*)
       usage "unknown option: '$*'"
       ;;
    *)
       break
       ;;
    esac
done

# Check environment variables
checkEnv


case "$rmdepMode" in
files)

    #-------------------------------------------------------------------------
    # Remove the selected .dep files from the object tree
    #-------------------------------------------------------------------------

    findObjectDir .

    # With the -a/-all option replace the current platform with a wildcard
    if [ "$all" = "all" ]
    then
        objectsDir=$(echo $objectsDir | sed s%$WM_OPTIONS%*% )
    fi

    if [ "$#" -eq 0 ]
    then
        echo "removing all .dep files ..."
        find $objectsDir -name '*.dep' -print | xargs -t rm 2>/dev/null
    else
        echo "removing .dep files referring to $1 ..."
        find $objectsDir -name '*.dep' -exec grep "$1" '{}' \; \
             -exec rm '{}' \;
    fi

    ;;

oldFolders)

    # Default is the current directory
    [ "$#" -gt 0 ] || set -- .

    for checkDir
    do
        findObjectDir $checkDir

        if [ -d $objectsDir ]
        then
            echo "Searching: $objectsDir"
        else
            echo "Skipping non-dir: $objectsDir"
            continue
        fi

        find $objectsDir -name '*.dep' -print | while read depFile
        do
            depToSource $depFile

            # Check C++ or Flex source file exists
            if [ ! -r "$sourceFile" ];
            then
                echo "rm $depFile"
                rm -f $depFile 2>/dev/null
            fi
        done
    done

    ;;

autoMode)

    [ -d bin -a -d src ] || usage "not in the project top level directory"

    echo "Purging all dep files that rely on files that no longer exist..."
    fileNameList=$(find -L src applications -name '*.[CHL]' -type l \
                        -exec basename {} \;)

    for fileName in $fileNameList
    do
        echo "Purging from 'src': $fileName"
        cd src
        $Script -a $fileName

        echo "Purging from 'applications': $fileName"
        cd ../applications
        $Script -a $fileName

        cd ..
    done

    ;;

esac

#------------------------------------------------------------------------------
# Cleanup local variables and functions
#------------------------------------------------------------------------------

unset Script usage rmdepMode all


#------------------------------------------------------------------------------
wrmdep (6,391 bytes)   

wyldckat

2015-12-13 18:33

updater   ~0005746

Attached the following files, both for folder "wmake":

  - wclean - Added the target "rmdir", which will clean up empty directories, using the relevant code that was in "wrmdepold". It will exit after removing the empty folders and it will not do the other standard "wclean" operations. Appended to the usage description the following information:

      rmdir
                    remove only empty directories for the requested dir


  - wrmdep - Upgraded to have both the functionality of "wrmdepold" and the previous "wcleanAllDep" proposition. The usage description was upgraded to the following:

    Usage:

        $Script [-a | -all | all] [file]

            Remove all .dep files or remove .dep files referring to <file>
            With the -a/-all/all option the .dep files are removed for all
            platform rather than just the current platform.

        $Script [-o | -old] [dir1 .. dirN]

            Remove *.dep files that are without a corresponding .C or .L file.
            This often occurs when a directory has been moved.
              - print questionable directory and file

            Note: For removing empty source code folders, run: wclean rmdir

        $Script -auto

            Search all the "src" and "application" directories of the project for
            broken symbolic links for source code files and then remove all .dep
            files that rely on those files that no longer exist.
            Must be executed in main project source code folder: $WM_PROJECT_DIR


I didn't place the code for the 3 variants of "wrmdep" in the function script "scripts/wmakeFunctions", because this felt it was specific to the work done by "wrmdep".

And with this, "wrmdepold" can be removed.

henry

2015-12-13 18:40

manager   ~0005748

Thanks Bruno, I agree with all these changes and it is good to remove "wrmdepold". The only change I am not sure about is the naming of the option "-auto" which I do not think conveys the purpose very well. I will consider alternatives.

henry

2015-12-13 21:00

manager   ~0005754

Thanks Bruno

Resolved in OpenFOAM-dev by commit 1185daf9b76b5059770ad1355853c38111bc3856

henry

2015-12-14 10:45

manager   ~0005759

I am adding a "-update" option to Allwmake to help with building after a "git pull" and it would be useful if the "-update" option on wrmdep would work at any level of the source tree. Why did you hard-code it to scan the "src" and "applications" directories at the top-level?

wyldckat

2015-12-14 12:17

updater   ~0005760

The reason for the top level checking is simple: it's easier to look for broken links for the whole project and then purge the missing files from the ".dep" files that rely on them.

In order for wrmdep to be generic, it would need to check all ".dep" files and check all files each ".dep" file relies upon.

Issue History

Date Modified Username Field Change
2015-12-07 20:00 wyldckat New Issue
2015-12-07 20:00 wyldckat File Added: wcleanAllDep
2015-12-07 20:01 wyldckat Tag Attached: contribution
2015-12-07 20:04 wyldckat Note Added: 0005723
2015-12-07 20:14 wyldckat Note Edited: 0005723
2015-12-07 20:31 henry Note Added: 0005724
2015-12-07 20:44 wyldckat Note Added: 0005725
2015-12-07 20:57 henry Note Added: 0005726
2015-12-07 21:29 wyldckat Note Added: 0005727
2015-12-07 21:36 henry Note Added: 0005728
2015-12-13 18:28 wyldckat File Added: wclean
2015-12-13 18:28 wyldckat File Added: wrmdep
2015-12-13 18:33 wyldckat Note Added: 0005746
2015-12-13 18:34 wyldckat Assigned To => henry
2015-12-13 18:34 wyldckat Status new => assigned
2015-12-13 18:40 henry Note Added: 0005748
2015-12-13 21:00 henry Note Added: 0005754
2015-12-13 21:00 henry Status assigned => resolved
2015-12-13 21:00 henry Resolution open => fixed
2015-12-14 10:45 henry Note Added: 0005759
2015-12-14 12:17 wyldckat Note Added: 0005760
2015-12-14 12:17 wyldckat Status resolved => feedback
2015-12-14 12:17 wyldckat Resolution fixed => reopened
2015-12-14 13:07 henry Status feedback => resolved
2015-12-14 13:07 henry Resolution reopened => fixed