View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0001941 | OpenFOAM | Bug | public | 2015-12-07 20:00 | 2015-12-14 13:07 |
Reporter | wyldckat | Assigned To | henry | ||
Priority | normal | Severity | feature | Reproducibility | sometimes |
Status | resolved | Resolution | fixed | ||
Product Version | dev | ||||
Summary | 0001941: Purging dep files that rely on outdated source code files - contribution attached | ||||
Description | The latest fvOption revamp resulted in files changing places and/or being renamed. Although we already have the scripts "wmakeLnIncludeAll", "wrmdepold" and "wrmdep", one critical script was still missing (or at least I can't find it) and attached is a proposition for it: wcleanAllDep This script is meant to be used after "wrmdepold" (if necessary) and before "wmakeLnIncludeAll", used at the top level of the project, which will take care of removing all ".dep" files that rely on dead symbolic links for "*.[CH]" files. It's meant to be used in the following scenario for a full "git pull + Allwmake" update of an existing build: foam git pull wrmdepold -rmdir src applications wcleanAllDep wmakeLnIncludeAll ./Allwmake This still needs some additional testing, since I'm not 100% certain yet if this will cover all broken/missing files. If it works as intended, this list of steps can be added to the git page for the next OpenFOAM-dev based release: http://www.openfoam.org/download/git.php | ||||
Additional Information | This issue has been bugging me for several years and I've finally managed to come around and propose a solution for it. At least since http://www.openfoam.org/mantisbt/view.php?id=321 , where Chris asked that I provide any suggestions about the last topic on that report, namely this item: Additional instructions would be welcome for when the latest git pull introduces new files that require a full (or partial) clean rebuild of the affected libraries and/or applications. | ||||
Tags | contribution | ||||
|
wcleanAllDep (2,690 bytes)
#!/bin/sh #------------------------------------------------------------------------------ # ========= | # \\ / F ield | OpenFOAM: The Open Source CFD Toolbox # \\ / O peration | # \\ / A nd | Copyright (C) 2011-2015 OpenFOAM Foundation # \\/ M anipulation | #------------------------------------------------------------------------------- # License # This file is part of OpenFOAM. # # OpenFOAM is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # OpenFOAM is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License # for more details. # # You should have received a copy of the GNU General Public License # along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>. # # Script # wcleanAllDep # # Description # search all the "src" and "application" directories of the project for # broken symbolic links for source code files and then remove all .dep # files that rely on those files that no longer exist. # #------------------------------------------------------------------------------ Script=${0##*/} usage() { exec 1>&2 while [ "$#" -ge 1 ]; do echo "$1"; shift; done cat<<USAGE Usage: $Script Remove all .dep files that rely on missing files USAGE exit 1 } #------------------------------------------------------------------------------ # Parse arguments and options #------------------------------------------------------------------------------ while [ "$#" -gt 0 ] do case "$1" in -h | -help) usage ;; *) usage "unknown option/argument: '$*'" ;; esac done [ -d bin -a -d src ] || usage "not in the project top level directory" echo "Purging all dep files that rely on files that no longer exist..." fileNameList=$(find -L src applications -name '*.[CH]' -type l \ -exec basename {} \;) for fileName in $fileNameList do cd src wrmdep -a $fileName cd ../applications wrmdep -a $fileName cd .. done #------------------------------------------------------------------------------ # Cleanup local variables and functions #------------------------------------------------------------------------------ unset Script usage #------------------------------------------------------------------------------ |
|
I forgot to mention the following details: - I didn't assign this immediately, as I wasn't certain to whom I should assign it, since this affects both development and documentation. - Additional testing and suggestions by fellow enthusiasts is very welcome! - (edit:) The attached file "wcleanAllDep" is meant to be placed inside the folder "wmake". I didn't suggest updating/upgrading another file with this feature, since none of the other scripts seemed to be the right choice, given the way all of the others do their own jobs. |
|
The reason I did not implement a wrmdepall is that now that the dep files are in the platforms directory they are cleaned out with wcleanMachine. Given that removing all the dep files with cause a complete rebuild I would recommend using wcleanMachine instead of wrmdepall. |
|
Mmm... sorry, then I made a bad choice of name for the script and I should have given the complete use scenario. The idea for the new script is to remove only the ".dep" files that really need to be rebuilt. With the latest update to fvOption, the following files changed location or were renamed: fvOptionList.C fvIOoptionList.H fvOptionIO.C fvOptionListTemplates.C fvOption.C createFvOptions.H makeFvOption.H fvOptionI.H fvOptionList.H fvIOoptionList.C fvOption.H Because these files were listed in several ".dep" files, this results in "wmake" simply terminating stating that a particular ".o" file could not be built because dependencies were missing, i.e. the files listed in ".dep" were not found. This is where the attached script tries to fill in this missing feature, namely to purge only the ".dep" files that need to be purged. As for other choices in names, the following didn't seem right to me: - "wclean dep" - would take out all dep files - "wrmdepold" - seemed conflicting/confusing to overload this script with this feature. - "wmakeLnIncludeAll" - not exactly part of its job, but we depend on the broken links this script will eliminate. - "wpurgedep" - this is outside of the current nomenclature, but it's the closest name to the actual function of the new script. |
|
I see. What I did after moving fvOption.H was wrmdep fvOption.H which removed all associated dep files and the build worked fine afterwards. It strikes me that your script is like wrmdep but instead of having to specify the files for which the associated dep files should be removed it is worked out automatically. So I think it should be an option of wrmdep. wrmdepold sholud probably also be an option of wrmdep. |
|
Mmm... OK, it didn't feel right to merge the two existing scripts, since operate in very different ways, but it would definitely consolidate the operations. Then should perhaps "wrmdep" change from simply this: Usage: wrmdep [-a | -all | all] [file] To the following options? Uses: wrmdep [-a | -all | all] [file] wrmdep [-o | -old] [-rmdir] [dir1 .. dirN] wrmdep -auto Where the second is re-adapted from "wrmdepold" and the third is re-adapted from the attached "wcleanAllDep". If you're not working on this already, I can propose such a re-adaptation of the script. |
|
I think this would be fine. The only option which doesn't really fit is -rmdir which doesn't relate to dep files at all and should probably be moved to wclean. I am not working on this so please go ahead and reorganize these scripts. |
|
wclean (5,757 bytes)
#!/bin/sh #------------------------------------------------------------------------------ # ========= | # \\ / F ield | OpenFOAM: The Open Source CFD Toolbox # \\ / O peration | # \\ / A nd | Copyright (C) 2011-2015 OpenFOAM Foundation # \\/ M anipulation | #------------------------------------------------------------------------------- # License # This file is part of OpenFOAM. # # OpenFOAM is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # OpenFOAM is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License # for more details. # # You should have received a copy of the GNU General Public License # along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>. # # Script # wclean # # Description # Clean up the wmake control directory Make/\$WM_OPTIONS and remove the # lnInclude directories generated for libraries. # #------------------------------------------------------------------------------ Script=${0##*/} usage() { while [ "$#" -ge 1 ]; do echo "$1"; shift; done cat<<USAGE Usage: $Script [OPTION] [dir] $Script [OPTION] target [dir [MakeDir]] options: -s | -silent ignored - for compatibility with wmake -help print the usage Clean up the wmake control directory Make/\$WM_OPTIONS and remove the lnInclude directories generated for libraries. The targets correspond to a subset of the 'wmake' special targets: all all subdirectories, uses any Allwclean or Allclean files if they exist exe | lib | libo | libso clean Make, any *.dep files and lnInclude directories rmdir remove only empty directories for the requested dir USAGE exit 1 } #------------------------------------------------------------------------------ # Parse arguments and options #------------------------------------------------------------------------------ while [ "$#" -gt 0 ] do case "$1" in -h | -help) usage ;; -s | -silent) # ignored - for compatibility with wmake shift ;; -*) usage "unknown option: '$*'" ;; *) break ;; esac done #------------------------------------------------------------------------------ # check arguments and change to the directory in which to run wclean #------------------------------------------------------------------------------ unset dir targetType MakeDir=Make if [ $# -ge 1 ] then if [ -d "$1" ] then dir=$1 else targetType=$1 fi # specified directory name: [ $# -ge 2 ] && dir=$2 # specified alternative name for the Make sub-directory: [ $# -ge 3 ] && MakeDir=$3 if [ "$dir" ] then cd $dir 2>/dev/null || { echo "$Script error: could not change to directory '$dir'" 1>&2 exit 1 } fi # provide some feedback echo "$Script ${dir:-./}" fi #------------------------------------------------------------------------------ # If target=rmdir, then only remove empty folders and exit #------------------------------------------------------------------------------ if [ "$targetType" = rmdir ] then # get subdirs ourselves so we can avoid particular directories for dir in $(find . -mindepth 1 -maxdepth 1 \ -type d \( -name .git -prune -o -print \) ) do echo "check dir: $dir" find $dir -depth -type d -empty -exec rmdir {} \; -print done exit 0 fi #------------------------------------------------------------------------------ # Recurse the directories tree #------------------------------------------------------------------------------ if [ "$targetType" = all ] then if [ -e Allwclean ] # consistent with Allwmake then ./Allwclean exit $? elif [ -e Allclean ] # often used for tutorial cases then ./Allclean exit $? else # For all the sub-directories containing a 'Make' directory for dir in `find . \( -type d -a -name Make \)` do echo $dir $0 ${dir%/Make} # parent directory - trim /Make from the end done fi fi # targetType is not needed beyond this point unset targetType #------------------------------------------------------------------------------ # Clean the 'Make' directory if present #------------------------------------------------------------------------------ if [ -d $MakeDir ] then objectsDir=$MakeDir/$WM_OPTIONS if echo $PWD | grep "$WM_PROJECT_DIR" then platformPath=$WM_PROJECT_DIR/platforms/${WM_OPTIONS} objectsDir=$platformPath$(echo $PWD | sed s%$WM_PROJECT_DIR%% ) fi rm -rf $objectsDir 2>/dev/null fi #------------------------------------------------------------------------------ # Remove the lnInclude directory if present #------------------------------------------------------------------------------ if [ -d lnInclude ] then rm -rf lnInclude 2>/dev/null fi #------------------------------------------------------------------------------ # Cleanup local variables and functions #------------------------------------------------------------------------------ unset Script usage #------------------------------------------------------------------------------ |
|
wrmdep (6,391 bytes)
#!/bin/sh #------------------------------------------------------------------------------ # ========= | # \\ / F ield | OpenFOAM: The Open Source CFD Toolbox # \\ / O peration | # \\ / A nd | Copyright (C) 2015 OpenFOAM Foundation # \\/ M anipulation | #------------------------------------------------------------------------------- # License # This file is part of OpenFOAM. # # OpenFOAM is free software: you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # OpenFOAM is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License # for more details. # # You should have received a copy of the GNU General Public License # along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>. # # Script # wrmdep [-a | -all | all] [file] # wrmdep [-o | -old] [dir1 .. dirN] # wrmdep -auto # # Description # This is a catch-all script for pruning .dep files, depending on the # provided arguments. # # [-a | -all | all] [file]: # Remove all .dep files from the object directory tree corresponding to the # current source derectory or remove only the .dep files referring to the # optionally specified [file]. With the -a/-all/all option the .dep files # are removed for all platforms rather than just the current platform. # # [-o | -old] [dir1 .. dirN]: # Remove *.dep files that are without a corresponding .C or .L source file. # This often occurs when a directory has been moved. # - print questionable directory and the *.dep file # # -auto: # Search all the "src" and "application" directories of the project for # broken symbolic links for source code files and then remove all .dep # files that rely on those files that no longer exist. # #------------------------------------------------------------------------------ Script=${0##*/} # Source the wmake functions . ${0%/*}/scripts/wmakeFunctions usage() { exec 1>&2 while [ "$#" -ge 1 ]; do echo "$1"; shift; done cat<<USAGE Usage: $Script [-a | -all | all] [file] Remove all .dep files or remove .dep files referring to <file> With the -a/-all/all option the .dep files are removed for all platform rather than just the current platform. $Script [-o | -old] [dir1 .. dirN] Remove *.dep files that are without a corresponding .C or .L file. This often occurs when a directory has been moved. - print questionable directory and file Note: For removing empty source code folders, run: wclean rmdir $Script -auto Search all the "src" and "application" directories of the project for broken symbolic links for source code files and then remove all .dep files that rely on those files that no longer exist. Must be executed in main project source code folder: $WM_PROJECT_DIR USAGE exit 1 } #------------------------------------------------------------------------------ # Parse arguments and options #------------------------------------------------------------------------------ # Default is for removing all .dep files in the current directory rmdepMode="files" # Default to processing only the current platform all= while [ "$#" -gt 0 ] do case "$1" in # Print help -h | -help) usage ;; # Non-stop compilation, ignoring errors -a | -all | all) all="all" shift ;; -o | -old) rmdepMode="oldFolders" shift ;; -auto) rmdepMode="autoMode" shift ;; -*) usage "unknown option: '$*'" ;; *) break ;; esac done # Check environment variables checkEnv case "$rmdepMode" in files) #------------------------------------------------------------------------- # Remove the selected .dep files from the object tree #------------------------------------------------------------------------- findObjectDir . # With the -a/-all option replace the current platform with a wildcard if [ "$all" = "all" ] then objectsDir=$(echo $objectsDir | sed s%$WM_OPTIONS%*% ) fi if [ "$#" -eq 0 ] then echo "removing all .dep files ..." find $objectsDir -name '*.dep' -print | xargs -t rm 2>/dev/null else echo "removing .dep files referring to $1 ..." find $objectsDir -name '*.dep' -exec grep "$1" '{}' \; \ -exec rm '{}' \; fi ;; oldFolders) # Default is the current directory [ "$#" -gt 0 ] || set -- . for checkDir do findObjectDir $checkDir if [ -d $objectsDir ] then echo "Searching: $objectsDir" else echo "Skipping non-dir: $objectsDir" continue fi find $objectsDir -name '*.dep' -print | while read depFile do depToSource $depFile # Check C++ or Flex source file exists if [ ! -r "$sourceFile" ]; then echo "rm $depFile" rm -f $depFile 2>/dev/null fi done done ;; autoMode) [ -d bin -a -d src ] || usage "not in the project top level directory" echo "Purging all dep files that rely on files that no longer exist..." fileNameList=$(find -L src applications -name '*.[CHL]' -type l \ -exec basename {} \;) for fileName in $fileNameList do echo "Purging from 'src': $fileName" cd src $Script -a $fileName echo "Purging from 'applications': $fileName" cd ../applications $Script -a $fileName cd .. done ;; esac #------------------------------------------------------------------------------ # Cleanup local variables and functions #------------------------------------------------------------------------------ unset Script usage rmdepMode all #------------------------------------------------------------------------------ |
|
Attached the following files, both for folder "wmake": - wclean - Added the target "rmdir", which will clean up empty directories, using the relevant code that was in "wrmdepold". It will exit after removing the empty folders and it will not do the other standard "wclean" operations. Appended to the usage description the following information: rmdir remove only empty directories for the requested dir - wrmdep - Upgraded to have both the functionality of "wrmdepold" and the previous "wcleanAllDep" proposition. The usage description was upgraded to the following: Usage: $Script [-a | -all | all] [file] Remove all .dep files or remove .dep files referring to <file> With the -a/-all/all option the .dep files are removed for all platform rather than just the current platform. $Script [-o | -old] [dir1 .. dirN] Remove *.dep files that are without a corresponding .C or .L file. This often occurs when a directory has been moved. - print questionable directory and file Note: For removing empty source code folders, run: wclean rmdir $Script -auto Search all the "src" and "application" directories of the project for broken symbolic links for source code files and then remove all .dep files that rely on those files that no longer exist. Must be executed in main project source code folder: $WM_PROJECT_DIR I didn't place the code for the 3 variants of "wrmdep" in the function script "scripts/wmakeFunctions", because this felt it was specific to the work done by "wrmdep". And with this, "wrmdepold" can be removed. |
|
Thanks Bruno, I agree with all these changes and it is good to remove "wrmdepold". The only change I am not sure about is the naming of the option "-auto" which I do not think conveys the purpose very well. I will consider alternatives. |
|
Thanks Bruno Resolved in OpenFOAM-dev by commit 1185daf9b76b5059770ad1355853c38111bc3856 |
|
I am adding a "-update" option to Allwmake to help with building after a "git pull" and it would be useful if the "-update" option on wrmdep would work at any level of the source tree. Why did you hard-code it to scan the "src" and "applications" directories at the top-level? |
|
The reason for the top level checking is simple: it's easier to look for broken links for the whole project and then purge the missing files from the ".dep" files that rely on them. In order for wrmdep to be generic, it would need to check all ".dep" files and check all files each ".dep" file relies upon. |
Date Modified | Username | Field | Change |
---|---|---|---|
2015-12-07 20:00 | wyldckat | New Issue | |
2015-12-07 20:00 | wyldckat | File Added: wcleanAllDep | |
2015-12-07 20:01 | wyldckat | Tag Attached: contribution | |
2015-12-07 20:04 | wyldckat | Note Added: 0005723 | |
2015-12-07 20:14 | wyldckat | Note Edited: 0005723 | |
2015-12-07 20:31 | henry | Note Added: 0005724 | |
2015-12-07 20:44 | wyldckat | Note Added: 0005725 | |
2015-12-07 20:57 | henry | Note Added: 0005726 | |
2015-12-07 21:29 | wyldckat | Note Added: 0005727 | |
2015-12-07 21:36 | henry | Note Added: 0005728 | |
2015-12-13 18:28 | wyldckat | File Added: wclean | |
2015-12-13 18:28 | wyldckat | File Added: wrmdep | |
2015-12-13 18:33 | wyldckat | Note Added: 0005746 | |
2015-12-13 18:34 | wyldckat | Assigned To | => henry |
2015-12-13 18:34 | wyldckat | Status | new => assigned |
2015-12-13 18:40 | henry | Note Added: 0005748 | |
2015-12-13 21:00 | henry | Note Added: 0005754 | |
2015-12-13 21:00 | henry | Status | assigned => resolved |
2015-12-13 21:00 | henry | Resolution | open => fixed |
2015-12-14 10:45 | henry | Note Added: 0005759 | |
2015-12-14 12:17 | wyldckat | Note Added: 0005760 | |
2015-12-14 12:17 | wyldckat | Status | resolved => feedback |
2015-12-14 12:17 | wyldckat | Resolution | fixed => reopened |
2015-12-14 13:07 | henry | Status | feedback => resolved |
2015-12-14 13:07 | henry | Resolution | reopened => fixed |