Man Linux: Main Page and Category List

NAME

       storeBackup.pl - fancy compressing managing checksumming hard-linking
       cp -ua

DESCRIPTION

       This program copies trees to another location. Every file copied is
       potentially compressed (see --exceptSuffix). The backups after the
       first backup will compaire the files with an md5 checksum with the last
       stored version. If they are equal, it will only make an hard link to
       it. It will also check mtime, ctime and size to recognize idential
       files in older backups very fast.  It can also backup big image files
       fast and efficiently on a per block basis (data deduplication).

       You can overwrite options in the configuration file on the command
       line.

SYNOPSIS

               $prog --help
       or
               $prog -g configFile
       or
               $prog [-f configFile] [-s sourceDir]
                     [-b backupDirectory] [-S series] [--print]
                     [-T tmpdir] [-L lockFile] [--unlockBeforeDel]
                     [--exceptDirs dir1,dir2,dir3] [--contExceptDirsErr]
                     [--includeDirs dir1,dir2,dir3]
                     [--exceptRule rule] [--includeRule rule]
                     [--exceptTypes types] [--cpIsGnu] [--linkSymlinks]
                     [--precommand job] [--postcommand job]
                     [--followLinks depth] [--highLatency]
                     [--ignorePerms] [--lateLinks [--lateCompress]]
                     [--checkBlocksSuffix suffix] [--checkBlocksMinSize size]
                     [--checkBlocksBS]
                     [--checkBlocksRule0 rule [--checkBlocksBS0 size]
                      [--checkBlocksCompr0] [--checkBlocksRead0 filter]
                      [--checkBlocksParallel0]]
                     [--checkBlocksRule1 rule [--checkBlocksBS1 size]
                      [--checkBlocksCompr1] [--checkBlocksRead1 filter]
                      [--checkBlocksParallel1]]
                     [--checkBlocksRule2 rule [--checkBlocksBS2 size]
                      [--checkBlocksCompr2] [--checkBlocksRead2 filter]
                      [--checkBlocksParallel2]]
                     [--checkBlocksRule3 rule [--checkBlocksBS3 size]
                      [--checkBlocksCompr3] [--checkBlocksRead3 filter]
                      [--checkBlocksParallel3]]
                     [--checkBlocksRule4 rule [--checkBlocksBS4 size]
                      [--checkBlocksCompr4] [--checkBlocksRead4 filter]
                      [--checkBlocksParallel4]]
                     [--checkDevices0 list [--checkDevicesDir0]
                      [--checkDevicesBS0] [checkDevicesCompr0]
                      [--checkDevicesParallel0]]
                     [--checkDevices1 list [--checkDevicesDir1]
                      [--checkDevicesBS1] [checkDevicesCompr1]
                      [--checkDevicesParallel1]]
                     [--checkDevices2 list [--checkDevicesDir2]
                      [--checkDevicesBS2] [checkDevicesCompr2]
                      [--checkDevicesParallel2]]
                     [--checkDevices3 list [--checkDevicesDir3]
                      [--checkDevicesBS3] [checkDevicesCompr3]
                      [--checkDevicesParallel3]]
                     [--checkDevices4 list [--checkDevicesDir4]
                      [--checkDevicesBS4] [checkDevicesCompr4]
                      [--checkDevicesParallel1]]
                     [--saveRAM] [-c compress] [-u uncompress] [-p postfix]
                     [--noCompress number] [--queueCompress number]
                     [--noCopy number] [--queueCopy number]
                     [--withUserGroupStat] [--userGroupStatFile filename]
                     [--exceptSuffix suffixes] [--addExceptSuffix suffixes]
                     [--minCompressSize size] [--comprRule]
                     [--doNotCompressMD5File] [--chmodMD5File] [-v]
                     [-d level][--progressReport number] [--printDepth]
                     [--ignoreReadError]
                     [--suppressWarning key] [--linkToRecent name]
                     [--doNotDelete] [--deleteNotFinishedDirs]
                     [--resetAtime] [--keepAll timePeriod] [--keepWeekday entry]
                     [[--keepFirstOfYear] [--keepLastOfYear]
                      [--keepFirstOfMonth] [--keepLastOfMonth]
                      [--firstDayOfWeek day] [--keepFirstOfWeek]
                      [--keepLastOfWeek] [--keepDuplicate] [--keepMinNumber]
                      [--keepMaxNumber]
                       | [--keepRelative] ]
                     [-l logFile
                      [--plusLogStdout] [--suppressTime] [-m maxFilelen]
                      [[-n noOfOldFiles] | [--saveLogs]]
                      [--compressWith compressprog]]
                     [--logInBackupDir [--compressLogInBackupDir]
                      [--logInBackupDirFileName logFile]]
                     [otherBackupSeries ...]

OPTIONS

       --help
                   show this help

       --generate, -g
                   generate a template of the configuration file

       --print
                   print configuration read from configuration file
                   or command line and stop

       --file, -f
                   configuration file (instead of or additionally to options
                   on command line)

       --sourceDir, -s
                   source directory (must exist)

       --backupDir, -b
                   top level directory of all backups (must exist)

       --series, -S
                   series directory, default is 'default'
                   relative path from backupDir

       --tmpdir, -T
                   directory for temporary files, default is </tmp>

       --lockFile, -L
                   lock file, if exists, new instances will finish if an old
                   is allready running, default is $lockFile

       --unlockBeforeDel
                   remove the lock file before deleting old backups
                   default is to delete the lock file after removing old
                   backups

       --exceptDirs, -e
                   directories to except from backing up (relative path),
                   wildcards are possible and should be quoted to avoid
                   replacements by the shell
                   use this parameter multiple times for multiple
                   directories

       --contExceptDirsErr
                   continue if one or more of the exceptional directories
                   do not exist (default is to stop processing)

       --includeDirs, -i
                   directories to include in the backup (relative path),
                   wildcards are possible and have to be quoted
                   use this parameter multiple times for multiple directories

       --exceptRule
                   Files to exclude from backing up.
                   see README: 'including / excluding files and directories'

       --includeRule
                   Files to include in the backug up - like exceptRule
                   see README: 'including / excluding files and directories'

       --writeExcludeLog
                   write a file name .storeBackup.notSaved.bz2 with the names
                   of all skipped files

       --exceptTypes
                   do not save the specified type of files, allowed: Sbcfpl
                       S - file is a socket
                       b - file is a block special file
                       c - file is a character special file
                       f - file is a plain file
                       p - file is a named pipe
                       l - file is a symbolic link
                       Sbc can only be saved when using option [cpIsGnu]

       --cpIsGnu
                   Activate this option if your systems cp is a full-featured
                   GNU version. In this case you will be able to also backup
                   several special file types like sockets.

       --linkSymlinks
                   hard link identical symlinks

       --precommand
                   exec job before starting the backup, checks lockFile (-L)
                   before starting (e.g. can be used for rsync)
                   stops execution if job returns exit status != 0
                   This parameter is parsed like a line in the configuration
                   file and normally has to be quoted.

       --postcommand
                   exec job after finishing the backup, but before erasing of
                   old backups  reports if job returns exit status != 0
                   This parameter is parsed like a line in the configuration
                   file and normally has to be quoted.

       --followLinks
                   follow symbolic links like directories up to depth
                   default = 0 -> do not follow links

       --highLatency
                   use this for a very high latency line (eg. vpn over
                   the internet) for better parallelization

       --ignorePerms
                   If this option choosen, files will not necessarily have
                   the same permissions and owner as the originals. This
                   speeds up backups on network drives a lot. Recovery with
                   storeBackupRecover.pl will restore them correctly.

       --lateLinks
                   do *not* write hard links to existing files in the backup
                   during the backup
                   you have to call the program storeBackupWriteLateLink.pl
                   later on your server if you set this flag to 'yes'

       --lateCompress
                   only in combination with --lateLinks
                   compression from files >= minCompressSize will be done
                   later, the file is (temporarily) copied into the backup

       --checkBlocksSuffix
                   Files with suffix for which storeBackup will make an md5
                   check on blocks of that file. Executed after
                   --checkBlocksRule(n)
                   This option can be repeated multiple times

       --checkBlocksMinSize
                   Only check files specified in --checkBlocksSuffix if there
                   file size is at least this value, default is 100M

       --checkBlocksBS
                   Block size for files specified with --checkBlocksSuffix
                   Default is $checkBlocksBSdefault (1 megabyte)

       --checkBlocksCompr
                   if set, the blocks generated due to checkBlocksSuffix
                   are compressed

       --checkBlocksRule0
                   Files for which storeBackup will make an md5 check
                   depending on blocks of that file.

       --checkBlocksBS0
                   Block size for option checkBlocksRule
                   Default is $checkBlocksBSdefault (1 megabyte)

       --checkBlocksCompr0
                   if set, the blocks generated due to this rule are
                   compressed

       --checkBlocksRead0
                   Filter for reading the file to treat as a blocked file
                   eg. 'gzip -d' if the file is compressed. Default is no
                   read filter.
                   This parameter is parsed like the line in the
                   configuration file and normally has to be quoted,
                   eg. 'gzip -9'

       --checkBlocksParallel0
                   Read files specified here in parallel to "normal" ones.
                   This only makes sense if they are on a different disk.
                   Default value is 'no'

       --checkBlocksRule1
       --checkBlocksBS1
       --checkBlocksCompr1
       --checkBlocksRead1
       --checkBlocksParallel1
       --checkBlocksRule2
       --checkBlocksBS2
       --checkBlocksCompr2
       --checkBlocksRead2
       --checkBlocksParallel2
       --checkBlocksRule3
       --checkBlocksBS3
       --checkBlocksCompr3
       --checkBlocksRead3
       --checkBlocksParallel3
       --checkBlocksRule4
       --checkBlocksBS4
       --checkBlocksCompr4
       --checkBlocksRead4
       --checkBlocksParallel4
       --checkDevices0
                   List of devices for md5 ckeck depending on blocks of these
                   devices

       --checkDevicesDir0
                   Directory where to store the backup of the device

       --checkDevicesBS0
                   Block size of option checkDevices0,
                   default is 1M (1 megabyte)

       --checkDevicesCompr0
                   Compress blocks resulting from option checkDevices0

       --checkDevicesParallel0
                   Read devices specified in parallel to the rest of the
                   backup. This only makes sense if they are on a different
                   disk. Default value is 'no'

       --checkDevices1
       --checkDevicesDir1
       --checkDevicesBS1
       --checkDevicesCompr1
       --checkDevicesParallel1
       --checkDevices2
       --checkDevicesDir2
       --checkDevicesBS2
       --checkDevicesCompr2
       --checkDevicesParallel2
       --checkDevices3
       --checkDevicesDir3
       --checkDevicesBS3
       --checkDevicesCompr3
       --checkDevicesParallel3
       --checkDevices4
       --checkDevicesDir4
       --checkDevicesBS4
       --checkDevicesCompr4
       --checkDevicesParallel4
       --saveRAM
                   write temporary dbm files in --tmpdir
                   use this if you do not have enough RAM

       --compress, -c
                   compress command (with options), default is <bzip2>
                   This parameter is parsed like the line in the
                   configuration file and normally has to be quoted,
                   eg. 'gzip -9'

       --uncompress, -u
                   uncompress command (with options), default is  <bzip2 -d>
                   This parameter is parsed like the line in the
                   configuration file and normally has to be quoted, eg.
                   'gzip -d'

       --postfix, -p
                   postfix to add after compression, default is <.bz2>

       --noCompress
                   maximal number of parallel compress operations,
                   default = choosen automatically

       --queueCompress
                   length of queue to store files before compression,
                   default = 1000

       --noCopy
                   maximal number of parallel copy operations,
                   default = 1

       --queueCopy
                   length of queue to store files before copying,
                   default = 1000

       --withUserGroupStat
                   write statistics about used space in log file

       --userGroupStatFile
                   write statistics about used space in name file
                   will be overridden each time

       --exceptSuffix
                   do not compress files with the following
                   suffix (uppercase included):
                   ('\.zip', '\.bz2', '\.gz', '\.tgz', '\.jpg', '\.gif',
                    '\.tiff', '\.tif', '\.mpeg', '\.mpg', '\.mp3', '\.ogg',
                    '\.gpg', '\.png')
                   This option can be repeated multiple times
                   If you do not want any compression, set this option
                   to '.*'

       --addExceptSuffix
                   like --exceptSuffix, but do not replace defaults, add

       --minCompressSize
                   Files smaller than this size will never be compressed
                   but copied

       --comprRule
                   alternative to --exceptSuffix and minCompressSize:
                   definition of a rule which files will be compressed

       --doNotCompressMD5File
                   do not compress .md5CheckSumFile

       --chmodMD5File
                   permissions of .md5CheckSumFile and corresponding
                   .storeBackupLinks directory, default is 0600

       --verbose, -v
                   verbose messages

       --debug, -d
                   generate debug messages, levels are 0 (none, default),
                   1 (some), 2 (many) messages, especially in
                   --exceptRule and --includeRule

       --resetAtime
                   reset access time in the source directory - but this will
                   change ctime (time of last modification of file status
                   information)

       --doNotDelete
                   check only, do not delete any backup

       --deleteNotFinishedDirs
                   delete old backups which where not finished
                   this will not happen if doNotDelete is set

       --keepAll
                   keep backups which are not older than the specified amount
                   of time. This is like a default value for all days in
                   --keepWeekday. Begins deleting at the end of the script
                   the time range has to be specified in format 'dhms', e.g.
                     10d4h means 10 days and 4 hours
                     default = 20d

       --keepWeekday
                   keep backups for the specified days for the specified
                   amount of time. Overwrites the default values choosen in
                   --keepAll. 'Mon,Wed:40d Sat:60d10m' means:
                     keep backups from Mon and Wed 40days + 5mins
                     keep backups from Sat 60days + 10mins
                     keep backups from the rest of the days like spcified in
                     --keepAll (default $keepAll)
                   if you also use the 'archive flag' it means to not
                   delete the affected directories via --keepMaxNumber:
                     a10d4h means 10 days and 4 hours and 'archive flag'
                   e.g. 'Mon,Wed:a40d5m Sat:60d10m' means:
                     keep backups from Mon and Wed 40days + 5mins + 'archive'
                     keep backups from Sat 60days + 10mins
                     keep backups from the rest of the days like specified in
                     --keepAll (default 30d)

       --keepFirstOfYear
                   do not delete the first backup of a year
                   format is timePeriod with possible 'archive flag'

       --keepLastOfYear
                   do not delete the last backup of a year
                   format is timePeriod with possible 'archive flag'

       --keepFirstOfMonth
                   do not delete the first backup of a month
                   format is timePeriod with possible 'archive flag'

       --keepLastOfMonth
                   do not delete the last backup of a month
                   format is timePeriod with possible 'archive flag'

       --firstDayOfWeek
                   default: 'Sun'. This value is used for calculating
                   --keepFirstOfWeek and --keepLastOfWeek

       --keepFirstOfWeek
                   do not delete the first backup of a week
                   format is timePeriod with possible 'archive flag'

       --keepLastOfWeek
                   do not delete the last backup of a week
                   format is timePeriod with possible 'archive flag'

       --keepDuplicate
                   keep multiple backups of one day up to timePeriod
                   format is timePeriod, 'archive flag' is not possible
                   default = 7d

       --keepMinNumber
                   Keep that miminum of backups. Multiple backups of one
                   day are counted as one backup. Default is 10.

       --keepMaxNumber
                   Try to keep only that maximum of backups. If you have more
                   backups, the following sequence of deleting will happen:
                   - delete all duplicates of a day, beginning with the old
                     once, except the last of every day
                   - if this is not enough, delete the rest of the backups
                     beginning with the oldest, but *never* a backup with
                     the 'archive flag' or the last backup

       --keepRelative, -R
                   Alternative deletion scheme. If you use this option, all
                   other keep options are ignored. Preserves backups depending
                   on their *relative* age. Example:
                   -R '1d 7d 61d 92b'
                   will (try to) ensure that there is always
                   - One backup between 1 day and 7 days old
                   - One backup between 5 days and 2 months old
                   - One backup between ~2 months and ~3 months old
                   If there is no backup for a specified timespan
                   (e.g. because the last backup was done more than 2 weeks
                   ago) the next older backup will be used for this timespan.

       --progressReport, -P
                   print progress report after each 'number' files

       --printDepth, -D
                   print depth of actual read directory during backup

       --ignoreReadError
                   ignore read errors in source directory; not readable
                   directories do not cause storeBackup.pl to stop processing

       --suppressWarning
                   suppress (unwanted) warnings in the log files;
                   to suppress warnings, the following keys can be used:
                     excDir (suppresses the warning that excluded directories
                            do not exist)
                     fileChange (suppresses the warning that a file has changed
                                during the backup)
                     crSeries (suppresses the warning that storeBackup had to
                              create the 'default' series)
                     hashCollision (suppresses the warning if a possible
                                   hash collision is detected)
                   This option can be repeated multiple times on the command line.

       --linkToRecent
                   after a successful backup, set a symbolic link to
                   that backup and delete existing older links with the
                   same name

       --logFile, -l
                   log file (default is STDOUT)

       --plusLogStdout
                   if you specify a log file with --logFile you can
                   additionally print the output to STDOUT with this flag

       --suppressTime
                   suppress output of time in logfile

       --maxFilelen, -m
                   maximal length of log file, default = 1e6

       --noOfOldFiles, -n
                   number of old log files, default = 5

       --saveLogs
                   save log files with date and time instead of deleting the
                   old (with [-noOldFiles])

       --compressWith
                   compress saved log files (e.g. with 'gzip -9')
                   default is 'bzip2'
                   This parameter is parsed like a line in the configuration
                   file and normally has to be quoted.

       --logInBackupDir
                   write log file (also) in the backup directory
                   Be aware that this log does not contain all error
                   messages of the one specified with --logFile!

       --compressLogInBackupDir
                   compress the log file in the backup directory

       --logInBackupDirFileName
                   filename to use for writing the above log file,
                   default is .storeBackup.log

       otherBackupSeries
                   List of other backup series to consider for
                   hard linking. Relative path from backupDir!
                   Format (examples):
                   backupSeries/2002.08.29_08.25.28 -> consider this backup
                   or
                   0:backupSeries ->last (youngest) in <backupDir>/backupSeries
                   1:backupSeries ->one before last in <backupDir>/backupSeries
                   n:backupSeries ->
                     n'th before last in <backupDir>/backupSeries
                   3-5:backupSeries ->
                     3rd, 4th and 5th in <backupDir>/backupSeries
                   all:backupSeries -> all in <backupDir>/backupSeries
                   default is to link to the last backup in every series

COPYRIGHT

       Copyright (c) 2000,2004,2008-2009 by Heinz-Josef Claes (see README).
       Published under the GNU General Public License or any later version.