NAME
rdup - generate a file list suitable for making backups
SYNOPSIS
rdup [-N timestamp] [OPTION]... FILELIST [DIR/FILE]...
DESCRIPTION
rdup is a utility inspired by rsync and the plan9 way of doing backups.
rdup itself does not backup anything. It only prints a list of files
that are changed, or all files in case of a null dump. It also handles
files that are removed, allowing for correct incremental backups. All
paths printed are absolute.
It works as follows, for a full dump
1. Crawl all directories, and print all the names found to standard
output.
2. Write a filelist with all the names found when crawling. Use
this list to calculate the correct incremental dump.
And for incremental dumps
1. Read in the filelist that was written when doing a full dump.
2. Crawl all the directories again.
3. Diff 1. and 2. to get two lists; one of removed items and one of
added/modified items.
4. Write the removed items to standard output
5. Write the modified/new items to standard output.
6. Write a new filelist.
7. Touch the time stamp file.
The FILELIST is a internal list rdup writes to, to keep track of which
files are in a backup. If you don’t want this (i.e. make a full
backup), use /dev/null here. The file /dev/null is handled specially by
rdup: if detected no new file list is written. This is useful when only
doing full backups and you want all files to be printed.
The DIRS/FILES can be specified multiple times. These are the
directories and files you want to backup. If omitted it defaults to the
current directory (.).
If the -N timestamp option is not given, all paths found are printed.
Only when a -N timestamp file is given, times can be compared and an
incremental output can be generated.
rdup prints a filelist to standard output. Subsequent programs in a
pipe line can be used to actually implement to backup scheme. If
FILELIST is empty or non existent all files in DIR are dumped. This is
the same as a null dump. After a run a new FILELIST is written. No
warning is given when FILELIST is an existing file, it just gets
overwritten by rdup. New runs will print out only those files that have
actually changed or are removed since the last run, thereby making
incremental backups possible.
Files are checked for changes by comparing the c-time (change time), if
this time is NEWER than the c-time of timestamp file the pathname is
printed to standard output. When files are removed they are also
printed to standard output, but they are prefixed with a ’-’. See
FORMAT below. The default format rdup uses is: "%p%T %b %m %u %g %l %s
%n\n"
Note, that rdup also supports hashing of files, this makes it possible
to check the local hash with the hash of the backed up file.
All errors are written to standard error. If the directory or file
does not exist, they are skipped and a warning is emitted.
The general idea is to be very UNIX like and create a bunch of simple
programs which each do a their specific thing very well. With rdup and
a small shell script (50 lines) one can implement encrypted and
compressed backups.
BACKUP POLICY
As rdup doesn’t backup anything, the backup policy; what you backup,
how you backup, how often and how you restore; is all left to the
scripts.
OPTIONS
-F format
Specify a printf-style format to use. See FORMAT below.
-N timestamp
use the c_time of file timestamp as the timestamp to decide what
to include in the incremental backup list. If timestamp does not
exist a full dump is performed. rdup will create/touch
timestamp after it has printed the file list. This means if
something goes wrong, you still have the original timestamp.
-M timestamp
As -N, but look at the m_time of timestamp.
-R Reverse the output of rdup. Tools accepting this ouput must
create leading directory as they see them. This option allows a
script -- running as a normal user -- to put files in a
directory which could have 0600 as its permission.
-E file
The file named ’file’ contains a list of Perl-compatible regular
expressions (PCRE) , one per line, that rdup will use to exclude
names. A ’#’ at the start of the line can be used to signal a
comment. Empty lines are discarded. The -0 option also affects
the format of this file.
If a directory is excluded, rdup won’t descend in that
directory, so all files in that directory are also excluded.
The directories leading up to the directory to be backed up can
not be excluded. If you use a command line like:
rdup /dev/null /home/miekg/bin
The directories ’/home’, ’/home/miekg’, ’/home/miekg/bin’ are
always printed.
If you want to exclude the file ’/home/miekg/blaat’ you need to
add the following regular expression: ’/home/miekg/blaat’.
If you want to exclude all .mozilla/cache directories of all
users you can use ’/home/.*/.mozilla/cache/.*’. This doesn’t
exclude the directory itself and I’m assuming that the users’
home directories are found under ’/home’.
Also note that rdup does not print directories with a trailing
slash.
-n Don’t honor .nobackup files. Normally if such a file is found
the directory and all files containing it, are not printed to
standard output. Now they are.
-c Print the files’ contents to standard output. This sets the
FORMAT string to: "%p%T %b %u %g %l %s\n%n%C"
Any file content is written in a block/chunk based manner. The
last block is signaled with a null block. A block start entry is
ascii and is formatted as follows: VVBLOCKBBBBB\n
Where ’VV’ is the version, now ’01’, then the literal string
’BLOCK’ and then the amount of bytes, typical ’08192’. And then
a newline.
An example:
01BLOCK08192
<START OF THE FIRST 8192 BYTES>01BLOCK00015
<ANOTHER 15 BYTES>01BLOCK00000
This option is used when streaming your backup to a remote
machine.
-r Only print removed files; entries that start with a ‘-’. This
option unsets -m.
-m Only print modified/new files; entries that start with a ‘+’.
This option unsets -r.
-v Be more verbose. When used once, processed .nobackup files will
be printed to standard error. When used twice each path will
also be printed to standard error. This is usefull in case of a
remote backup (-c) where the normal output is not seen. Using
tee(1) might even be better...
-s size
Don’t output files larger than size bytes. This can be used to
limit the amount of data to be transferred when doing a remote
backup. This option only applies to files.
-0 Delimit filelist with NULL’s instead of a newline. Use ’\0’ in
the format string to change rdup’s output.
-x Stay on the local filesystem.
-V Print rdup’s version.
-h Give an overview of the options.
BACKUPS
With:
rm -f timestamp && rdup -N timestamp LIST DIR
A full-dump filelist is printed to standard output. And with:
rdup -N timestamp LIST DIR
An incremental dump filelist is printed. The file timestamp is used to
save the exact time of rdup’s run. The file LIST is used to calculate
the correct incremental dump list, this is needed for files that are
removed, or have a different type.
FORMAT
The default format rdup uses is: "%p%T %b %u %g %l %s %n\n"
The following escape sequences are understood by rdup:
’p’: ’+’ if file is new/modified, ’-’ if removed
’b’: permission bits from lstat(2), octal in four digits
’m’: the file mode bits, st_mode from lstat(2), decimal digits
’u’: uid
’g’: gid
’l’: path name length
’s’: file size, zero if directory, major,minor for devices and
see CAVEATS for soft- and hardlinks.
’n’: path name
’N’: path name, but in case of a soft- or hardlink only the
link name.
’t’: time of modification (seconds from epoch)
’H’: the SHA1 hash of the regular file, all zeros ("0") for all
other types
’T’: file type
- normal file, l symlink, h hardlink, d directory,
c character device, b block device, p named pipe
and s socket.
’C’: the content of the file (none for all other types)
To delimit the output of rdup with NULLs you can use ’\0’ in the format
string.
FILELIST
rdup writes the FILELIST in the following format:
MODE DEV INODE LINk UID GID PATH_SIZE FILE_SIZE PATH
33204 2050 31970 * 1000 1000 42 887
/home/miekg/git/rdup/.git/hooks/commit-msg
Where MODE is the st_mode from stat(2), DEV is the dev id as returned
by the stat call and INODE is the inode number - rdup needs this info
to decide if a directory is renamed. LINK is equal to ’h’ for
hardlinks, other wise it is ’*’. UID and GID are the numeric user and
group id of the file. PATH_SIZE is the length of PATH. FILE_SIZE the
file size. And finally PATH is the path of the file.
A typical example is:
16893 2050 32085 * 1000 1000 30 4096
/home/miekg/git/rdup/.git/logs
OUTPUT FORMAT
The default output generated by rdup is formatted like:
+|-TYPE BITS UID GID PATH_SIZE FILE_SIZE PATH
Where:
o +|- plus or minus, indicating whether PATH should added or removed.
o TYPE the type of the file, see %T in FORMAT.
o BITS the permission of the file, this is a subset of the st_mode from
lstat(2). These are four octal digits.
o UID the numerical user id of PATH. Note that if the first character
of the line is ’-’ (i.e. remove) the UID will be zero.
o GID the numerical group id of PATH. Note that if the first character
of the line is ’-’ (i.e. remove) the GID will be zero.
o PATH_SIZE
the size of PATH.
o FILE_SIZE
the size of file pointed to by PATH. Note that if the first
character of the line is ’-’ (i.e. remove) the SIZE will be
zero. For directories this size will always be zero. Symbolic
and hard links are handled differently, see CAVEATS.
o PATH the pathname
A typical example might look like this:
+- 0755 1000 1000 8 11288 bin/rdup
This example show that the file should be backed up, has a user and
group id of 1000, the length of the path is 8 bytes, the size of the
file it 11288 and it has "bin/rdup" as a path.
Directories are always printed by rdup.
OUTPUT FORMAT WITH -c
The output generated by rdup -c is formatted like:
+|-TYPE BITS UID GID PATH_SIZE FILE_SIZE\n
PATH FILE_CONTENTS
This makes it possible possible for a remote shell script to receive
the actual file and make a backup.
All field are identical as described in OUTPUT FORMAT, but there is one
extra field and also see CAVEATS. The extra field is the
FILE_CONTENTS, which concatenates the entire file to standard output.
The output when using the -c is changed as follows, for directories:
the FILE_SIZE is zero and no content is printed. Thus:
+d 0755 1000 1000 11 0\n
/home/miekg
For regular files the following is a sample output:
+- 0644 1000 1000 32 6\n
/home/miekg/svn/rdup/trunk/aaa/ahello
Where aaa/a is a regular file containing the word ’hello\n’
CAVEATS
Soft- and hardlinks are handled differently when using %n, if you don’t
like this behavior use %N. The PATH name is generated from the link’s
name and its target. A symlink like
/home/bin/blaat -> /home/bin/bliep
is printed as ’/home/bin/blaat -> /home/bin/bliep’. The PATH_SIZE is
modified accordingly, where ’ -> ’ (4 characters) is also counted. The
FILE_SIZE is not needed for soft- or hardlinks, so it is set the length
of the link’s name -- the part left of the ’ ->’, in this case the
length of ’/home/bin/blaat’.
If rdup encounters a hardlink it is handled in the same way, but the
output type is set to ’h’ instead of ’l’. A hardlink is only detected
if rdup finds a file with the same inode and device number as a
previous one, i.e. such hardlinks must be contained in your backup.
Again note: with ’%N’ only the link’s name is printed. The FILE_SIZE is
still set to the length of the link’s name.
For devices the size field (%s) is changed to hold the major,minor
number of the device. So if a major number is 8 and the minor number is
0 (under Linux this is /dev/sda), its size will be 8,0. The numbers are
only separated with a comma ‘,’.
EXIT CODE
rdup return a zero exit code on success, otherwise 1 is returned. rdup
will abort if a file can not be concatenated, if a regular expression
can not be compiled or if a signal is received.
EXAMPLES
The next set of examples will all make a full dump -- because of the
use of /dev/null. See rdup-tr(1) for much more advanced examples.
cpio
rdup -R -F ’%N\n’ /dev/null ~/bin | cpio -o -Hcrc > my-
archive.cpio
Restore with:
cpio -i -d -Hcrc < my-archive.cpio
tar
rdup -F ’%N\n’ /dev/null ~/bin | tar c -f my-archive.tar -T
- --no-recursion
Restore:
tar x -f my-archive.tar
AUTHOR
Written by Miek Gieben.
REPORTING BUGS
Report bugs to <miek@miek.nl>.
BUGS/LIMITATIONS
See the -c flag for explanation about a small race condition when doing
remote dumps.
SEE ALSO
http:/www.miek.nl/projects/rdup is the main site of rdup. Also see
rdup-tr(1), rdup-up(1) or rdup-backups(7).
COPYRIGHT
Copyright (C) 2005-2009 Miek Gieben. This is free software. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.
Licensed under the GPL version 3. See the file LICENSE in the source
distribution of rdup.