NAME
mkls-lR - make ls-lR files on FTP server for mirror use
SYNOPSIS
mkls-lR -d dirpath [-a] [-n basename] [-t tod] [-h] [--help] [-V]
[--version] ["] [-Ipath ...] [path ...] ["]
DESCRIPTION
mkls-lR creates or updates the four files timezone, ls-lR.gz, ls-
lR.patch.gz amd ls-lR.times in the directory dirpath. With the -a
option a fifth file ls-lR is included. More than the disk space of ls-
lR is briefly used in /tmp.
The command ls -lR >ls-lR stores a long recursive directory listing in
the file ls-lR. FTP servers make these files once a day so a directory
listing is ready to download whenever a mirror logs in. For gigabyte
servers these files are several megabytes and are usually compressed to
ls-lR.gz. Many mirrors still download ls-lR.gz daily, compare it with
their own directory and then request the new files they want.
Many large archives now have ls-lR.gz about a megabyte and the average
total size of the daily new files is similar. The bandwidth and
(horror) charges for downloading the new files are increased by the
size of the daily ls-lR.gz.
ls-lR.patch.gz is a compressed unified difference of the previous and
the current ls-lR file. Mirror scripts and programs can download this
file and patch a local copy of the remote server’s ls-lR daily. ls-
lR.patch.gz is sometimes only 1% of the size of ls-lR.gz.
ls-lR.times contains just the modification times of the previous and
current ls-lR expressed as decimal seconds since the beginning of 1st
January 1970 in the server’s time zone. A mirror can download this 20
byte file first to check whether its local copy of the server’s ls-lR
can be patched with the ls-lR.patch.gz currently available or whether
it needs to download a fresh ls-lR.gz.
mkls-lR makes the above files with careful attention to preserving and
recording names and times. Mirrors can check the names and times
against their local copy of the server’s ls-lR before applying ls-
lR.patch.gz as well as checking whether the patch fits. This ensures
the mirror’s ls-lR is aligned with the server’s. mkls-lR does not make
new ls-lR files unless there is a difference in the directories further
saving in unnecessary mirror downloads.
Heavy server load increases ls runtime delaying ls-lR files until
mirrors miss downloading them. mkls-lR can store new ls-lR files in
/tmp for up to an hour before release at a precise time of day tod set
with the -t option. The time mkls-lR starts should under normal load
allow making of the final file ls-lR.times in the middle of the
interval before tod. This prevents mirrors missing ls-lR files when
server load is heavy even if mirroring starts only 1 minute after tod.
A mirror using the timezone file with tod can set its start time
precisely when appropriate.
mkls-lR makes and places the files as follows. If the file
/etc/timezone is readable mkls-lR copies it to timezone in dirpath. It
then makes ls-lR and ls-lR.patch.gz in the /tmp directory but relative
to dirpath. If ls-lR is made before time of day tod its modification
time is set to tod. After compressing ls-lR to ls-lR.gz, mkls-lR sets
the modification time of ls-lR.gz equal to ls-lR. By default mkls-lR
then deletes ls-lR but for the -a option it moves ls-lR to dirpath
instead. mkls-lR makes ls-lR.times last. If tod has passed or at is
unavailable, mkls-lR moves ls-lR.times, ls-lR.patch.gz and ls-lR.gz
into dirpath immediately but otherwise asks at to move them at tod
precisely.
mkls-lR is a bash script using GNU utilities intended for use on FTP
servers with archives over 100 MB in size.
EXAMPLES
mkls-lR would normally be started by cron shortly before it is desired
mirrors should log in. For these examples to work cron must have write
permission to the dirpath subdirectory path argument. If you have more
than one of these commands you should put them in a shell script and
start the script with cron.
50 20 * * * /usr/bin/mkls-lR -d /home/ftp/pub -t 21:00
At 8:50 pm local time start making the ls-lR files in /tmp.
Move them to /home/ftp/pub where mirrors can download them at
9:00 pm precisely.
mkls-lR -d /home/ftp/pub "-Ils-lR.*"
Make the ls-lR files in /home/ftp/pub and do not include them in
the listing. This stops any badly configured mirrors from
downloading the ls-lR files twice.
mkls-lR -d /home/ftp/tex-archive "-Ils-lR.* -IFILES.by*"
Include in the listing neither ls-lR files nor FILES.bydate nor
FILES.byname nor FILES.bysize. A common fault of CTAN mirrors
is downloading daily the 12 MB of index files mentioned. A
mirror using these ls-lR files cannot make that mistake.
mkls-lR -d /home/ftp/pub/debian -n ls-lR_stable\ "dists/slink/*"
The basename, ls-lR_stable, replaces ls-lR so the output
filenames become ls-lR_stable.gz, ls-lR_stable.patch.gz and ls-
lR_stable.times. Only subdirectories matching dists/slink/* and
their contents are listed so a mirror using the above files will
only download these. This example may be obsolete when you read
this manpage.
OPTIONS
-a Move ls-lR to directory dirpath immediately after compression.
Default: Delete after compression.
-d dirpath
Copy any /etc/timezone into directory dirpath. Make the files
ls-lR.gz, ls-lR.patch.gz and ls-lR.times relative to directory
dirpath. Move them into dirpath. Write permission to dirpath
is necessary.
-n basename
Substitute basename for ls-lR in each of the output file names
ls-lR occurs in. Default: ls-lR.
-t tod Move ls-lR.times, ls-lR.patch.gz, and ls-lR.gz from /tmp to
dirpath at a time of day tod not more than 1 hour after mkls-lR
starts. The modification times of ls-lR and ls-lR.gz are set to
tod. If -a requires ls-lR, it moves when created. tod must be
a single word date understands. Usually tod is in the format
hh:mm, for example 21:00. An example for testing is now+1min.
To release files at 5 minutes past midnight tomorrow use 23:65.
Default: now.
-h, --help
Display help message then exit.
" expand special pattern characters *, ?, [ and ] in path relative
to directory dirpath. Default: Expand relative to current
default directory.
-Ipath path is a shell pattern of files or directories not to appear in
ls-lR, ls-lR.gz or ls-lR.patch.gz.
path path is a shell pattern of files or directories to appear in ls-
lR, ls-lR.gz or ls-lR.patch.gz. Default: All files in and below
dirpath.
-V, --version
Display version and copyright message then exit.
SEE ALSO
getls-lR(1), cron(8), crontab(1), mirror(1), ftp(1), ftpd(8), ls(1),
diff(1), gzip(1), date(1), at(1), tzset(3), wu-ftpd(8).
mirror-lslR.html describes how programs like Lee McLoughlin’s perl
script mirror can use ls-lR.patch.gz and ls-lR.times.
AUTHOR
Ian Maclaine-cross 31st August 1999