NAME
dnshistory - processes various log file formats doing dns IP Address
lookups. Store these pairs in a database for later retrieval.
SYNOPSIS
dnshistory [OPTION]... [--file=FILE]
COPYRIGHT
dnshistory is Copyright © 2004, 2005, 2006, 2007 Stephen McInerney
DESCRIPTION
dnshistory currently processes Apache CLF and Combined logs, Squid
access logs, FTP xferlog files and iptables based logs. The log format
is auto-detected.
dnshistory has five modes of operation:
· Do Lookups. The default mode. Given a web log file, dnshistory
will perform DNS reverse lookups on each unique IP Address and
store the results in a history database.
· Do Translations. Given a raw web log file, dnshistory will make
use of a previously created history database and send to STDOUT
the same web log but with addresses replaced by the Fully
Qualified Domain Name as previously looked up.
· Do Recombining. Given two web log files, one raw and one
previously translated (eg. by using dnstran): Create a history
database from the values in these separate log files.
· Do Dump. Dump a given history database to STDOUT.
· Do Import. Import a previously dumped history database from a
given file.
· Show History. Given one or more IP Addresses on the command
line, show the history of those addresses.
The lookups make use of threads for maximum speed, and use the standard
resolution libraries on a system. Thus hosts files, NIS, LDAP and other
name resolution methods should work transparently.
It is strongly recommended that for massive lookups a DNS server is
"nearby". Preferably not a forwarding server.
dnshistory can read .gz files. STDIN is assumed to not be gz encoded.
OPTIONS
-L --dolookups
The default mode. Given a log file, either via STDIN or via
’-f’, do the lookups and store the results.
-T --dotranslate
Given a log file, either via STDIN or via ’-f’, lookup each IP
Address from the history database; replace the IP Address with
the FQDN and send the newly updated log line to STDOUT.
-R --dorecombine=FILE
Given a previously translated file (eg. via dnstran) via this
option for the names, do the lookups for a file given via STDIN
or ’-f’ and store the results. This file can be gz encoded.
The date/time of each stored entry is taken as being the actual
time for the lookup stored in the recombine log file. This is
probably incorrect, but "Good Enough".
-D --dodump
Dump the history database to STDOUT.
-I --doimport=FILE
Given a previously dumped database, import that into a new
database. Will fail and exit if the chosen database already
exists.
-S --showhistory
Given one or more IP Addresses show their history. Address are
the last item(s) on the command line. Addresses with no as yet
discovered FQDN will display ’NONAME’. The Date/Time displayed
is formatted as YYYY-MM-DD:hh:mm:ss, vs the ’seconds since
epoch’ for "--dodump"
--logtype=LOGTYPE
By default dnshistory will attempt to autodetect what type of
logfile is being processed. By using this option, the
autodetection is overridden. The choices are: auto, clf or www,
squid, ftp or iptables.
-c --cache=SIZE
Set the size of the memory cache to use. Value is in Mb.
Default is 20Mb.
-d --database=FILE
Change the default database file to use to store stateful data.
-f --file=FILE
Web Log File to process. This file can be gz encoded.
Will use STDIN if not set
-h --help
Help screen. Very brief.
-l --maxlookups=NUMBER
The maximum number of lookups to attempt. The default is 1. This
has not shown to be at all useful in testing...
-m --maxthreads=NUMBER
How many name lookup threads to spawn off. The default is 100.
Setting this too high can do evil things to bandwidth and the
CPU usage of any queried DNS server(s).
If doing lots of DNS queries, setting this too high can have a
very negative impact on the ability to successfully resolve
anything.
-t --timeout=VALUE
The time in seconds before a stored DNS value is deemed "old".
The default is 7 days.
-v --verbose
Verboseness of a run. More v’s will increase the level of
verbosity, up to a maximum of 5. All of the higher levels are
only of value for debugging purposes.
-V --version
Display the version information and exit
-w --wait
Delay time between query retries within a single run
RESULTS
At verbose level 1 (-v) some success/failure counters will be
displayed. As well as any problematic log lines to STDERR.
At verbose level 2 (-vv) lines that may not match up (eg. Due to
dnstran modifying referrers or URL’s) will be sent to STDERR.
EXAMPLES
A typical run, using a database in /tmp/ (/tmp/c.db), and a log file in
the current directory (test.log). First, do the lookups:
dnshistory -d /tmp/c.db -f test.log
Then the translation run for input into, for example, a web log
analyser:
dnshistory -T -d /tmp/c.db -f test.log | webalizer ....
Do three attempts on failed queries, with a 2 second delay between
retries:
dnshistory -l 3 -w 2 -d /tmp/c.db -f test.log
Lookup and Display the history of three IP Addresses:
127.0.0.1,192.168.1.254,10.10.10.10
dnshistory -S -d /tmp/c.db 127.0.0.1 192.168.1.254 10.10.10.10
Import a previously dumped database via dnsdb.dump
dnshistory -I dnsdb.dump -d /tmp/d.db
FILES
/var/lib/dnshistory/dnshistory.db
The default history database file.
BUGS
· Ignores IP Addresses located in the URL and Referrer fields.
AUTHOR
Stephen McInerney <spm@stedee.id.au>