NAME
tenshi - Log Monitoring and Reporting tool
SYNOPSIS
tenshi [ -c <conf file> ] [ -C ] [ -d <debug level> ] [ -f ] [ -h ] [
-p ] [ -P <pid file> ]
DESCRIPTION
tenshi is a log monitoring program, designed to watch one or more log
files for lines matching user defined regular expressions and report on
the matches. The regular expressions are assigned to queues which have
an alert interval and a list of mail recipients.
Queues can be set to send a notification as soon as there is a log line
assigned to it, or to send periodic reports.
Additionally, uninteresting fields in the log lines (such as PID
numbers) can be masked with the standard regular expression grouping
operators ( ). This allows cleaner and more readable reports. All
reports are separated by hostname and all messages are condensed when
possible.
The program reads a configuration file (tenshi.conf) and then forks a
deamon for monitoring the specified log files.
OPTIONS
-c <conf file>
Read configuration from file. The default file is
/etc/tenshi/tenshi.conf .
-C Perform a syntax check of the configuration file. The program
exits after parsing the configuration with either a return code
of 0 or an error.
-d <debug level>
Enable debug messages. Default level is 1 if none is specified,
level 2 enables SMTP connection debug messages. In this mode the
main process remains in the foreground.
-f Enable foreground mode. In this mode the main process operates
normally but remains in the foreground, this is needed for some
process supervisors.
-p Enable profiling mode. In this mode the main process remains in
the foreground and expects log lines to be fed to standard in.
When it receives an EOF it will stop processing. No alerts will
be sent in this mode, it is used solely for measuring tenshi’s
line processing speed. For example: time $(cat
/var/log/messages|tenshi -p > /dev/null)
-P <pid file>
Define the file containing the PID of the process, this
overrides any ’pidfile’ option present in the configuration
file.
CONFIGURATION FILE
All directives are shown with the standard default value where
applicable, if omitted the default value will be used.
EXTERNAL CONFIGURATION FILES
All configuration directives can be optionally split into different
configuration files and then read with the two following statements.
include <configuration file>
Parse the specified configuration file.
includedir <directory>
Parse all files inside <directory>. The files will be parsed in
alphabetical order, keep in mind that regexps order matters so
includedir should be used carefully, see REGEXP DEFINITIONS for
details.
STATIC OPTIONS
These options will be read the first time tenshi reads its config file.
They cannot be changed by re-reading the config file. If you change one
of these options and HUP tenshi it will die. You have been warned.
set uid tenshi
Specify the effective user ID of the process when in daemon
mode. The user must be able to read the selected log files, the
configuration file and write the specified pid file. Never use
privileged users here since it’s not usually necessary (log
files perms can be set accordingly with most syslog
implementations).
set gid tenshi
Specify the effective group ID of the process when in daemon
mode.
set pidfile /var/run/tenshi.pid
The file containing the PID of the process, useful for
start/stop scripts.
set logfile <log file path>
A log file to monitor, this may be specified multiple times to
watch more than one log file. Depending on your tail
implementation you might need to use the tail_multiple setting
for multiple files to work. This mode can be used along with
fifo and listen settings.
set tail /usr/bin/tail -q --follow=name --retry -n 0
Specify the path and arguments for the tail binary used for
reading the log files. The invocation must be tuned against your
current ’tail’ implementation. Default values are configured for
standard GNU coreutils tail. The --follow=name and --retry flags
should deal properly with log rotation, if missing on your
implementation we suggest that you use something like ’cp
/dev/null logfile’ as a safe way for clearing the log file upon
rotation.
set tail_multiple <on|off>
Some tail implementations do not handle more than one log file.
When this option is enabled multiple tail commands will be
forked, instead of a single command with multiple arguments.
This option is disabled by default.
set fifo <fifo path>
A FIFO file to monitor. This option allows you to use a syslog-
ng pipe() destination (or any other syslog implementation that
allows FIFO usage). This may be specified multiple times to
watch more than one fifo file. This option is meant to be used
only when the installed ’tail’ binary doesn’t behave properly
with FIFOs, please test your tail implementation before using
this. A suitable test is ’echo foo | tail -f’, if this dies
complaining of an illegal seek, you have a broken version of
tail installed. GNU tail is broken in this way, most BSD tails
will work. This mode can be used along with logfile and listen
settings.
set listen 0.0.0.0:514
Enables syslog server mode. With this option tenshi will bind to
the specified address:port pair and read messages acting like a
syslog server. We always recommend to filter the port
accordingly and possibly use something like stunnel for
encrypting the traffic. This mode can be used along with logfile
and fifo settings.
DYNAMIC OPTIONS
These options are set each time the config file is read. tenshi reads
its config file once on start-up and whenever it receives a HUP.
set sleep 5
The loop sleep time for the notification process. The value must
be <= 60 seconds.
set limit <number of lines>
The maximum number of messages per hostname allowed in a report,
any lines after the maximum will be omitted and a warning
included. If this option is omitted then no limit is applied.
set pager_limit <number of lines>
The maximum number of messages per hostname allowed in pager
friendly reports, any lines after the maximum will be omitted.
If this option is omitted then no limit is applied.
set logprefix <regexp>
All valid syslog messages are parsed by default, while non-
syslog ones are discarded unless the special noprefix queue is
set. This option allows to define an additional valid prefix for
watching other type of logs. If the regexp is matched then the
prefix is removed from the log and the first grouped string is
used for the hostname field. This may be specified multiple
times to watch many different non-syslog logs.
set mask ______
The mask for strings enclosed by the grouping operators ( ). See
the REGEXP DEFINITIONS section. ’set mask’ on its own will set
the mask to an empty string.
set mailserver localhost
The mail server to be contacted for sending out reports.
set mailtimeout 10
The timeout in seconds for mail server reply.
set subject tenshi report
The subject of report emails, the queue name is always
automatically appended.
set hidepid <on|off>
This option turns on automatic stripping of ’foo[1234]:’ style
PID strings from the start of log lines i.e. ’foo[1234]:’
becomes ’foo:’. This allows you to write regexs without worrying
about masking the PID. Bear in mind that any time you change
this option you will need to re-write your regex rules or they
will not work. This option is disabled by default.
set filter <queue> <filter path> <arguments>
When this option is enabled all reports matching the specified
queue will be passed as STDIN to the specified filter, the
resulting output is sent via smtp instead of the original
report. The full path of the filter application must be
specified.
set csv <cron_spec> <filter path> <arguments>
This feature allows periodic reporting, using a five-field cron-
style specification like the set queue option, to the specified
filter. The output is pre-formatted as CSV (Comma Separated
Values) with hostname,log,hits format. This feature was coded
for using AfterGlow (http://afterglow.sf.net) as a filter and
graphing tenshi output. Check the FAQ for sample usage.
set sort_order <descending|ascending>
The sorting order for reports. It can be either descending or
ascending, the number of messages is used as a key for sorting
the log messages. The default order is ascending.
set resolve <on|off>
This option turns on resolution of the fully qualified domain
name for the hostname passed along with log messages and, if
found, reports it along with the original one. This only affects
reports and not pager messages. The name resolution is cached in
order to avoid re-resolving addresses that have been seen
already, you have to restart or HUP tenshi in order to flush the
cache. This option is disabled by default.
QUEUES OPTIONS
All messages are assigned to queues. Every queue is processed
periodically according to its notification interval. There are four
default builtin queues, trash to which unwanted messages can be
assigned (think /dev/null), repeat which is used for smart repeat
messages handling, group and group_host , see REGEXP DEFINITIONS for
details. There’s also a special noprefix queue, read further for
details about it.
All queues are automatically flushed before shutdown when a SIGTERM is
received. Please see section SIGNALS for additional information.
The syntax is the following:
set queue <queue_name> <mail_from> [pager:]<mail_to> <cron_spec>
[<subject>]
<queue_name>
The queue name. Can be any alphanumeric character string except
for the builtin queues name.
<mail_from>
The mail sender for reports related to the queue.
<mail_to>
The mail recipient(s) for reports related to the queue. Multiple
address can be specified, separated by commas. Using the pager:
prefix enables a pager friendly report.
[<cron_spec>]
This is a five-field cron-style specification for when the
reports should be emailed. Ranges and skip values are supported
as per the de facto crontab syntax with a few exceptions. Please
see crontab man page for crontab syntax explanation. The
supported day names are: Mon, Tue, Wed, Thu, Fri, Sat, Sun.
Monday is 1, Sunday 0 or 7. Supported month names are: Jan, Feb,
Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec. Day and Month
names are not case sensitive. Additionally, ’now’ can be
specified for immediate notifications.
<subject>
This is the subject for to use for email reports regarding this
queue. If this isn’t specified then the default subject will be
used.
The special noprefix queue can be used and defined like any other queue
with the difference that it will get all messages that don’t match any
configured prefix.
Examples:
set queue report tenshi@localhost sysadmin@localhost [0 9-17 * * *]
set queue report tenshi@localhost sysadmin@localhost [30 18 * * *]
set queue report tenshi@localhost sysadmin@localhost [*/10 * * * *]
set queue critical tenshi@localhost sysadmin@localhost,noc@localhost
[now] CRITICAL WARNING -
set queue pager tenshi@localhost
pager:sysadmin_pager@localhost,pager:noc_pager@localhost [now] ALERT
REGEXP DEFINITIONS
All valid syslog messages are matched against standard perl regexps,
all regexps are defined with the following syntax:
<queue_name>[,<queue_name>[:<escalation_number>]..] <regexp>
The regexps are evaluated in order so a matched message is not checked
against the subsequent regexps. Keep this in mind when assembling the
configuration file. It’s advisable to catch all messages by placing an
all matching regexp at the end of the configuration file. It’s also
good for performance having trash rules not logically connected with
other matching rules at the beginning of the section. Multiple queues
can be defined with a comma separated list, builtin queues cannot be
used when using this syntax.
If an escalation number is provided for a queue, the matched message
will only be placed into the queue when <escalation_number> messages
have matched the regexp. The queue will receive the message that
matched the regexp at the time of escalation, with a count equal to the
escalation number. The count of messages matching the regexp will be
reset when the left most queue mentioned in the queue list is mailed.
The left most queue cannot have an escalation number unless it is the
only queue listed. When the number of messages that match the regexp
reaches the greatest escalation number mentioned, escalation will begin
again into the escalation queues, modulus the greatest escalation
number. For example, using the queues ‘a,b:10,c:50’, when 10 messages
match the regexp, a message will go into b, when 50 match, one will go
into c. At 60, another will go into b, and at 100, another into c, 110
to b, 150 to c, and so on. Escalation numbers must be positive integers
greater than zero and must be listed in increasing order from left to
right. All queues without escalation numbers must be listed more left
than the queues with escalation numbers.
The standard grouping operators ( ) can be used for string masking,
literal "(" and ")" can be protected with the standard quotation
operator "\". There’s a lot of documentation about regular expressions,
a good start could be perl perlre and perlretut manual pages.
You can also use the (?: ) operators to use groups without masking.
This allows you to match, for example, output from several programs in
a similar format. There is an example of this below (the sudo/su
line).
The builtin queue repeat can be used for special handling of "last
message repeated x times" style log lines. When the assigned regexps
are matched the line count for the last line received from the same
host is incremented by the first grouped string. Keep in mind that it
is possible for syslog lines to be received from remote hosts out of
order. If this happens you should not use this feature because tenshi
will mis-report line counts.
The builtin queue group can be used to group sets of regex together to
speed up line matching. If a line fails to match a regex assigned to
the group queue then tenshi will skip all the regex up until the next
group_end statement. Nested groups are allowed. An example of this is
included below.
The builtin group_host queue can be used for selective hostname
matching. Like the group queue it is also terminated with the group_end
statement. All regex definitions within that group will only apply if
the hostname associated to the log entries matches the regex passed to
the group_host definition.
The regexs below assume hidepid is turned on. If you have it turned off
then you will need to add in \[(.+)\] to the regex following the progam
name to get them to work.
For example: mail ^sendmail: (.+): to=(.+),(.+)delay=(.+) becomes: mail
^sendmail\[(.+)\]: (.+): to=(.+),(.+)delay=(.+)
Examples:
trash ^xinetd
repeat ^(?:last message repeated|above message repeats) (\d+) time
group ^sendmail:
mail ^sendmail: (.+): to=(.+),(.+)delay=(.+)
mail ^sendmail: (.+): to=(.+),(.+)relay=(.+),(.+)stat=Sent
group_end
group_host mailserver1
mail1 ^sendmail
mail1 ^sendmail:.+
critical,mail1 ^sendmail:.+SYSERR.+
group_end
mail ^ipop3d: Login user=(.+)
critical,report ^sshd: Illegal user
general,urgent:200,critical:1000 ^sshd: Illegal user
root ^sshd\(pam_unix\): session opened for user root by root\(uid=0\)
report ^sshd: Accepted rsa for (.+) from (.+) port (.+)
trash ^sshd
critical ^(?:sudo|su):
critical,pager ^Oops
misc .*
SIGNALS
tenshi can handle different signals sent to the process, here’s the
list of supported ones:
TERM flush all queues and then exit
INT flush all queues and then exit
USR1 flush any queues which have reached their notification interval
USR2 force all queues to be flushed, even if they have not reached
their notification interval
HUP force all queues to be flushed, even if they have not reached
their notification interval, re-read the config file and
continue as normal.
WARNING: If you change a STATIC OPTION in the config file and send
tenshi a HUP it will die. You will need to restart tenshi for changes
to STATIC OPTIONs to take effect.
EXAMPLES
See the included tenshi.conf.
REQUIREMENTS
tenshi needs a working ’tail’ implementation when not using FIFO mode.
It also requires Net::SMTP module for mailing reports, which should be
included in your perl installation, and IO::BufferedSelect. If you miss
any of them you can grab them at CPAN (http://www.cpan.org) or using
the CPAN shell (‘perl -e shell -MCPAN‘).
BUGS
- Double quotation characters present in your logs might break csv
output (depending on how you pipe/process it in the filter) since
there’s no escape code (yet).
Please report any bugs you find at <tenshi@inversepath.com>
TODO
- custom reports layout
- add other actions besides mail notifications
- find/design a logo
- everything that’s reported at
http://dev.inversepath.com/trac/tenshi/report
Any volunteers ? ;)
LICENSE
tenshi is distributed under the terms of the following ISC-style
license:
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR
BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES
OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS
SOFTWARE.
DISTRIBUTION
The tenshi project page is http://dev.inversepath.com/trac/tenshi
NOTES
tenshi was formerly known as wasabi but the name was changed as we were
informed that wasabi is a registered a trademark relating to another
piece of software.
Two mailing lists are also available:
<tenshi-user@lists.inversepath.com> for general discussion, subscribe
by sending a message to <tenshi-user+subscribe@lists.inversepath.com>
<tenshi-announce@lists.inversepath.com> for announcements, subscribe by
sending a message to <tenshi-announce+subscribe@lists.inversepath.com>
SEE ALSO
It should be noted that tenshi was initially a perl rewrite of oak
(http://www.ktools.org).
Friedl, Jeffrey E. F. Mastering Regular Expressions, 2nd Edition.
OReilly
AUTHORS
tenshi was initially written by Andrea Barisani. It is now maintained
by Andrea Barisani and Rob Holland.
Copyright 2004-2009 Andrea Barisani <andrea@inversepath.com>
Rob Holland <rob@inversepath.com>