NAME
t-prot - TOFU Protection - Display Filter for RFC 5322 messages
SYNOPSIS
t-prot [OPTIONS]...
DESCRIPTION
This program is a filter to improve the readability of internet
messages (emails and usenet posts) by *hiding* some annoying parts,
e.g. mailing list footers, signatures, and TOFU (see definition below),
as well as squeezing sequences of blank lines or punctuation. The
program also detects TOFU or a high quoting ratio in a message (so you
may take appropriate action, e.g. when submitting messages to a mailing
list or a news server).
The filter is written in Perl and relies on input to be a single
message conforming to RFC 822 or its successors, RFC 2822 and RFC 5322.
In messages conforming to MIME (RFCs 2045-2049) t-prot handles
text/plain parts, others are not touched.
Already reformatted messages are handled well: the script was initially
designed to cope with the output of the MUA mutt (which is the reason
for not using standard CPAN modules for handling messages).
T-prot offers example configuration files for mutt, Heirloom mailx and
metamail. Also coming with the t-prot package is the example S-Lang
macro t-prot.sl for using t-prot from within slrn. There is a
proof-of-concept filter for INN2, which you will have to adapt to the
needs of the news site you host. For use with sendmail's alias(5) file,
please see below (the option -p provides an example line).
OPTIONS
If you do not specify any options, t-prot does ... nothing. Every
feature you want must be turned on explicitly. Admittedly, we have
quite a number of options for t-prot. To limit confusion they are
grouped into five sections: Input/Output Options, Advertisement And
Mailing List Footers, Filtering Options, Detection Options, and Other
Options. While the others should be quite clear, filtering and
detection might deserve a word (or two).
If you want to tune the appearance of your mail from within your MUA
(or news messages from within your NUA), then go for the Filtering
Options section.
If you want to use t-prot to check on mails before they are submitted
to mailing lists, fed to your news server, or delivered by your MDA,
then have a peek at the Detection Options section. You may accept or
reject/bounce messages depending on t-prot's result.
INPUT/OUTPUT OPTIONS
-i=FILE
Defines an input file; default is '-' i.e. STDIN.
-o=FILE
Defines the output file; default is STDOUT.
--body Input consists just of the message's body. There are no RFC 5322
header lines.
NOTE: This does not work with --pgp-short, and multipart
messages will not be detected due to missing headers.
--lax-security
Allow insecure writing method. DO NOT USE UNLESS YOU REALLY KNOW
WHAT YOU ARE DOING. (This ugly workaround is needed for some
early mutt versions and should NEVER be used as a default,
otherwise it will probably turn into a security issue.)
You can use this option safely to enable -o /dev/null (or other
files which cannot be changed with the user's privileges).
--max-lines=x
Maximum number of lines a message may count (with headers). If
the message is longer than x lines, the message will not be
processed but printed unmodified. Exit status will be EX_DATAERR
except when called with -Mmutt.
ADVERTISEMENT AND MAILING LIST FOOTERS
-a "commercial signature": Hides "footers" (signatures) from
commercial email providers.
This option compares the last lines of the message body with any
footer file found in the directory specified with -A DIRECTORY
(which is mandatory for this option). The comparison is done by
perl's index() function (please try perldoc -f index for
details).
NOTE: This option is not needed if --ftr-ad is specified.
--ftr-ad
"enable aggressive ad footer matching": With this option
enabled, t-prot makes footer detection really greedy: We assume
that commercial email providers aren't even frightened to append
changing texts *under* their ads which are appended to the
message body. Because these texts even have changing lengths we
simply detect the lines of the footer *anywhere* in the body of
the message and assume that everything below belongs to the
footer. (Man, if life where always that easy! ;)
With this option even GMX ads should be easy to hide -- you buy
this with a slight performance hit (which is the reason this
option is disabled by default), and with the possibility that
sometimes the algorithm is just a little *too* greedy.
NOTE: This requires a directory with footer files to be given
with option -A=DIRECTORY.
-A=DIRECTORY
"ad footer directory": Defines the directory which contains the
advertisement list footers (one footer per file) which are to be
tested when removing them with options -a or --ftr-ad.
This option is also needed if you do not want signature lengths
to be counted wrong or fullquotes get undetected when an ad
footer is appended at the bottom of the message (especially when
using -S or -t).
-l "list signature": Hides "footers" (signatures) from mailing
lists. Footer detection works like the -a option.
NOTE: This requires a directory with footer files to be given
with option -L=DIRECTORY. -l is not needed if --ftr-ml is
specified.
--ftr-ml
"enable aggressive mailing list footer matching": With this
option enabled t-prot makes footer detection really greedy:
Should be helpful with broken list servers, or even if your
email provider munges the bodies of your messages.
Works similar to --ftr-ad, just that it is intended for mailing
list footers.
NOTE: This requires a directory with footer files to be given
with option -L=DIRECTORY.
-L=DIRECTORY
"list footer directory": Defines the directory which contains
the mailing list footers (one footer per file) which are to be
tested when removing them with the options -l or --ftr-ml.
This option is also needed if you do not want signature lengths
to be counted wrong or fullquotes get undetected when a mailing
list footer is appended at the bottom of the message (especially
when using -S or -t).
FILTER OPTIONS
--bigq[=n[,x]]
"shrink big quotes": Blocks of quotes with more than n lines
will be shrunk to x lines. Defaults are 30 for n and 10 for x.
-c[=n] "compress": Squeezes a sequence of blank lines to just n blank
lines. n defaults to 2.
--diff Tolerate unified diff (see diff(1) and patch(1)) appended after
the signature (which usually makes the signature too long to be
valid).
Also, protect diff standard output from hiding (which would
otherwise be easy prey for -t).
-e "ellipsis": Squeezes a sequence of four or more dots,
exclamation marks, or question marks to only three dots or
marks, respectively.
--fixind
Fix broken quotes to adhere to RFC 3676 by removing spaces
between quote characters and adding a space after them.
NOTE: This may produce false positives if spaces in between
quote characters are intended (thus changing the quoting level,
see RFC 3676 for details).
--groupwise
Hides TOFU as produced by Novell Groupwise.
-k "anti Kammquote": Tries (not too aggressively) to fix those
broken zig-zag-shaped lines wrapped around by some MUAs which
are known as "Kammquoting" in German.
NOTE: This option is considered stable by now. However,
sometimes Kammquotes should have been removed but weren't.
Please send a bug report if this happens to you (after carefully
reading the BUGS and REPORTING BUGS section of this man page,
that is).
Please also note that enabling this option is quite a
performance hit.
--kdiff=n
Minimum length difference between two lines for wrapped line
detection on Kammquotes. For details, please see the source
code.
Anyway, lower values make the algorithm more aggressive, higher
values make Kammquotes harder to detect. Default is 20.
Requires -k.
--kmaxl=n
Maximum line length for wrapped line detection on Kammquotes.
For details, please see the source code.
Anyway, higher values make the algorithm more aggressive, lower
values make Kammquotes harder to detect. Default is 80.
Requires -k.
--kminl=n
Minimum line length for wrapped line detection on Kammquotes.
For details, please see the source code.
Anyway, lower values make the algorithm more aggressive, higher
values make Kammquotes harder to detect. Default is 65.
Requires -k.
--locale=LOCALE
Specify which locale to use for correct parsing of your MUA's
formatting of the displayed message (usually it is the locale
your MUA uses). Right now this option is only used when -Mmutt
is specified, but this may change in future. You need the Perl
module Locale::gettext for this feature.
NOTE: If you use mutt or gnupg with locales, t-prot will only
work correctly if you specify the corresponding locale string.
Alternatively, you can use the environment variables LC_ALL,
LC_MESSAGES, or LANG to specify the locale string.
NOTE also: You also have to make sure you are running t-prot
with matching gnupg and mutt versions. T-prot detects gnupg and
mutt locales of the recent stable versions of those programs,
earlier versions might not work well with a recent version of t-
prot. There are patches available to make t-prot fit into
environments with some other mutt and gnupg versions.
-M, --mua=MUA
"mail user agent": Turn on special treatment for some mail user
agents. (Right now only mutt(1) is supported, but more might be
added in future.) Caveat: If your MUA is supported by this
feature you must ensure t-prot makes use of it when called from
within your MUA to work as desired.
-m "Microsoft TOFU": Hides TOFU as given by some Microsoft mailers.
(You all surely know these fullquotes beginning with
"----- Original Message -----"
and some header lines...)
--ms-smart
Burn CPU cycles trying to be smart with MS style TOFU: If there
are PGP signed parts inside the TOFU, the text still might
conceal other message parts and therefore should not be deleted.
Please note that this is probably just a waste of time because
most MS Outlook users who do produce this kind of TOFU won't
care about making their messages the least bit readable or even
predictable. So this option will probably just be interesting
for mutt message hooks (to activate it on demand when you know
the sender tries to write legible messages).
Requires -Mmutt and -m.
--pgp-move
Move PGP verification output to bottom; requires -Mmutt.
--pgp-move-vrf
Move PGP verification output to bottom only if verification
shows a good signature and the signature could be verified as
authentic (using a trust path). If there is any problem with the
signature, the PGP output should not be moved so the user is
more likely to notice. Requires -Mmutt.
NOTE: If gpg is terminated before finished (e.g. hitting Ctrl-C,
or using kill(1)), we cannot always detect if the check was
interrupted. Though t-prot tries to be smart, there will be
false positives.
--pgp-short
Hide non-relevant PGP key uids; requires -Mmutt.
-r "rip header off": Hides all mail header lines.
--reply
Subject lines with multiple reply prefixes (Re: and translations
into other languages) get squeezed to only one prefix.
-S[=n] "supression of overlong signatures": Signatures are to be n
lines (not including the one containing dash-dash-space) or
less. If there are more, it is probably not that spirited after
all. So with this option you trade it for a truely nice line.
If no n is given, default is 4. (We do not recommend using a
value other than 4. Consider this old-fashioned, but we actually
do *like* RFC conformance.)
NOTE: The line containing "-- " ist not counted when testing for
an overlong signature, but it is included when displaying how
many lines were deleted.
-s "signature deletion": Hides signatures, i.e. all lines after a
"signature dashes" line, i.e. a line with three characters:
dash-dash-space (no more, no less).
--sani Sanitize headers "To:", "From:" and "Subject:": Quoted-printable
gets fixed to the corresponding chars. German Umlauts are
translated to their "ae", "oe", "ue" pendants.
Useful e.g. for searching by subject within MUAs like Berkeley
mailx.
--sigsmax[=n]
"maximum number of tolerated signatures": Here you can define
how many signatures you accept to be treated as such. (Most
significant behaviour is when microsoft style quotes are
removed. Experts please see the code for the more subtle
implications of this option.)
Leave empty or specify zero to have an unlimited number of sigs.
Default is 1.
--spass
"SpamAssassin workaround": SpamAssassin (available at
http://spamassassin.org/) often is configured that it adds some
lines to the message body containing information about the spam
criteria which were found matching for the message. This option
enables an extra test to avoid false positives for Microsoft
style TOFU on such messages.
-t "TOFU deletion": Hides "traditional style" TOFU, where each line
begins with the indent string ">".
-w "whitespace deletion": Hides trailing whitespace (sequences of
space and tab). CAVEAT: This may lead to interesting effects
with crossposts between mailing lists or with undetected
signature attempts.
DETECTION OPTIONS
-P=MESSAGE
"user defined bounce message for picky delivery": You may
specify your own bounce message to be returned when we try to
deliver an email and bounce it because there is TOFU inside. See
-p.
-p[=ADDRESS]
"picky delivery": If we really find some TOFU, abort with exit
code EX_UNAVAILABLE. Otherwise redirect the message to ADDRESS
if given.
Intended for use from within mail delivery agents (MDAs) or mail
transport agents (MTAs), or even from within INN, so the message
bounces if TOFU is detected, and does not get on *your* nerves.
:)
As an example for usage with sendmail, put this line into your
alias file and invoke newaliases:
notofu: |"/usr/local/bin/t-prot -mt -p=user@mydomain"
This will bounce messages for <notofu@domainname> if any TOFU is
detected inside the message, and deliver it to <user@mydomain>
otherwise. Note that TOFU is only detected if you specify -t
respectively -m.
PLEASE be careful not to bounce messages to mailing lists!
--check[=FLAGS]
Run checks. If successful, print an error message and quit with
an appropriate exit code. Useful e.g. for rejecting messages
from within INN2.
Flags are separated by commas (no whitespaces), and can be the
following (right now just one flag):
ratio[=n]
If the quoting ratio is n or more, the message is rejected. Must
be between 0 and 1, or else it is entirely disabled. Default is
0.75 (i.e., 75% of the message lines are quotes).
-d, --debug
Print envelope info to syslog when bouncing TOFU contaminated
email. Default syslog facility is mail.debug. Requires -p.
OTHER OPTIONS
-h, --help
Displays a short help text with a summary on all options, and
exits.
-v, --version
Prints the current version number and exits.
ENVIRONMENT
The environment variables LC_ALL, LC_MESSAGES, and LANG are read and
respected when interpreting output by mutt or gnupg (unless they are
overruled by the --locale option). T-prot's own output is English
regardless of any locale setting.
EXIT STATUS
On program exit, t-prot uses exit codes from /usr/include/sysexits.h
and thus behaves in a manner that sendmail and others understand when
calling t-prot.
Currently, the codes used are
EX_OK
EX_USAGE
EX_DATAERR
EX_UNAVAILABLE
EX_SOFTWARE
EX_IOERR
If, however, perl fails to compile and execute t-prot, perl's normal
exit codes will be returned.
TOFU?
TOFU is an abbreviation which mixes German and English words; it
expands to "text oben, full-quote unten" which means "text above - full
quote below" and describes the style of so many users who let their
mailer or newsreader quote everything of the previous message and just
add some text at the top; obviously they think that quoted text must
not be changed at all. This is quite annoying as it needlessly sends a
lot of data even when it is not required. Some editing of messages is
desired. Please point these people to the page
http://www.river.com/users/share/etiquette/edit.html - thank you!
PERFORMANCE
There are several ways to fine-tune t-prot's performance:
Some command line options are quite grave a performance hit -- do not
use -k and especially --ms-smart if you are content without them.
Checking for special footers is very costly as well. Put as few footer
files as absolutely needed in any footer directory.
All PGP related options are eating up lots of CPU time. Try to avoid
them on unsigned and unencrypted messages.
When calling t-prot from within mutt, you might use mutt's folder-hook
and message-hook facilities to turn options on only when needed, e.g.
to set up a different footer directory for each mailing list folder.
TROUBLESHOOTING
Q: I want to make my mailing list footer files match more different
mailing list footers. Can I use regular expressions, or how can
I accomplish that?
A: Nope, regexp's do not work here. The comparison is made by the
perl builtin index() function (see perldoc for more detailed
info), so you must exactly match the beginning of the line. The
longer the line you specify, the more precise the match; if your
line is empty you match unconditionally.
Q: I use the options -l and -L to supress mailing list footers when
displaying messages in mutt(1). This does work sometimes, but
sometimes it does not: the footer is not detected, and therefore
full quotes are not deleted and signatures are detected as too
long (which they aren't).
A: This might occur if the message is badly encoded, so mutt cannot
resolve all encoded characters, e.g. if you have an encoded
message on a mailing list, and majordomo appends a mailing list
footer in a different encoding (or even plain us-ascii). "-- "
simply does not match "--=20".
Another problem are non-us-ascii characters. Just avoid them,
and everything should work fine.
See the preceding Q+A for a solution.
Q: I want to write a message which contains parts that should *not*
be deleted even when filtered with t-prot. Is this possible?
A: Yes, but please do not spread word of it. Make unobstrusive use
of the verbatim instruction:
#v+
This line is protected from being filtered by t-prot !!!!!!!
#v-
Text coming now is not.
AUTHOR
Written by Jochen Striepe <t-prot@tolot.escape.de>.
COPYRIGHT
All of the documentation and software included in the t-prot releases
is copyrighted by Jochen Striepe (except when explicitly stated
otherwise).
Copyright (C) 2001-2010 Jochen Striepe. All rights reserved.
Redistribution and use, with or without modification, are permitted
provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. All advertising materials mentioning features or use of this
software must display the following acknowledgement:
This product includes software developed by Jochen Striepe and
others.
3. Neither the name of the author nor the names of any contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
IDEAS AND INSPIRATION
Many good ideas, bug reports and support from (in alphabetical order)
Bjoern Buerger, Bjoern Laessig, Christian Borss, Gerfried Fuchs, Martin
Neitzel, Martin Dietze, Matthias Kilian, Ralf Doeblitz, Sven Guckes and
many more (see the ChangeLog for active contributors). Many thanks to
all of them!
Many thanks to Gerhard H. Wrodnigg who uses a TOFU protection script in
order to keep the responses to his cancel bot reasonably short. The
entire inspiration for this hack came from the "TOFU protection" line
of his script on many usenet postings.
AVAILABILITY
You can get the latest version from
http://www.escape.de/users/tolot/mutt/.
BUGS
There is a problem when mutt gives a PGP verified or even a multipart
message to t-prot: The information where the PGP encrypted/signed data
or even attachments begin and end is plainly embedded in the text, not
really cleanly recognizable for t-prot. The problem should be worked
around by now, please send a bug report if it does not work for you.
REPORTING BUGS
Please note that t-prot development happens on current stable perl
versions only. If you do run t-prot on earlier (or unstable) perl
versions, you might encounter perl compiler bugs (or funny t-prot
behaviour). One solution is to upgrade your perl, another is simply to
write a bug report. If you do not run a current perl version, please
include this information in your bug report.
Please do not report a bug if
* you found it in the TODO file coming with the distribution. We do
know those and try to fix them as soon as possible.
* you have an old t-prot version. If you encounter a problem, first
see if there is a new t-prot version which fixes the issue. If you
upgraded to the latest version and it still occurs, a bug report is
just great.
If you noticed a bug when processing a message and want to provide the
t-prot team with some useful info, please:
* if invoking t-prot by mutt's display_filter facility, just set
display_filter to something like
"tee ~/foobar | t-prot <your options>"
and include ~/foobar in the bug report -- this way we might reproduce
the bug much easier if you are using a different environment than we
do.
* provide information on what command line options you use t-prot
with, what perl version t-prot runs on your system, and what else might
be important to enable us reproducing the bug.
Send your bug report to <t-prot-bugs@tolot.escape.de>. Thank you.
TODO
Fix bugs (see the BUGS section). Beside that, all main features should
be implemented by now. See the TODO file for more information.
SEE ALSO
mutt(1), muttrc(5) and the part about "display_filter", perl(1),
aliases(5),
RFCs 2045-2049, 3676 and 5322,
http://freshmeat.net/articles/t-prot/ (a nice, solid introduction),
http://got.to/quote/ (German language),
http://www.river.com/users/share/etiquette/edit.html (the Learn To Edit
Messages HowTo has found a new home).