Man Linux: Main Page and Category List

NAME

       mailfoot - a full-online-ordered-training simulator for use with dbacl.

SYNOPSIS

       mailfoot command [ command_arguments ]

DESCRIPTION

       mailfoot  automates  the  task   of   testing   email   filtering   and
       classification  programs  such as dbacl(1).  Given a set of categorized
       documents, mailfoot initiates test runs to estimate the  classification
       errors  and  thereby  permit  fine  tuning  of  the  parameters  of the
       classifier.

       Full Online Ordered Training is a learning method for email classifiers
       where  each  incoming  email  is learned as soon as it arrives, thereby
       always  keeping  category  descriptions  up  to  date  for   the   next
       classification.    This   directly  models  the  way  that  some  email
       classifiers are used in practice.

       FOOT’s error rates depend directly on the order  in  which  emails  are
       seen.   A  small  change in ordering, as might happen due to networking
       delays, can  have  an  impact  on  the  number  of  misclassifications.
       Consequently,  mailfoot  does  not  give meaningful results, unless the
       sample emails  are  chosen  carefully.   However,  as  this  method  is
       commonly  used  by  spam filters, it is still worth computing to foster
       comparisons. Other methods  (see  mailcross(1),mailtoe(1))  attempt  to
       capture the behaviour of classification errors in other ways.

       To  improve and stabilize the error rate calculation, mailfoot performs
       the FOOT simulations several times on slightly reordered email streams,
       and  averages  the  results.  The reorderings occur by multiplexing the
       emails from each category mailbox in random order. Thus  if  there  are
       three  categories,  the  first email classified is chosen randomly from
       the front of the sample email streams of each type.  The  second  email
       is also chosen randomly among the three types, from the front of the
        streams  after  the first email was removed. Simulation stops when all
       sample streams are exhausted.

       mailfoot uses the environment variable MAILFOOT_FILTER when  executing,
       which  permits  the  simulation  of  arbitrary  filters, provided these
       satisfy the compatibility conditions stated in the ENVIRONMENT  section
       below.

       For   convenience,  mailfoot  implements  a  testsuite  framework  with
       predefined wrappers for several open source classifiers.  This  permits
       the  direct  comparison  of  dbacl(1) with competing classifiers on the
       same set of email samples. See the USAGE section below.

       During preparation, mailfoot builds a subdirectory named mailfoot.d  in
       the  current  working directory.  All needed calculations are performed
       inside this subdirectory.

EXIT STATUS

       mailfoot returns 0 on success, 1 if a problem occurred.

COMMANDS

       prepare size
              Prepares a subdirectory named mailfoot.d in the current  working
              directory,  and  populates  it  with  empty  subdirectories  for
              exactly size subsets.

       add category [ FILE ]...
              Takes a set of emails from either FILE if specified,  or  STDIN,
              and  associates  them  with  category.   The  ordering of emails
              within FILE is preserved, and subsequent FILEs are  appended  to
              the  first  in  each  category.   This  command  can be repeated
              several times, but should be executed at least once.

       clean  Deletes the directory mailfoot.d and all its contents.

       run    Multiplexes randomly from the email streams added  earlier,  and
              relearns  categories  only  when a misclassification occurs. The
              simulation is repeated size times.

       summarize
              Prints average error rates for the simulations.

       plot [ ps | logscale ]...
              Plots the number  of  errors  over  simulation  time.  The  "ps"
              option,  if present, writes the plot to a postscript file in the
              directory mailfoot/plots, instead of being shown on-screen.  The
              "logscale"  option, if present, causes the plot to be on the log
              scale for both ordinates.

       review truecat predcat
              Scans the last run statistics  and  extracts  all  the  messages
              which  belong  to category truecat but have been classified into
              category predcat.  The extracted  messages  are  copied  to  the
              directory mailfoot.d/review for perusal.

       testsuite list
              Shows  a  list of available filters/wrapper scripts which can be
              selected.

       testsuite select [ FILTER ]...
              Prepares the filter(s) named FILTER to be used  for  simulation.
              The  filter  name is the name of a wrapper script located in the
              directory /usr/share/dbacl/testsuite.  Each filter has  a  rigid
              interface  documented  below, and the act of selecting it copies
              it to the mailfoot.d/filters  directory.  Only  filters  located
              there are used in the simulations.

       testsuite deselect [ FILTER ]...
              Removes    the    named    filter(s)    from    the    directory
              mailfoot.d/filters so that they are not used in the  simulation.

       testsuite run [ plots ]
              Invokes  every selected filter on the datasets added previously,
              and calculates misclassification rates. If the "plots" option is
              present,  each filter simulation is plotted as a postscript file
              in the directory mailfoot.d/plots.

       testsuite status
              Describes the scheduled simulations.

       testsuite summarize
              Shows the cross validation results for all filters.  Only  makes
              sense after the run command.

USAGE

       The  normal  usage pattern is the following: first, you should separate
       your email collection into several categories (manually or  otherwise).
       Each  category  should be associated with one or more folders, but each
       folder should not contain more than  one  category.  Next,  you  should
       decide how many runs to use, say 10.  The more runs you use, the better
       the predicted error rates. However, more runs take more time.  Now  you
       can type

       % mailfoot prepare 10

       Next,  for  every  category,  you must add every folder associated with
       this category. Suppose you have three categories named spam, work,  and
       play,  which  are  associated with the mbox files spam.mbox, work.mbox,
       and play.mbox respectively. You would type

       % mailfoot add spam spam.mbox
       % mailfoot add work work.mbox
       % mailfoot add play play.mbox

       You should aim for a similar number of emails in each category, as  the
       random  multiplexing  will be unbalanced otherwise. The ordering of the
       email messages in each *.mbox  file  is  important,  and  is  preserved
       during each simulation. If you repeatedly add to the same category, the
       later mailboxes will be appended to the first, preserving  the  implied
       ordering.

       You   can  now  perform  as  many  FOOT  simulations  as  desired.  The
       multiplexed emails are  classified  and  learned  one  at  a  time,  by
       executing    the    command   given   in   the   environment   variable
       MAILFOOT_FILTER. If not set, a default value is used.

       % mailfoot run
       % mailfoot summarize

       The testsuite commands are designed to simplify  the  above  steps  and
       allow  comparison  of  a wide range of email classifiers, including but
       not limited  to  dbacl.   Classifiers  are  supported  through  wrapper
       scripts, which are located in the /usr/share/dbacl/testsuite directory.

       The first stage when using the testsuite is deciding which  classifiers
       to compare.  You can view a list of available wrappers by typing:

       % mailfoot testsuite list

       Note  that  the  wrapper  scripts are NOT the actual email classifiers,
       which must be installed separately  by  your  system  administrator  or
       otherwise.   Once this is done, you can select one or more wrappers for
       the simulation by typing, for example:

       % mailfoot testsuite select dbaclA ifile

       If some of the selected classifiers cannot be found on the system, they
       are  not  selected.  Note  also  that some wrappers can have hard-coded
       category  names,  e.g.  if  the   classifier   only   supports   binary
       classification. Heed the warning messages.

       It  remains  only  to  run the simulation. Beware, this can take a long
       time (several hours depending on the classifier).

       % mailfoot testsuite run
       % mailfoot testsuite summarize

       Once you are all done, you can delete the working files, log files etc.
       by typing

       % mailfoot clean

SCRIPT INTERFACE

       mailfoot testsuite takes care of learning and classifying your prepared
       email corpora for each  selected  classifier.  Since  classifiers  have
       widely  varying  interfaces,  this  is  only possible by wrapping those
       interfaces individually into a standard  form  which  can  be  used  by
       mailfoot testsuite.

       Each  wrapper  script  is  a  command  line tool which accepts a single
       command followed by zero or more optional arguments,  in  the  standard
       form:

       wrapper command [argument]...

       Each  wrapper  script  also  makes  use  of  STDIN and STDOUT in a well
       defined way. If no behaviour is described,  then  no  output  or  input
       should be used.  The possible commands are described below:

       filter In this case, a single email is expected on STDIN, and a list of
              category filenames is expected in $2, $3, etc. The script writes
              the category name corresponding to the input email on STDOUT. No
              trailing newline is required or expected.

       learn  In this case, a standard mbox stream is expected on STDIN, while
              a  suitable  category  file name is expected in $2. No output is
              written to STDOUT.

       clean  In this case, a directory is expected in $2, which  is  examined
              for  old  database  information. If any old databases are found,
              they are purged or reset. No output is written to STDOUT.

       describe
              IN this case, a single  line  of  text  is  written  to  STDOUT,
              describing  the  filter’s functionality. The line should be kept
              short to prevent line wrapping on a terminal.

       bootstrap
              In this case, a directory is expected in $2. The wrapper  script
              first checks for the existence of its associated classifier, and
              other prerequisites.  If  the  check  is  successful,  then  the
              wrapper  is  cloned  into  the  supplied  directory.  A courtesy
              notification should be given on STDOUT  to  express  success  or
              failure.   It  is  also  permissible to give longer descriptions
              caveats.

       toe    Used by mailtoe(1).

       foot   In this case, a list of categories is expected in $3,  $4,  etc.
              Every possible category must be listed. Preceding this list, the
              true category is given in $2.

ENVIRONMENT

       Right after loading, mailfoot reads the hidden file .mailfootrc in  the
       $HOME  directory, if it exists, so this would be a good place to define
       custom values for environment variables.

       MAILFOOT_FILTER
              This variable contains a shell command to be executed repeatedly
              during  the  running  stage.  The command should accept an email
              message on STDIN and output a resulting category  name.  On  the
              command  line,  it  should  also  accept first the true category
              name, then a list of all possible category file names.   If  the
              output  category  does  not  match  the  true category, then the
              relevant  categories  are  assumed   to   have   been   silently
              updated/relearned.   If  MAILFOOT_FILTER  is undefined, mailfoot
              uses a default value.

       TEMPDIR
              This directory is exported for the benefit of  wrapper  scripts.
              Scripts which need to create temporary files should place them a
              the location given in TEMPDIR.

NOTES

       The subdirectory mailfoot.d can grow quite large. It  contains  a  full
       copy  of the training corpora, as well as learning files for size times
       all the added categories, and various log files.

       FOOT simulations for dbacl(1) are very, very slow (order n squared) and
       will take all night to perform. This is not easy to improve.

WARNING

       Because  the ordering of emails within the added mailboxes matters, the
       estimated error rates are not well defined or  even  meaningful  in  an
       objective  sense.   However,  if  the sample emails represent an actual
       snapshot of a user’s incoming email, then the error rates are  somewhat
       meaningful.  The  simulations  can  then  be  interpreted  as alternate
       realities where a given classifier would have intercepted the  incoming
       mail.

SOURCE

       The  source code for the latest version of this program is available at
       the following locations:

       http://www.lbreyer.com/gpl.html
       http://dbacl.sourceforge.net

AUTHOR

       Laird A. Breyer <laird@lbreyer.com>

SEE ALSO

       bayesol(1) dbacl(1), mailcross(1), mailinspect(1), mailtoe(1), regex(7)