Man Linux: Main Page and Category List

NAME

       harvestman - multithreaded desktop webcrawler written in Python

SYNOPSIS

       harvestman [options] [-C configfile]

DESCRIPTION

       HarvestMan  is  a  desktop  WebCrawler written completely in the python
       programming language. It allows you to download a  whole  website  from
       the Internet and mirror it to the disk for browsing offline. HarvestMan
       has many customizable options for the end-user.   HarvestMan  works  by
       scanning  a  web page for links that point to other web pages or files.
       It downloads  the  files  and  copies  them  to  the  disk.  HarvestMan
       maintains the directory structure of the remote website when it mirrors
       the website  to  the  disk.  Every  html  file  is  scanned  like  this
       recursively, till the whole website is downloaded.

       Once  the  download is complete, the links in downloaded html files are
       localized to point to the files on the disk. This makes sure that  when
       the  user  browses the downloaded pages, he does not need to connect to
       the Internet again. If any file  failed  to  get  downloaded  for  some
       reason,  HarvestMan will convert its relative Internet address to point
       to the complete Internet address, so that the user will be connected to
       the  Internet  when he clicks on the link, and does not get a dead-link
       error. (404 error).

       From version 1.2, HarvestMan uses two family of threads, the "Fetchers"
       and the "Getters", for downloading. The Fetchers are threads which have
       the responsibility of crawling  webpages  and  finding  links  and  the
       Getters are threads which download those links (the non-html files).

       HarvestMan,  as  of  latest version is a console application. It can be
       launched by running the HarvestMan script (HarvestMan.py)  if  you  are
       using  the  source code, or the HarvestMan executable, if you are using
       the executable (available on Win32 platforms).It  prints  informational
       messages to the console while it is working. These messages can be used
       to debug the program and locate any errors.

       HarvestMan works by reading its options either from the command line or
       from a configuration file. The configuration file is named "config.xml"
       by default.

       The is  a  major  change  from  HarvestMan  1.5  onwards  is  that  the
       configuration  is  now in an XML file called "config.xml". You can also
       use the convertconfig.py script, present in HarvestMan/tools/  of  your
       installation  to  convert  your configuration from text to XML and vice
       versa. For full details, see the Changes.txt file and see  the  website
       at http://harvestmanontheweb.com

       HarvestMan  writes  a  binary  project  file  using  the  python pickle
       protocol.  This  project  file  is  saved  under  the  HarvestMan  base
       directory with the extension .hbp. This is a complete record of all the
       settings which were used to start HarvestMan and can be read back later
       using the -- projectfile option to restart a HarvestMan project.

MODES OF OPERATION

       HarvestMan   has   two  major  modes  of  operation.  One  is  a  fully
       multithreaded mode, also called as a fast mode.

       Fast Mode
              Fast Mode is the most useful mode of HarvestMan. In  this  mode,
              HarvestMan  launches  multiple  threads  for  each url link, and
              stores them in an internal queue. Also, HarvestMan will launch a
              separate  download  thread  for  each non-html file encountered.
              This process is very fast and you  can  download  websites  very
              quickly  using this mode as multiple downloads occur at the same
              time.

              This mode is the default. You can use this mode if  you  have  a
              relatively  large  bandwidth,  and  a reliable connection to the
              Internet.

              Since HarvestMan is network-bound, using multiple threads speeds
              up the download.

       Slow Mode
              In  the  Slow  Mode,  download  of  websites  happen in a single
              thread, the main program thread.  Each  download  will  have  to
              wait  for  the  previous  one  to  get  completed,  so this is a
              relatively slow process. You can use this mode, if you  have  an
              unreliable  Internet connection or a relatively small bandwidth,
              which does not support opening of multiple sockets at  the  same
              time.

              This  mode  is disabled by default. You can enable it by setting
              the  variable  FASTMODE  in  the  configuration  file  to  zero.
              (Described somewhere in this document)

              If  you  see  a  lot  of  "Socket" type errors when you launch a
              HarvestMan project by using the default mode (fastmode),  switch
              to  this  mode.  This  would  give you a very reliable download,
              though a slow one.

USAGE

       As said earlier, HarvestMan reads its options from a configuration file
       or  from  the  command line. The configuration file by default is named
       "config.xml". You can pass  another  configuration  file  name  to  the
       program by using the command line options -configfile/-C.

       HarvestMan can also read options from the command line.

       From  version  1.1, HarvestMan would also be able to read back previous
       project files by using the command line option -projectfile.

       We will first discuss the structure of the configuration file  and  how
       it  can be used to create a HarvestMan project. For more information on
       the command line arguments, run  the  program  with  the  -help  or  -h
       option.

CONFIGURATION FILE

       The  configuration  file  is a simple text file with many options which
       are a pair of variable/value strings separated by tabs or spaces.  Each
       variable/value  pair  appears  in  a  separate line. Comments can be by
       adding the hash character ’#’, before any line.

       HarvestMan has three basic options and some 50 advanced options.

BASIC OPTIONS

       HarvestMan needs three basic configuration options to work.  These  are
       described below:

       project.name:  This  is  the  project  name  of  the  current download.
       HarvestMan creates a directory of  this  name  in  its  base  directory
       (described  below) where it keeps all the downloaded files. The project
       name needs to be a non-empty string. (Spaces are allowed.)

       project.url: This is the starting url for the  program  from  where  it
       starts  download.  HarvestMan supports the WWW/HTTP/HTTPS/FTP protocols
       in this url. If a url does not begin with any  of  these,  it  will  be
       considered   as   an  HTTP  url.  For  example,  http://www.python.org,
       www.yahoo.com, cnn.com

       project.basedir: This is the base directory for the  program  where  it
       creates  the  project  directories  and stores all downloaded files. If
       this directory does not exist, HarvestMan will attempt to create it.

ADVANCED OPTIONS

       For precisely configuring your download, HarvestMan supports  about  30
       advanced  options. You will need to use many of them, if you would like
       to control your download  exactly  the  way  you  want.  The  following
       section describes each of these settings and what they do. Read on.

       The Fetchlevel setting
              From Version 1.2, there is a change in this setting. Read on.

              This  is one of the most useful options to tweak in a HarvestMan
              project.   The   option   is   controlled   by   the    variable
              download.fetchlevel in the configuration file.

              Make sure you read the following documentation very carefully.

              When  you are downloading files from a website, you would prefer
              to limit your download to certain areas  of  the  Internet.  For
              example, you might want to download all links pointed by the url
              http://www.foo.com/bar (a hypothetical example), that come under
              the  www.foo.com  web  server. Or you might want to download all
              links under the directory path  http://www.foo.com/pics  and  no
              more. You can use this option to do exactly that.

              The  option download.fetchlevel has 5 possible values that range
              from 0 - 4.

              A value of 0 limits the download to a directory path from  where
              you  start  your download. For example, if your starting url was
              http://www.foo.com/bar/index.html, this option makes  sure  that
              all links downloaded will be belonging to the directory url path
              http://www.foo.com/bar and below it. Any web links  pointing  to
              directories outside or other web servers would be ignored.

              A  value  of  1  limits the download to the starting server, but
              does not limit it to paths below the starting directory.

              For     example,     if     your      starting      url      was
              http://www.foo.com/bar/index.html,   this   option   would  also
              download  files  from  the   http://www.foo/com/other/index.html
              page, since it belongs to the starting server.

              A  value  of  2  performs the next level fetching. It allows all
              paths in the starting server, and also all urls external to  the
              starting  server, but linked directly from pages in the starting
              server.    For     example,     if     your     starting     url
              http://www.foo.com/bar/index.html    contained    a    link   to
              http://www.foo2.com/bar2/index.html   (an   external    server),
              HarvestMan  will  try  to  download this link also. But all urls
              linked     linked     from     this     link,      i.e      from
              http://www.foo2.com/bar2/index.html, would be ignored.

              A  value  of  3  performs  a  fetching similar to above, but the
              difference is that it  does  not  get  files  which  are  linked
              outside the directory of the starting url, but gets the external
              links which are linked one level  from  the  starting  url.  For
              example,          if          your          starting         url
              <http://www.foo.com/bar/index.html>   contained   a   link    to
              http://www.foo2.com/bar2/index.html    (an   external   server),
              HarvestMan will try to download this link also. But a  url  like
              <http://www.foo.com/other/index.html>   (a   link   outside  the
              starting url’s directory) will be ignored.

              A value of 4 gives you no control to the  fetching  process.  It
              will  allow  all web pages to be downloaded, including web pages
              linked from external server links, encountered in  the  starting
              url’s  page.  Setting  this  option  will  mostly  result in the
              crawler trying to crawl the entire Internet, assuming that  your
              starting  url  has  links  to  other  outside  servers. Set this
              option, only if you are very sure of what  you  are  doing.  Any
              value above 4 has no special meaning, and would behave just like
              above.

              For most downloads, this value can be specified between 0 and 2.

       The Depth Setting

              This  is  another  setting  that  gives  you  control  over your
              download.  It is denoted by the variable  control.depth  in  the
              configuration file.

              This  value  specifies the distance of any url from the starting
              url’s directory in terms of the directory path offset.  This  is
              applicable  only  to  the  directories  (links)  in the starting
              server, below the starting url’s directory. The default value is
              10.

              If  a  directory  is found whose offset is more than this value,
              any links under it will not be downloaded.

              You can specify zero depths in which case the download  will  be
              limited to files just below the directory of the starting url.

              Examples:        If        the       starting       url       is
              http://www.foo.com/bar/foo.html,       then       the        url
              http://www.foo.com/bar/images/graphics/flowers/flower.jpg  has a
              depth of 3 relative to the starting url.

       The External Depth Setting

              This option also helps you to control downloads. It  is  denoted
              by variable control.extdepth in the configuration file.

              This  value specifies the distance of a url from its base server
              directory. This is applicable to urls which belong  to  external
              servers and to urls outside the directory of the starting url.

              If a directory is found whose distance from the base server path
              is more than this value, any files under it will be ignored.

              Note that this option does not support the notion of zero depth.
              A valid value for this has to be greater than or equal to one.

              Examples:  The  url  http://www.foo.com/bar/images.html  has  an
              external depth of 1  relative  to  the  base  server  directory,
              http://www.foo.com.

       The External Servers Setting

              This  option tells the program whether to follow links belonging
              to  outside  web  servers.  This  is  denoted  by  the  variable
              control.extserverlinks. By default, the program ignores external
              server links.

              The option has  lesser  precedence  to  the  download.fetchlevel
              setting.   If  download.fetchlevel  is  set  to  a value of 2 or
              above, this setting is conveniently ignored.

       The External Directories Setting

              This  option  tells  the  program  whether  to  download   files
              belonging  to  outside  directories ,i.e directories external to
              the directory of the starting url. This is denoted by the option
              control.extpagelinks in the configuration file.

              This  option tells the program whether to follow links belonging
              to outside directories.

              The  default  value  is  1  (Enabled).  The  download.fetchlevel
              setting  has  precedence over this value. If download.fetchlevel
              is set to a value of 1 or more,  this  setting  is  conveniently
              ignored.

       The Images Setting

              Specifies  the  program  whether  to  download  images linked to
              pages.  Enabled by  default.  This  option  is  denoted  by  the
              variable download.images in the configuration file.

       The Html Setting

              Tells  the  program  whether  to download html files. Enabled by
              default. Denoted by the variable download.html.

       Maximum limit of External Servers

              You can put a check on the number of external servers from which
              you  want  to  download  files from, by setting this option to a
              non-zero value. It takes precedence to  the  download.fetchlevel
              setting.   This   option   is   controlled   by   the   variable
              control.maxextservers in the  configuration  file.  The  default
              value is zero which means that this option is ignored.

              To enable this option, set it to a value greater than zero.

       Maximum limit on External Directories

              You  can  put a check on the number of external directories from
              which you want to download files from, by setting this option to
              a    non-zero    value.    It    takes   precedence   over   the
              download.fetchlevel setting. This option is  controlled  by  the
              variable control.maxextdirs, in the configuration file.

              The  default  value  is  zero  which  means  that this option is
              ignored.

              To enable this option, set it to a value greater than zero.

       Maximum limit on Number of Files

              You can precisely control the number of total files you want  to
              download  by setting this option. It is denoted by the variable,
              control.maxfiles. The default value is 3000.

       Default download of images

              This option tells the program to always fetch images linked from
              pages,    though   they   might   be   belonging   to   external
              servers/directories or might be violating the depth rules.

              This      option      takes      precedence       over       the
              control.extpagelinks/control.extserverlinks   settings  and  the
              control.depth/control.extdepth settings.

              The download.image setting has a  higher  precedence  than  this
              setting.

              This  option  is  enabled  by  default.  Denoted by the variable
              download.linkedimages.

       Default download of style sheets (.css files)

              Same as the above option, but only that this options checks  for
              stylesheet   (css)   links.  This  has  higher  precedence  over
              control.extpagelinks/control.extserverlinks       and        the
              control.depth/control.extdepth settings. Enabled by default.

              This      option      is     denoted     by     the     variable
              download.linkedstylesheets.

       Maximum thread setting

              This options  sets  the  number  of  separate  threads(trackers)
              launched  by  the  program  at  a  time. This is not an accurate
              setting. Note that a given time does not  really  mean  that  so
              many  connections  are  running  per  second  but only tells the
              program that it cannot launch threads above this limit.

              This option makes sense only in  multithreaded  downloads,  i.e,
              only  when the program is running in fastmode. In slowmode, this
              setting has no effect.

              Denoted by the variable system.maxtrackers. The default value is
              10.

       Separate threads for file download

              This  option  controls  the  ,multithreaded download of non-html
              files in the fastmode. In fastmode,  separate  download  threads
              are  launched  to  retrieve  non-html files. If you disable this
              option, these files will be downloaded in the main thread of the
              downloader thread.

              By  default,  this  option  is  enabled. You can tweak it by the
              variable system.usethreads.

       Mode Selection

              As  described  in  the  beginning,  there  are  two  modes   for
              HarvestMan,  the  fast  one and the slow one. This option allows
              you to choose your mode of operation.

              The variable for this option  is  system.fastmode.  The  default
              value  is  1,  which  means  that  the program uses fastmode. To
              disable fastmode, and switch to slowmode, set this  variable  to
              zero.

       Size of the thread pool

              This value controls the size of the thread pool used to download
              non-html  files  when  the  program   runs   in   fastmode   and
              system.usethreads is enabled. The default value is 10.

              This option is controlled by the variable system.threadpoolsize.
              It makes sense only if the program is running  in  fastmode  and
              the system.usethreads option is enabled.

       Timeout value for a thread

              This  specifies  the timeout value for a single download thread.
              The default value is 200 seconds.  Threads  which  overrun  this
              value are eventually killed and cleaned up.

              This  option is controlled by the variable system.threadtimeout.

              This value is ignored  when  you  are  running  the  program  in
              slowmode, without using multiple threads.

       Robot Exclusion Protocol

              The  Robot  Exclusion  Principle  control  flag.  This tells the
              spider whether to follow rules specified by the robots.txt  file
              on some web servers. Enabled by default.

              We advice you to always enabled this option, since it shows good
              Internet etiquette and respect for the download rules laid  down
              by  webmasters of sites. Disable it after reading any legalities
              laid down by the website, according to your discretion.  We  are
              not  responsible  for  any  eventuality  that arises from a user
              violating these rules. (See LICENSE.txt file.)

              The variable for this value is control.robots.

       Proxy Server Support

              HarvestMan is written taking into account corporate users  (like
              the    authors!)   who   connect   to   Internet   from   behind
              firewalls/proxies. Such users should set this option to  the  IP
              address/name  of their proxy server with the proxy port appended
              to it.

              The  variables  for  this  option  are  network.proxyserver  and
              network.proxyport.  Set  the first one to the ip address/name of
              your proxy server and the second one to its port number.

              Default values: proxy and 80.

              Note: If you are  creating  the  configuration  file  using  the
              script  provided for that purpose, the proxy server string would
              be  encrypted  and  does  not  appear  in  plain  text  in   the
              configuration file.

       Proxy Authentication Support

              HarvestMan    also    supports   proxies   that   require   user
              authentication.

              The   variables   for    this    are    network.proxyuser    and
              network.proxypasswd.

              Note:  If  you  are  creating  the  configuration file using the
              script  provided  for  that  purpose,  these  values  would   be
              encrypted and does not appear in plain text in the configuration
              file.

       Intranet Crawling

              This  option  is  disabled  from  version  1.3.9  onwards  since
              HarvestMan  can  now  intelligently figure out whether url is in
              the intranet or internet by trying to resolve the host  name  in
              the url. Hence the option is not required anymore.

              From  version 1.3.9, we can mix urls in the internet/intranet in
              the same project.

       Renaming of Dynamically Generated Files

              Dynamically generated files (images/html) will usually have file
              extensions  that bear no connection to their actual content. You
              will not be able to open these files  correctly,  especially  on
              the  Windows platform which depends on file extensions to launch
              applications. This option will tell HarvestMan to try to  rename
              these  files  by  looking at their content. HarvestMan will also
              appropriately rename any link which points to these files.

              This option right now works well only  for  gif/jpeg/bmp  files.
              Disabled by default.

              The variable for this option is download.rename.

       Console Message Settings

              HarvestMan  prints  out  a  lot of informational messages to the
              console while it is running. These  can  be  controlled  by  the
              project.verbosity variable in the configuration file. This value
              ranges from 0 to 5.

              The default value is 2.

              Here is each value and a  description  of  its  meaning  to  the
              program.

              0:    Minimal    messages,    displays    only    the    Program
              Information/Copyright.

              1: Basic messaging, displays  above,  plus  information  on  the
              current project including the statistics.

              2:  More messaging, displays above, plus information on each url
              as it is being downloaded.

              3: Extended messaging, displays above, plus information on  each
              thread  that is downloading a certain file. Also displays thread
              killing/joining  information  and   directory   creation,   file
              saving/deletion information.

              4:  Debug  messaging, displays above, plus debugging information
              for the programmer. Not recommended for the end-user.

              5:  Extended  debug  messaging,   displays   maximal   messages,
              including  the  debug information from the web page parser. (Use
              this at your own risk!)

              Please note that these guidelines are flexible and can change as
              new  versions  are  being  developed, especially the behavior of
              values from 3 - 5.

       Filters

              HarvestMan allows  the  user  to  refine  downloads  further  by
              specifying filtering options for urls. These are of two kinds:

              1.  Filters for urls (plain vanilla links), which are controlled
              by the control.urlfilter variable.

              2. Filters for external servers, which  are  controlled  by  the
              control.serverfilter variable.

              The  filter  strings  are a kind of regular expression. They are
              internally  converted  to  python  regular  expressions  by  the
              program.

              Writing filter regular expressions

              a. URL Filters (for the control.urlfilter setting)

              URL filters supported by HarvestMan are of 3 types. These are:

              1.   Filename  extensions  2.  Servers/urls  3.  Servers/urls  +
              filename extensions

              An example of the first type is *.gif

              Examples of the second type are,:  www.yahoo.com,  */advocacy/*,
              */images/sex/*, */avoid.gif, ad.doubleclick.net/*

              Examples    of    the    third    type    are,:   /images/*.gif,
              ad.doubleclick.net/images/*.jpg, yimg.yahoo.com/*.gif

              You can build a ’no-pass’ (block) filter by prepending a regular
              expression as described above with a ’-’ (minus) sign. (Example:
              -*.gif).

              You can build a ’go-through’  (allow)  filter  by  prepending  a
              regular  expression  as  described above with a ’+’ (plus) sign.
              (Example: +*.gif).

              You can concatenate regular expressions of the block/allow  kind
              and create custom url filters.

              Example:  (Block all jpeg images, as well as all urls containing
              "/images/"  in  their  path,   but   always   allow   the   path
              "’/preferred/images/"):

              -*.jpg+*/preferred/images/*-*/images/*

              Example:    (Block    all    gif    files    from   the   server
              "toomanygifs.com"):

              -toomanygifs.com/*.gif

              Example: (Block all files  with  the  name  "bad.jpg"  from  all
              servers.)

              -*/bad.jpg

              Example:  (Block  all jpeg/gif/png/ images but allow pdf/doc/xls
              files.):

              -*.jpg-*.jpeg-*.gif-*.png+*.pdf+*.doc+*.xls

              If there is a collision between  the  results  of  an  inclusion
              filter  and an exclusion filter, the program gives precedence to
              the decision of the filter  which  comes  first  in  the  filter
              expression. If there is still ambiguity, the inclusion filter is
              given precedence.

              b. Server filters (for the control.serverfilter setting)

              If you are enabling fetching links from  external  servers,  you
              can  write a server filter in a similar way to url filters. This
              also allows you to write no-pass  and  go-through  filters.  The
              main  difference  is  that  in  urlfilters, the character "*" is
              ignored, whereas in server filters, this matches  any  character
              or sequence of characters.

              Example:   Block   all   files  from  the  server  adserver.com:
              -adserver.com/*

              Example: Block all files from the server niceimages.com  in  the
              path /advertising/, but allow all other paths.

              -*niceimages.com/*/advertising/*

              Note  that  the  control.serverfilter  if  specified, is checked
              before    control.urlfilter.    So    any    result    of    the
              control.serverfilter setting takes precedence.

       Retrieval of failed links

              Tells the program whether to try refetching links that failed to
              retrieve at the end. Retry will be attempted by  the  number  of
              times specified by this variable’s value.

              Retry  will  be  attempted  after a gap of 0.5 seconds after the
              first attempt for every url  that  failed  due  to  a  non-fatal
              error.  Also  retry  will be attempted for all failed links once
              again at the end of the mirroring.

              This option is controlled by the variable  download.retryfailed.
              The  default value is 1. (Retry will be attempted once for every
              failed link, and once again at the end of the download.)

              To disable retry, set this variable to zero.

       Localization of URLs

              Tells the program whether to localize (Internet  links  modified
              to  file  links)  the  links  in all html files downloaded. This
              helps user to browse the website as if it were local. HarvestMan
              also  converts  any relative url links to absolute url links, if
              their files were not downloaded.

              This is enabled by default. It is a good idea to  always  enable
              it.

              Note  that  localization  of  links  is  done  at the end of the
              download.

              Controlled by the variable indexer.localise.

              From version 1.1.2, this option supports 3 values.  A  value  of
              zero   of  course  disables  it.  A  value  of  1  will  perform
              localization by replacing url  links  with  absolute  file  path
              names.

              A  value  of  2 will perform localization by replacing url links
              with relative file path names. Relative localization  helps  you
              to  browse  the  downloaded  website from different file systems
              since the  url  paths  are  relative  (to  directory).  Absolute
              localization  locks your downloaded website to the filesystem of
              the machine where you ran HarvestMan. From  version  1.1.2,  the
              default  value  of  this option is 2, i.e it performs a relative
              localization by default.

              Another variable related to localization has been added  in  the
              1.1.2  release.  This  allows  you to perform JIT (Just In Time)
              localization of html files,  i.e,  immediately  after  they  are
              downloaded, instead of at the end of download.

              This option is described somewhere below.

       URL List File

              You can tell HarvestMan to dump a list of crawled urls to a file
              by   setting   this   option.   The   variable   for   this   is
              files.urlslistfile and is disabled by default.

       Error log file

              A   file   to   write  error  logs  into.  This  by  default  is
              ’errors.log’.   This  file  will  be  created  in  the   project
              directory of the current project.

              Variable: files.errorfile

              Note:  From version 1.2, this feature is disabled. Don’t use it.

       Message Log File

              From version 1.4 (this version), the message log file  is  named
              <project>.log  for  a  project  ’project’  and  is automatically
              created in the project directory of the project. This is  not  a
              configurable option anymore.

       Browse Index Page

              HarvestMan  creates  an html project browser page in the Project
              Directory and appends the starting (index) files of each project
              to  this  page,  at  the end of each project. This option can be
              enabled or disabled by setting the  variable  display.browsepage
              By default, this is enabled.

       JIT Localization

              HarvestMan,  from  version 1.1.2, has an option to localize HTML
              files immediately after they are downloaded, instead of  at  the
              end  of  the  project. This option can be enabled by setting the
              variable, indexer.jitlocalise, to a value greater than zero.

              By default this is disabled.

              Note: From version 1.2, this option is disabled. Don’t use it.

       File Integrity Verification

              HarvestMan  verifies  the  integrity  of  downloaded  files   by
              performing  an  md5 check summation check. From version 1.4 this
              option is disabled and is not  available  in  the  configuration
              file.

       Cookie Support

              From version 1.2, we have added support for Cookies. The support
              is basic based on RFC 2109. By default cookies in web pages  are
              saved  in  a  cookie  file inside the project directory and read
              back  for  pages  which  require  these  cookies.  This  can  be
              controlled  by  the variable download.cookies. The default value
              is 1.

              For disabling cookies, set this variable to zero (0).

       Files Caching

              From version 1.2, we support caching/update of downloaded files.
              An  binary  cache  file  is created for every project. This file
              contains an md5 checksum of the file, its location on  the  disk
              and  the url from which it was downloaded. Next time the project
              is re-started, the program checks the urls  against  this  cache
              file.  The  files  are downloaded only if their checksum differs
              from the  checksum  of  the  cached  file,  otherwise  they  are
              ignored.

              This  option  is  enabled  by  default.  It is controlled by the
              variable  control.pagecache.  To  disable  caching,   set   this
              variable to zero (0).

              From   version  1.4,  a  sub-opton  named  control.datacache  is
              available.  If set to 1(default), data of each url is also saved
              in  the  cache file. So if you lose your original files, but the
              cach is present,  HarvestMan  can  recreate  the  files  of  the
              project  from the cache, if the cache files are not out of date.

              You can enable data caching for small projects where the  number
              of files downloaded are not too much. If the project downloads a
              lot of files, say > 5000, you might disable data caching.

       Number of Simultaneous Network Connections

              From version 1.2, the number of simultaneous network connections
              can be controlled by modifying a config variable.

              For  all  1.0  (major)  versions  and  the  1.2  alpha  version,
              HarvestMan had a global download lock that denied more than  one
              network   connection  at  a  given  instant.  This  slowed  down
              downloads considerably.

              From  1.2  onwards,   many   simultaneous   downloads   (network
              connections)  are  possible  apart  from  multiple  threads. The
              number of simultaneous connections by default is 5. The user can
              change this by modifying the variable control.connections in the
              config file. If set to a higher value, the many download threads
              can  use  more  connections  at  a given instant and download is
              faster. If set to a lower value, the threads will have  to  wait
              for  a  free connection slot, if the number of connections reach
              the limit. You can set it to reasonable value depending on  your
              network  bandwidth.  A  value  below  10  is  desirable for low-
              bandwidth  connections   and   above   10   for   high-bandwidth
              connections.  If you have a broadband or DSL connection allowing
              very high speeds, set this to a relatively large value like  20.

              It  the  number of connections is much less when compared to the
              number of url trackers, downloads will suffer. It is a good idea
              to keep these two values approximately the same.

       Project Timeout

              From  version  1.2  onwards, HarvestMan allows for a way to exit
              projects which hang due to some network or  system  problems  in
              threading.  The program monitors reads/writes from the url queue
              and keeps a time difference  value  between  now  and  the  last
              read/write  operation  on  the  queue. If no threads are writing
              to/reading from the queue, the program  exits  automatically  if
              this time difference exceeds a certain timeout value. This value
              can be controlled by the  variable  control.projtimeout  in  the
              config file. Its value by default is 5 minutes (300 seconds).

       Javascript retrieval

              From  version  1.2, HarvestMan can fetch javascript source files
              (.js files) from webpages.  This  has  been  done  by  using  an
              enhanced HTML parser that can download javascript files and java
              applets.

              The variable for this is  download.javascript.  This  option  is
              enabled by default.

              For skipping javascript files, set this option to zero(0).

       Java applets retrieval

              From  version  1.2,  HarvestMan  can  fetch  java applets(.class
              files) from webpages. This has been done by  using  an  enhanced
              HTML parser that can download javascript files and java applets.

              The variable for this is  download.javaapplet.  This  option  is
              enabled by default.

              For skipping java applet files, set this option to zero(0).

       Keyword(s) Search ( Word Filtering )

              This  is  a new feature from the 1.3 release. HarvestMan accepts
              complex boolean regular expressions for word matches inside  web
              pages. HarvestMan will download only those pages which match the
              word regular expressions.

              For example, to download  only  those  webpages  containing  the
              words,  HarvestMan and Crawler, you create the following regular
              expression and pass it as the config option  control.wordfilter.

              control.wordfilter           (HarvestMan & Crawler)

              Only  the  webpages  which  contain  both  these  words  will be
              spidered and downloaded. Note that the filter is not applied  to
              the starting page.

              This  feature  is  based  on  an  ASPN  recipe  by  Anand Pillai
              available               at                the                URL
              http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252526

       Subdomain Setting

              New feature from 1.3.1 release. HarvestMan allows you to control
              whether subdomains in a domain are treated as  external  servers
              or  not, using the variable control.subdomain. If this is set to
              1, then subdomains will be considered as external servers.

              If set to 0, which is the default, subdomains in a  domain  will
              not be considered as external servers.

              For  example,  if  the  starting server is http://www.yahoo.com,
              then if this variable  is  disabled  (set  to  zero),  then  the
              domain,  http://in.yahoo.com  will be considered as part of this
              domain and not as an external server.

       Skipping query forms

              To skip server side or cgi query forms, set this variable to  1.
              The  variable  is  named  control.skipqueryforms and is set to 1
              (enabled) by default.

              This        skips        links        of        the         form
              http://server.com/formquery?key=value

              To download these links set the variable to 0.

       Controlling number of requests per server

              This  is  a  new  feature  in version 1.3.2. You can control the
              number of simultaneous requests to the same  server  by  editing
              the config variable named control.requests. This is set to 10 by
              default.

       Html cleaning up (Tidy Interface)

              From version 1.3.9, HarvestMan has an option to  clean  up  html
              pages  before  sending them to the parser. This allows to remove
              errors from web pages so that they are parsed correctly  by  the
              parser.  This in turn helps to download web sites that otherwise
              might not get  downloaded  due  to  the  parser  errors  of  the
              starting html page, for example.

              The  tidylib  source  code  is  included  along  with HarvestMan
              distribution, so you don’t need to install it separately.

              This option is enabled by  default  and  is  controlled  by  the
              variable "control.tidyhtml".

       URL and Website Priorities

              From this version onwards, HarvestMan allows the user to specify
              priorities for urls and servers.

              Every  url  has  a  default  priority,  assigned  based  on  its
              "generation".  The  generation of a url is a number based on the
              level at which the url was generated, based on the starting url.
              The  starting url has a generation 0, all urls generated from it
              have a generation 1, and so on.

              URLs with a lower generation number are given  higher  priiority
              when  compared  to urls with a higher generation. Also html /web
              page urls get a higher priority than  other  urls  in  the  same
              generation.

              User  can  specify  his  priority  for  urls by using the config
              variable named "control.urlpriority". This works on the basis of
              file  extensions,  and  has  a  range  from -5 to 5, -5 denoting
              lowest priority and 5 denoting maximum priority.

              For example, to specify that pdf  files  should  have  a  higher
              priority we can make the following entry in the config file.

              control.urlpriority         pdf+1

              If  you  want  to give word documetns a higher priority than pdf
              files, you can give the following priority specification.

              control.urlpriority         pdf+1,doc+2

              Priroty settings are separated by commas.

              If you want to put gif images at the  lowest  priority  and  jpg
              images at the highest priority,

              control.urlpriority         gif-5, jpg+5

              Similar  synatx  can  be used for setting server priorities. The
              variable named control.serverpriority can  be  used  to  control
              this.

              Assume   that  you  want  to  download  files  from  the  server
              http://yahoo.com with a higher priority  when  compared  to  the
              server http://www.cnn.com, in the same download project.

              control.serverpriority           yahoo.com+1, cnn.com-1

              There can be other combinations also.

              A  priority which is lesser than -5 or greater than 5 is ignored
              by the config parser.

       Time Limits

              From version 1.4, a project can specify a time limit in which to
              complete  downloads.  When this time limit is reached HarvestMan
              automatically terrminates the project by stopping  all  download
              threads and cleaning up.

              This   option   can   be   specified   by   using  the  variable
              control.timelimit.

       Asynchronous URL Server

              From  1.4  version,  another  way  of  managing   downloads   is
              available.   This  is  an  asynchronous url server, which serves
              urls to the fetcher threads. Crawler threads send  urls  to  the
              server  and  fetcher threas receives them from it. The server is
              based on the asyncore module in Python, hence it offers superior
              performance  and  faster multiplexing of threads than the simple
              Queue. The server uses an internal queue  to  store  urls  which
              also increases performance.

              If  you  enable  the variable network.urlserver you can avail of
              this feature. This option is disabled by default.

              The server listens by default to the port 3081. You  can  change
              it by modifying the variable network.urlport in the config file.

       Locale Settings

              From 1.4 version, you can set a specific locale for  HarvestMan.
              Sometimes when parsing non-English websites, the parser can fail
              to report some pages, because the language is  not  set  to  the
              language  of the webpage. In such cases, you can manually change
              the language and  other  settings  by  changing  the  locale  of
              HarvestMan.

              Locale  can be changed by modifying the variable system.locale .
              This is set to the american locale  by  default  on  non-Windows
              platforms  and to the default locale (’C’) on Windows platforms.

              For example, if you see lot of html parsing errors when browsing
              a  Russian  site,  you  could  try  setting  the  locale  to say
              ’russian’.

       Maximum File Size

              A new option from version 1.4. HarvestMan fixes the maximum size
              of  a  single  file  as 1 MB. A url whose file size is more than
              this will be skipped. This can be  controlled  by  the  variable
              control.maxfilesize.

       URL Tree File

              From  version  1.4,  a  url tree file ,i.e a file displaying the
              relation of parent and child urls in a project can be  saved  at
              the  end  of the project. This file can be saved in two formats,
              in text or html. This option is controlled by the variable named
              files.urltreefile.  The  program figures out which format to use
              by looking at the file name extension.

       Ad Filtering

              A  new  feature  from  version  1.4.  URLs   which   look   like
              adveritsement graphics or banners or pop-ups will be filtered by
              HarvestMan. This works by using regular expressions.  The  logic
              of  this  is  borrowed from the Internet Junkbuster program. The
              option is control.junkfilter.

              This option is enabled by default.

OPTIONS

       -h, --help
              Show help message and exit

       -v, --version
              Print version information and exit

       -p, --project=PROJECT
              Set the (optional) project name to PROJECT.

       -b, --basedir=BASEDIR
              Set the (optional) base directory to BASEDIR.

       -C, --configfile=CFGFILE
              Read all options from the configuration file CFGFILE.

       -P, --projectfile=PROJFILE
              Load the project file PROJFILE.

       -V, --verbosity=LEVEL
              Set the verbosity level to LEVEL. Ranges from 0-5.

       -f, --fetchlevel=LEVEL
              Set the fetch-level of this project to LEVEL. Ranges from 0-4.

       -N, --nocrawl
              Only download the passed url (wget-like behaviour).

       -l, --localize=yes/no
              Localize urls after download.

       -r, --retry=NUM
              Set the number of retry attempts for failed urls to NUM.

       -Y, --proxy=PROXYSERVER
              Enable and set proxy to PROXYSERVER (host:port).

       -U, --proxyuser=USERNAME
              Set username for proxy server to USERNAME.

       -W, --proxypass=PASSWORD
              Set password for proxy server to PASSWORD.

       -n, --connections=NUM
              Limit number of simultaneous network connections to NUM.

       -c, --cache=yes/no
              Enable/disable caching of downloaded files.  If  enabled,  files
              won’t  be  downloaded  unless  their timestamp is newer than the
              cache timestamp.

       -d, --depth=DEPTH
              Set the limit on the depth of urls to DEPTH.

       -w, --workers=NUM
              Enable worker threads and set the number of  worker  threads  to
              NUM.

       -T, --maxthreads=NUM
              Limit the number of tracker threads to NUM.

       -M, --maxfiles=NUM
              Limit the number of files downloaded to NUM.

       -t, --timelimit=TIME
              Run the program for the specified time TIME.

       -s, --urlserver=yes/no
              Enable/disable urlserver running on port 3081.

       -S, --subdomain=yes/no
              Enable/disable  subdomain  setting.  If this is enabled, servers
              with the same base server name such  as  http://img.foo.com  and
              http://pager.foo.com will be considered as distinct servers.

       -R, --robots=yes/no
              Enable/disable Robot Exclusion Protocol.

       -u, --urlfilter=FILTER
              Use regular expression FILTER for filtering urls.

       --urlslist=FILE
              Dump a list of urls to file FILE.

       --urltree=FILE
              Dump a file containing hierarchy of urls to FILE.

FILES

       config.xml

SEE ALSO

       python(1),

AUTHOR

       harvestman  was  written by Anand Pillai <anandpillai@letterboxes.org>.
       For latest info, visit http://harvestmanontheweb.com

       This manual page was written by Kumar  Appaiah  <akumar@ee.iitm.ac.in>,
       for the Debian project (but may be used by others).

                               February  5, 2006