Man Linux: Main Page and Category List


       MPE -  MultiProcessing Environment


       The  Multi-Processing Environment (MPE) attempts to provide programmers
       with a complete suite of  performance  analysis  tools  for  their  MPI
       programs  based on post processing approach.  These tools include a set
       of profiling libraries, a  set  of  utility  programs,  and  a  set  of
       graphical visualization tools.

       The  first  set of tools to be used with user MPI programs is profiling
       libraries which provide a collection of routines that create log files.
       These  log  files can be created manually by inserting MPE calls in the
       MPI program, or automatically  by  linking  with  the  appropriate  MPE
       libraries,  or  by combining the above two methods.  Currently, the MPE
       offers the following 3 profiling libraries.

       Tracing Library
              - This traces all MPI calls.  Each MPI call  is  preceded  by  a
              line  that  contains  the  rank in MPI_COMM_WORLD of the calling
              process, and followed by another line indicating that  the  call
              has completed.  Most send and receive routines also indicate the
              values of count, tag, and partner (destination for sends, source
              for receives).  Output is to standard output.

       Animation Library
              -  This  is  a  simple  form  of real-time program animation and
              requires X window routines.

       Logging Library
              - This is the most useful and widely used profiling libraries in
              MPE.   They  form  the basis to generate log files from user MPI
              programs.  There are currently  3  different  log  file  formats
              allowed  in  MPE.   The  default log file format is CLOG.  It is
              basically a collection of events with  single  timestamps.   And
              there  is  ALOG  which  is  provided  for backward compatibility
              reason and it is not being developed.  And the most powerful one
              is  SLOG,  stands  for  Scalable  LOGfile  format,  which can be
              converted  from  CLOG  after  CLOG  file  has   been   generated
              (preferred  approach),  or  can  be  generated directly when MPI
              program is executed (through setting the environmental  variable
              MPE_LOG_FORMAT to SLOG).

              The set of utility programs in MPE includes log format converter
              (e.g.  clog2slog), logfile print (e.g. slog_print)  and  logfile
              viewer  wrappers, logviewer, which selects the correct graphical
              tool  to  display  the  logfile  based  on  the  logfile’s  file

              Currently,  MPE’s  graphical  tools includes 3 display programs,
              upshot for ALOG, jumpshot-2 for CLOG and  jumpshot-3  for  SLOG.
              The  logviewer  script  eliminates the need for user to remember
              the relationship of logfile formats to display programs.


       MPE can be configured  and  installed  as  an  extension  to  most  MPI
       standard compliant MPI implementations, e.g. MPICH, LAM, SGI’s MPI, HP-
       UX’s MPI and IBM’s MPI.  It has been integrated seamlessly  into  MPICH
       distribution,  so  MPE  will  be installed automatically during MPICH’s
       installation process.

       For details of configuring and building of MPE,  please  refer  to  the
       README file in the MPE source directory.


       As  previously  noted,  the  mpe  library  is  composed  of 3 different
       profiling libraries.   Each  MPI  implementation  requires  a  slightly
       different way in which to link with these libraries.  During configure,
       the link path and appropriate libraries are determined and assigned  to
       variables.   These  variables  are first substituted in the Makefile in
       the directory mpe/contrib/test .  The Makefile is then  installed  into
       directory  share/examples  during  the final installation process.  The
       following is a list of these variables.

       LOG_LIBS   =  link path needed to link with the logging library

       TRACE_LIBS =  link path needed to link with the tracing library

       ANIM_LIBS  =  link path needed to link with the animation library

       The variable FLIB_PATH is the link path  needed  to  link  fortran  MPI
       programs with the logging library.

       During  make,  small  test programs cpi.c and fpi.f will be linked with
       each of the above libraries.  In the output from Make, a  message  will
       be  written  regarding  the  success  of each attempted link test.  The
       success of these linkage tests  will  also  be  included  in  the  Make
       output.   If  the linkage tests are successful, then these library link
       paths can be used for your programs as well.


       In addition to using the predefined MPE logging libraries  to  log  all
       MPI calls, MPE logging calls can be inserted into user’s MPI program to
       define and log states.  These states are  called  User-Defined  states.
       States  may be nested, allowing one to define a state describing a user
       routine that contains several MPI calls, and  display  both  the  user-
       defined state and the MPI operations contained within it.

       The  routine  MPE_Log_get_event_number()  has  to be used to get unique
       event numbers (this is important if you are writing a library that uses
       the   MPE   logging  routines)  from  the  MPE  system.   The  routines
       MPE_Describe_state() and MPE_Log_event()  are  then  used  to  describe
       user-defined states.

       int eventID_begin, eventID_end;
       eventID_begin = MPE_Log_get_event_number();
       eventID_end   = MPE_Log_get_event_number();
       MPE_Describe_state( eventID_begin, eventID_end, "Amult", "bluegreen" );
       MyAmult( Matrix m, Vector v )
       /* Log the start event along with the size of the matrix */
       MPE_Log_event( eventID_begin, m->n, (char *)0 );
       Amult code, including MPI calls ...
       MPE_Log_event( eventID_end, 0, (char *)0 );

       The  log  file generated by this code will have the MPI routines within
       the routine MyAmult() indicated by a containing bluegreen rectangle.

       If the MPE logging library, liblmpe.a, are NOT  linked  with  the  user
       program, MPE_Init_log() and MPE_Finish_log() need to be used before and
       after all the MPE calls.    Sample  programs  cpilog.c  and  fpi.f  are
       available  in  MPE  source  directory  contrib/test  or  the  installed
       directory share/examples to illustrate the use of these MPE routines.


       There are 2 environmental variables, TMPDIR and MPE_LOG_FORMAT  ,  that


              -  determines  the  format  of  the  logfile  generated from the
              execution of application linked with MPE logging libraries.  The
              allowed  value  for MPE_LOG_FORMAT are CLOG, SLOG and ALOG. When
              MPE_LOG_FORMAT is NOT set, CLOG is assumed.

       TMPDIR - specifies a directory to be used as temporary storage for each
              process.  By default, when TMPDIR is NOT set, /tmp will be used.
              When user needs to generate  a  very  large  logfile  for  long-
              running  MPI  job,  user  needs  to make sure that TMPDIR is big
              enough to hold the temporary logfile which will  be  deleted  if
              the  merged  logfile  can  be created successfully.  In order to
              minimize the overhead of the logging to the MPI program,  it  is
              highly  recommended user to use a *local* file system for TMPDIR

              Note : The final merged logfile will be written back to the file
              system where process 0 is.


       In   bin/,   user   can  find  several  useful  utility  programs  when
       manipulating logfiles.  These  includes  log  format  converters,  e.g.
       clog2slog,  log format print programs, e.g. slog_print, and a script to
       launch display program, logviewer.


              - a  CLOG  to  SLOG  logfile  converter.   Since  the  automatic
              generation   of  SLOG  file  through  setting  of  environmental
              variable MPE_LOG_FORMAT to SLOG may NOT work for some non  well-
              behaved  MPI  programs,  using  the logfile format converter can
              generate extra diagnostic information about the condition of the
              logfile.   Also  the  converter  allows  one  to  adjust certain
              parameters of the logfile, like frame size which is the  segment
              of the logfile to be displayed by Jumpshot-3’s time line window.
              For non well behaved MPI program, one may need to  increase  the
              frame  size  from  the default 64KB to a bigger value.  For more
              information about the converter, do "clog2slog -h"

              - a CLOG to ALOG logfile converter.  It is not being  developed.
              It is provided here for backward compatibility purpose.


              -  a stdout print program for SLOG file.  It serves to check the
              content of the logfile.  If the SLOG file is too big, it may NOT
              be  useful  to use slog_print.  Also, when slog is NOT complete,
              slog_print won’t work.  So it serves as a simple test  to  check
              if the SLOG file is generated completely.

              - stdout print program for CLOG file.


              - the script which involves appropriate viewer based on the file
              extension of logfile.  For instance, if the logfile is foo.slog,
              logviewer   will  invoke  jumpshot-3  to  display  the  logfile.
              Jumpshot-3  resides  in  share/.   For   more   information   of
              logviewer, do "logviewer -help" to list all available options.


       To view a list of configure options, type ./configure--help in the top-
       level mpe directory.

       In the Installation Manual, there is  a  section  on  MPE  which  gives
       instructions and examples for each type of installation.

       The README in the top-level mpe directory.

       In  the Users Guide, there is additional information describing the MPE
       routines and the profiling libraries.

       The man pages included in the distribution  has  information  regarding
       the  MPE  routines.  The command mpich/bin/mpiman is a script that runs
       xman on these man pages.