NAME
MPE - MultiProcessing Environment
DESCRIPTION
The Multi-Processing Environment (MPE) attempts to provide programmers
with a complete suite of performance analysis tools for their MPI
programs based on post processing approach. These tools include a set
of profiling libraries, a set of utility programs, and a set of
graphical visualization tools.
The first set of tools to be used with user MPI programs is profiling
libraries which provide a collection of routines that create log files.
These log files can be created manually by inserting MPE calls in the
MPI program, or automatically by linking with the appropriate MPE
libraries, or by combining the above two methods. Currently, the MPE
offers the following 3 profiling libraries.
Tracing Library
- This traces all MPI calls. Each MPI call is preceded by a
line that contains the rank in MPI_COMM_WORLD of the calling
process, and followed by another line indicating that the call
has completed. Most send and receive routines also indicate the
values of count, tag, and partner (destination for sends, source
for receives). Output is to standard output.
Animation Library
- This is a simple form of real-time program animation and
requires X window routines.
Logging Library
- This is the most useful and widely used profiling libraries in
MPE. They form the basis to generate log files from user MPI
programs. There are currently 3 different log file formats
allowed in MPE. The default log file format is CLOG. It is
basically a collection of events with single timestamps. And
there is ALOG which is provided for backward compatibility
reason and it is not being developed. And the most powerful one
is SLOG, stands for Scalable LOGfile format, which can be
converted from CLOG after CLOG file has been generated
(preferred approach), or can be generated directly when MPI
program is executed (through setting the environmental variable
MPE_LOG_FORMAT to SLOG).
The set of utility programs in MPE includes log format converter
(e.g. clog2slog), logfile print (e.g. slog_print) and logfile
viewer wrappers, logviewer, which selects the correct graphical
tool to display the logfile based on the logfile’s file
extension.
Currently, MPE’s graphical tools includes 3 display programs,
upshot for ALOG, jumpshot-2 for CLOG and jumpshot-3 for SLOG.
The logviewer script eliminates the need for user to remember
the relationship of logfile formats to display programs.
CONFIGURATION AND INSTALLATION
MPE can be configured and installed as an extension to most MPI
standard compliant MPI implementations, e.g. MPICH, LAM, SGI’s MPI, HP-
UX’s MPI and IBM’s MPI. It has been integrated seamlessly into MPICH
distribution, so MPE will be installed automatically during MPICH’s
installation process.
For details of configuring and building of MPE, please refer to the
README file in the MPE source directory.
EXAMPLE MPE PROGRAMS
As previously noted, the mpe library is composed of 3 different
profiling libraries. Each MPI implementation requires a slightly
different way in which to link with these libraries. During configure,
the link path and appropriate libraries are determined and assigned to
variables. These variables are first substituted in the Makefile in
the directory mpe/contrib/test . The Makefile is then installed into
directory share/examples during the final installation process. The
following is a list of these variables.
LOG_LIBS = link path needed to link with the logging library
TRACE_LIBS = link path needed to link with the tracing library
ANIM_LIBS = link path needed to link with the animation library
The variable FLIB_PATH is the link path needed to link fortran MPI
programs with the logging library.
During make, small test programs cpi.c and fpi.f will be linked with
each of the above libraries. In the output from Make, a message will
be written regarding the success of each attempted link test. The
success of these linkage tests will also be included in the Make
output. If the linkage tests are successful, then these library link
paths can be used for your programs as well.
CUSTOMIZING LOGFILES
In addition to using the predefined MPE logging libraries to log all
MPI calls, MPE logging calls can be inserted into user’s MPI program to
define and log states. These states are called User-Defined states.
States may be nested, allowing one to define a state describing a user
routine that contains several MPI calls, and display both the user-
defined state and the MPI operations contained within it.
The routine MPE_Log_get_event_number() has to be used to get unique
event numbers (this is important if you are writing a library that uses
the MPE logging routines) from the MPE system. The routines
MPE_Describe_state() and MPE_Log_event() are then used to describe
user-defined states.
int eventID_begin, eventID_end;
.
.
.
eventID_begin = MPE_Log_get_event_number();
eventID_end = MPE_Log_get_event_number();
.
.
.
MPE_Describe_state( eventID_begin, eventID_end, "Amult", "bluegreen" );
.
.
.
MyAmult( Matrix m, Vector v )
{
/* Log the start event along with the size of the matrix */
MPE_Log_event( eventID_begin, m->n, (char *)0 );
.
.
.
Amult code, including MPI calls ...
MPE_Log_event( eventID_end, 0, (char *)0 );
}
The log file generated by this code will have the MPI routines within
the routine MyAmult() indicated by a containing bluegreen rectangle.
If the MPE logging library, liblmpe.a, are NOT linked with the user
program, MPE_Init_log() and MPE_Finish_log() need to be used before and
after all the MPE calls. Sample programs cpilog.c and fpi.f are
available in MPE source directory contrib/test or the installed
directory share/examples to illustrate the use of these MPE routines.
ENVIRONMENTAL VARIABLES
There are 2 environmental variables, TMPDIR and MPE_LOG_FORMAT , that
user
MAY NEED TO SET BEFORE THE GENERATION OF LOGFILES
MPE_LOG_FORMAT
- determines the format of the logfile generated from the
execution of application linked with MPE logging libraries. The
allowed value for MPE_LOG_FORMAT are CLOG, SLOG and ALOG. When
MPE_LOG_FORMAT is NOT set, CLOG is assumed.
TMPDIR - specifies a directory to be used as temporary storage for each
process. By default, when TMPDIR is NOT set, /tmp will be used.
When user needs to generate a very large logfile for long-
running MPI job, user needs to make sure that TMPDIR is big
enough to hold the temporary logfile which will be deleted if
the merged logfile can be created successfully. In order to
minimize the overhead of the logging to the MPI program, it is
highly recommended user to use a *local* file system for TMPDIR
.
Note : The final merged logfile will be written back to the file
system where process 0 is.
UTILITY PROGRAMS
In bin/, user can find several useful utility programs when
manipulating logfiles. These includes log format converters, e.g.
clog2slog, log format print programs, e.g. slog_print, and a script to
launch display program, logviewer.
LOG FORMAT CONVERTERS
clog2slog
- a CLOG to SLOG logfile converter. Since the automatic
generation of SLOG file through setting of environmental
variable MPE_LOG_FORMAT to SLOG may NOT work for some non well-
behaved MPI programs, using the logfile format converter can
generate extra diagnostic information about the condition of the
logfile. Also the converter allows one to adjust certain
parameters of the logfile, like frame size which is the segment
of the logfile to be displayed by Jumpshot-3’s time line window.
For non well behaved MPI program, one may need to increase the
frame size from the default 64KB to a bigger value. For more
information about the converter, do "clog2slog -h"
clog2alog
- a CLOG to ALOG logfile converter. It is not being developed.
It is provided here for backward compatibility purpose.
LOG FORMAT PRINT PROGRAMS
slog_print
- a stdout print program for SLOG file. It serves to check the
content of the logfile. If the SLOG file is too big, it may NOT
be useful to use slog_print. Also, when slog is NOT complete,
slog_print won’t work. So it serves as a simple test to check
if the SLOG file is generated completely.
clog_print
- stdout print program for CLOG file.
DISPLAY PROGRAM SELECTOR
logviewer
- the script which involves appropriate viewer based on the file
extension of logfile. For instance, if the logfile is foo.slog,
logviewer will invoke jumpshot-3 to display the logfile.
Jumpshot-3 resides in share/. For more information of
logviewer, do "logviewer -help" to list all available options.
ADDITIONAL INFORMATION
To view a list of configure options, type ./configure--help in the top-
level mpe directory.
In the Installation Manual, there is a section on MPE which gives
instructions and examples for each type of installation.
The README in the top-level mpe directory.
In the Users Guide, there is additional information describing the MPE
routines and the profiling libraries.
The man pages included in the distribution has information regarding
the MPE routines. The command mpich/bin/mpiman is a script that runs
xman on these man pages.
LOCATION
/home/MPI/mansrc/mpepage
9/2/2000