This chapter describes the rudiments of developing, compiling and linking, executing, and debugging a Sun MPI program. The chapter focuses on what is specific to the Sun MPI implementation and, for the most part, does not repeat information that can be found in related documents. Information about programming with the Sun MPI I/O routines is in Chapter 4, Programming With Sun MPI I/O.
For complete information about developing MPI programs, see some of the MPI publications listed in the preface. For complete information about executing programs with LSF Suite and Sun HPC ClusterTools software, see the documentation that came with your LSF software and the Sun MPI 4.0 User's Guide: With LSF. For information about executing programs with the Sun Cluster Runtime Environment (CRE), see the Sun MPI 4.0 User`s Guide: With CRE.
Include syntax must be placed at the top of any program that calls Sun MPI routines.
For C, use
#include <mpi.h>
For C++, use
#include <mpi.h>
For Fortran, use
INCLUDE 'mpif.h'
These lines allow the program to access the Sun MPI version of the mpi header file, which contains the definitions, macros, and function prototypes required when compiling the program. Ensure that you are referencing the Sun MPI include file.
The include files are usually found in /opt/SUNWhpc/include/ or /opt/SUNWhpc/include/v9/. If the compiler cannot find them, check that they exist and are accessible from the machine on which you are compiling your code. The location of the include file is specified by a compiler option (see "Compiling and Linking").
/* * Test the connectivity between all processes. */ #pragma ident "@(#)connectivity.c 1.1 99/02/02" #include <errno.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <netdb.h> #include <unistd.h> #include <mpi.h> int main(int argc, char **argv) { MPI_Status status; int verbose = 0; int rank; int np; /* number of processes in job */ int peer; int i; int j; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &np); if (argc>1 && strcmp(argv[1], "-v")==0) verbose = 1; for (i=0; i<np; i++) { if (rank==i) { /* rank i sends to and receives from each higher rank */ for(j=i+1; j<np; j++) { if (verbose) printf("checking connection %4d <-> %-4d\n", i, j); MPI_Send(&rank, 1, MPI_INT, j, rank, MPI_COMM_WORLD); MPI_Recv(&peer, 1, MPI_INT, j, j, MPI_COMM_WORLD, &status); } } else if (rank>i) { /* receive from and reply to rank i */ MPI_Recv(&peer, 1, MPI_INT, i, i, MPI_COMM_WORLD, &status); MPI_Send(&rank, 1, MPI_INT, i, rank, MPI_COMM_WORLD); } } MPI_Barrier(MPI_COMM_WORLD); if (rank==0) printf("Connectivity test on %d processes PASSED.\n", np); MPI_Finalize(); return 0; } |
! ! Estimate pi via Monte-Carlo method. ! ! Each process sums how many of samplesize random points generated ! in the square (-1,-1),(-1,1),(1,1),(1,-1) fall in the circle of ! radius 1 and center (0,0), and then estimates pi from the formula ! pi = (4 * sum) / samplesize. ! The final estimate of pi is calculated at rank 0 as the average of ! all the estimates. ! program monte include 'mpif.h' double precision drand external drand double precision x, y, pi, pisum integer*4 ierr, rank, np integer*4 incircle, samplesize parameter(samplesize=2000000) call MPI_INIT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD, np, ierr) ! seed random number generator x = drand(2 + 11*rank) incircle = 0 do i = 1, samplesize x = drand(0)*2.0d0 - 1.0d0 ! generate a random point y = drand(0)*2.0d0 - 1.0d0 if ((x*x + y*y) .lt. 1.0d0) then incircle = incircle+1 ! point is in the circle endif end do pi = 4.0d0 * DBLE(incircle) / DBLE(samplesize) ! sum estimates at rank 0 call MPI_REDUCE(pi, pisum, 1, MPI_DOUBLE_PRECISION, MPI_SUM, & 0 , MPI_COMM_WORLD, ierr) if (rank .eq. 0) then ! final estimate is the average pi = pisum / DBLE(np) print '(A,I4,A,F8.6,A)','Monte-Carlo estimate of pi by ',np, & ' processes is ',pi,'.' endif call MPI_FINALIZE(ierr) end |
Sun MPI programs are compiled with ordinary C, C ++, or Fortran compilers, just like any other C, C ++, or Fortran program, and linked with the Sun MPI library.
The tmf77 and tmcc utilities may be used to compile Fortran 77 and C programs. For example, one might use
% tmf77 -fast -xarch=v8plusa -o a.out a.f -lmpi
to compile an f77 program that uses Sun MPI.
For performance, the single most important compilation switch is -fast. This is a macro that expands to settings appropriate for high performance for a general set of circumstances. Because its expansion varies from one compiler release to another, you may prefer to specify the underlying switches explicitly. To see what -fast expands to, use -v for "verbose" compilation output. Also, -fast assumes native compilation, so you should compile on UltraSPARC processors.
The next important compilation switch is -xarch. While -fast picks many performance-oriented settings by default, optimizations specific to UltraSPARC must be specified explicitly to override certain binary-compatible defaults. Specify
-xarch=v8plusa
or
-xarch=v9a
after -fast for 32-bit or 64-bit binaries, respectively. To run 64-bit binaries, you must use Solaris 7.
See the documents that came with your compiler for more information.
If you will be using the Prism debugger, you must compile your program with Sun WorkShop(TM) Compilers C/C ++ or Sun WorkShop Compilers Fortran, either v4.2 or v5.0. If the code is threaded, you will not be able to debug with Prism. (See "Debugging ".)
Table 3-1 Compile and Link Line Options for Sun MPI and Sun MPI I/O
When using . . . |
Use . . . |
---|---|
C (nonthreaded example) |
% cc filename.c -o filename \ -I/opt/SUNWhpc/include -L/opt/SUNWhpc/lib \ -R/opt/SUNWhpc/lib -lmpi |
C++ Note that x.y represents the version of your C++ compiler. |
% CC filename.cc -o filename \ -I/opt/SUNWhpc/include -L/opt/SUNWhpc/lib \ -R/opt/SUNWhpc/lib -L/opt/SUNWhpc/lib/SCx.y \ -R/opt/SUNWhpc/lib/SCx.y -mt -lmpi++ -lmpi |
tmcc, tmCC |
% tmcc -o filename filename.c -lmpi % tmCC -o filename filename.cc -mt -lmpi |
Fortran 77 (nonthreaded example) |
% f77 -dalign filename.f -o filename \ -I/opt/SUNWhpc/include -L/opt/SUNWhpc/lib \ -R/opt/SUNWhpc/lib -lmpi |
Fortran on a 64-bit system |
% f77 -dalign filename.f -o filename \ -I/opt/SUNWhpc/include/v9 -L/opt/SUNWhpc/lib/sparcv9 \ -R/opt/SUNWhpc/lib/sparcv9 -lmpi |
Fortran 90 |
Replace f77 with f90. |
tmf77, tmf90 |
% tmf77 -o -dalign filename filename.f -lmpi % tmf90 -o -dalign filename filename.f -lmpi |
Multithreaded programs and programs containing nonblocking MPI I/O routines |
Replace -lmpi with -lmpi_mt. |
For the Fortran interface, the -dalign option is necessary to avoid the possibility of bus errors. (The underlying C routines in Sun MPI internals assume that parameters and buffer types passed as REALs are double-aligned.)
If your program has previously been linked to any static libraries, you will have to relink it to libmpi.so before executing it.
The eight Sun MPI 4.0 libraries are described in "The Libraries". The paths for each of these libraries, which you must specify when you are compiling and linking your program, are listed in the following table.
Table 3-2 Sun MPI 4.0 Libraries
Category |
Description |
Path: /opt/SUNWhpc/lib/... |
---|---|---|
32-Bit Libraries |
Default, not thread-safe |
libmpi.so |
C++ (in addition to libmpi.so) |
SC4.2/libmpi++.so |
|
|
Thread-safe |
libmpi_mt.so |
Trace |
Trace, not thread-safe |
tnf/libmpi.so |
|
Trace, thread-safe |
tnf/libmpi_mt.so |
64-Bit Libraries |
Non-thread-safe |
sparcv9/libmpi.so |
C++ (in addition to sparcv9/libmpi.so) |
SC5.0/libmpi++.so |
|
|
Thread-safe |
sparcv9/libmpi_mt.so |
Trace |
Trace, not thread-safe |
tnf/sparcv9/libmpi.so |
|
Trace, thread-safe |
tnf/sparcv9/libmpi_mt.so |
As shown in the sample compile and link lines in Table 3-1, you use the -R flag in the compile and link line to specify the path for a run-time library when you are compiling. At run time, you can override the library specified in the -R argument by setting the LD_LIBRARY_PATH environment variable. For example, to link to the 32-bit trace libraries before running your program, do this:
% setenv LD_LIBRARY_PATH /opt/SUNWhpc/lib/tnf
(This is a C shell example.)
The libthread.so libraries are automatically linked into the respective libmpi.so libraries. This means that any thread-function calls in your program will be resolved by the libthread.so library. Simply omitting libthread.so from the link line will not cause thread calls to be stubbed out -- you must remove the thread calls yourself. For more information about the libthread.so library, see its man page. (For the location of Solaris man pages at your site, see your system administrator.)
The Sun MPI 4.0 User's Guide: With LSF, the LSF Batch User's Guide, and the lsfintro and bsub man pages provide thorough instructions for executing jobs with the LSF Suite. Likewise, the Sun MPI 4.0 User's Guide: With CRE and the mprun man page provide detailed information about running jobs with the CRE. In this section you will find some basic information about executing jobs with either resource manager.
Before starting your job, you may want to set one or more environment variables, which are also described in the Sun MPI user's guides.
Running parallel jobs with LSF Suite is supported on up to 1024 processors and up to 64 nodes.
Parallel jobs can either be launched by LSF's Parallel Application Manager (PAM) or be submitted in queues configured to run PAM as the parallel job starter. LSF's bsub command launches both parallel interactive and batch jobs. For example, to start a batch job named mpijob on four CPUs, use this command:
% bsub -n 4 pam mpijob
To launch an interactive job, add the -I argument to the command line. For example, to launch an interactive job named earth on a single CPU in the queue named sun, which is configured to launch jobs with PAM):
% bsub -q sun -Ip -n 1 earth
Running parallel jobs with the CRE is supported on up to 256 processors and up to 64 nodes.
When using the CRE, parallel jobs are launched using the mprun command. For example, to start a job with 6 processes named mpijob, use this command:
% mprun -np 6 mpijob
Debugging parallel programs is notoriously difficult, since you are in effect debugging a program potentially made up of many distinct programs executing simultaneously. Even if the application is an SPMD one (single process, multiple data), each instance may be executing a different line of code at any instant. Prism eases the debugging process considerably.
Prism is recommended for debugging in the Sun HPC ClusterTools environment. However, if you need to debug multithreaded Sun MPI programs at the thread level, you should see "Debugging With dbx". See also "Debugging With MPE", if you are using the multiprocessing environment (MPE) from Argonne National Laboratory.
This section provides a brief introduction to the Prism development environment. For complete information about Prism, see the Prism 6.0 User's Guide.
Prism can debug only one Sun MPI job at a time. Therefore, if an MPI job spawns or connects to another job (using MPI_Comm_accept and MPI_Comm_connect to implement client/server communication, for example, or MPI_Comm_spawn to spawn jobs), the Prism session nonetheless has control of only the original MPI job to which it is attached. For example, a Prism session debugging a server job cannot also debug the clients of that job.
To use Prism to debug a Sun MPI program, the program must be written in the SPMD (single process, multiple data) style -- that is, all processes that make up a Sun MPI program must be running the same executable.
MPI_Comm_spawn_multiple can create multiple executables with only one job id. You cannot use Prism to debug jobs with different executables that have been spawned with this command.
To debug a Sun MPI program with Prism, you need to have compiled your program using one of the compilers included in either the Sun Performance WorkShop Fortran or Sun Performance WorkShop C++/C suite of tools.
To start Prism on a Sun MPI program, use the -n option to bsub to specify how many processors you want to run on. For example,
% prism -n 4 foo
launches Prism on executable foo with four processes.
This starts up a graphical version of Prism with your program loaded. You can then debug and visualize data in your Sun MPI program.
You can also attach Prism to running processes. First determine the job id (not the individual process id), or jid, using either bsub (in LSF) or mpps (in the CRE). (See the LSF Batch User's Guide for further information about bjobs. See the Sun MPI 4.0 User`s Guide: With CRE for further information about mpps.) Then specify the jid at the command line with the -n (or -np, -c, -p) option:
% prism -n 4 foo 12345
This will launch Prism and attach it to the processes running in job 12345.
To run graphical Prism, you must be running Solaris 2.6 or Solaris 7 with either OpenWindows(TM) or the Common Desktop Environment (CDE), and with your DISPLAY environment variable set correctly. See the Prism 6.0 User's Guide for information.
One important feature of Prism is that it lets you debug the Sun MPI program at any level of detail. You can look at the program as a whole or at subsets of processes within the program (for example, those that have an error condition), or at individual processes, all within the same debugging session. For complete information, see the Prism 6.0 User's Guide.
To debug your multithreaded program at the thread level, you can use dbx. The following example illustrates this method of debugging with LSF Suite.
Add a variable to block the process until you attach with dbx.
In this sample program, simple-comm, the wait_for_dbx variable is set to 1 to create a wait loop. It is placed before the function or functions to be debugged.
#include <stdio.h> #include "mpi.h" void main( int argc, char **argv ) { MPI_Comm comm_dup; int error; int wait_for_dbx = 1; if((error = (MPI_Init(&argc, &argv))) != MPI_SUCCESS) { printf("Bad Init\n"); exit(-1); } while (wait_for_dbx); error = MPI_Comm_dup(MPI_COMM_WORLD, &comm_dup); if (error != MPI_SUCCESS) { printf("Bad Dup\n"); exit(-1); } error = MPI_Comm_free(&comm_dup); if (error != MPI_SUCCESS) { printf("Bad Comm free\n"); exit(-1); } MPI_Finalize(); } |
Compile the code, then run it.
After compiling the program, run it using bsub. (See the LSF Batch User's Guide for more information.)
% bsub -n 4 -Ip simple-comm
Identify the processes to which you want to attach the debugger.
Use bjobs to obtain information about the processes in the task. (See the LSF Batch User's Guide for more about getting information about processes.)
Attach the debugger to the processes that you would like to debug.
Attach the debugger to the processes. If you would like to debug only a subset of the processes, you must set up a conditional in such a way that the while statement is executed in only the process(es) that you will debug.
% dbx simple-comm 10838 Attached to process 10838 with 2 LWPs t@1 (l@1) stopped in main at line 18 in file "simple-comm.c" 18 while (wait_for_dbx);
Set your variable such that it will allow the process to unblock.
At the dbx prompt, use assign to change the value of the variable (here wait_for_dbx) and, hence, unblock the processes.
(dbx) assign wait_for_dbx = 0
Debug the processes.
After you have attached and set the instrumentation code appropriately, you can start debugging the processes as you normally would with dbx.
The multiprocessing environment (MPE) available from Argonne National Laboratory includes a debugger that can also be used for debugging at the thread level. For information about obtaining and building MPE, see "MPE: Extensions to the Library".