Using MCA Parameters With mpirun

Open MPI uses Modular Component Architecture (MCA) parameters to provide a way to tune your runtime environment. Each parameter corresponds to a specific function. You change the value of the parameter in order to change the function.

Developing an Open MPI application that uses MCA parameters poses a number of advantages. Developers and administrators can customize the Open MPI environment to suit the specific needs of hardware or the operating environment. For example, a system administrator might use MCA parameters to optimize an Open MPI installation on a network so that users only need to run with the default values to obtain the best performance.

In order to understand how MCA parameters fit within Open MPI, you must understand how the Modular Component Architecture is constructed.

About the Modular Component Architecture

The Modular Component Architecture (MCA) is the backbone for much of Open MPI’s functionality. It is a series of frameworks, components, and modules that are assembled at runtime to create an MPI implementation.

An MCA framework manages a specific Open MPI task (such as process launching for ORTE). Each MCA framework supports a single component type, but can support multiple versions of that type. The framework uses the services from the MCA base functionality to find and/or load components.

An MCA component is an implementation of a framework’s interface. It is a standalone collection of code that can be bundled into a plug-in that can be inserted into the Open MPI code base, either at runtime and/or at compile time.

An MCA module is an instance of a component. For example, if a node running an Open MPI application has multiple Ethernet NICs, the Open MPI application will contain one TCP MPI point-to-point component, but two TCP point-to-point modules.

For more information about the Open MPI Modular Component Architecture, see the Open MPI FAQ on runtime tuning at:

Open MPI Frameworks

You might think of these frameworks as ways to group MCA parameters by function. For example, the OMPI btl framework controls the functions in the byte transfer layer, or BTL (point-to-point byte movement) in the network. All of the MCA parameters that are grouped under btl affect the BTL layer.

In addition to the parameters that are grouped under the individual frameworks, there are top-level MCA parameters that affect the frameworks themselves and specify values to your Open MPI installation.

TABLE 7-1 Top-Level MCA Parameters
Parameter Group	Description
`mca`	Specify paths or functions for MCA parameters
`mpi`	Specify MPI behavior at runtime
`orte`	Specify debugging functions and components for ORTE
`opal`	Specify stack trace information

To view the available top-level parameters in each group, type the following command:

% ompi_info --param groupname groupname

where groupname stands for the parameter group you want to view. For example, to view the available MPI parameters, you would type:

% ompi_info --param mpi mpi

The ompi_info Command

The ompi_info command returns information about your Sun HPC ClusterTools/Open MPI installation. When you issue the command without any modifiers, ompi_info returns the following information:

Using the ompi_info Command With MCA Parameters

The ompi_info command can list the parameters for a given component, all the parameters for a specific framework, or all parameters. The ompi_info output for most parameters contains a description of the parameter. The output for any parameter shows the current value of that parameter.

% ompi_info --param all all

The output from ompi_info lists all of the installed frameworks, their MCA parameters, and their current values.

% ompi_info --param btl all

In this example, the command output will list all available MCA parameters for the btl framework.

% ompi_info --param btl tcp

In this example, the command output will list all available MCA parameters for the tcp component, which is part of the btl framework.

Using MCA Parameters

1. Setting the parameter from the command line using the mpirun --mca command. This method assumes the highest precedence; values set for parameters using this method override any other values specified for the same parameter.

2. Using the parameter as an environment variable. Values for parameters set in this fashion assume the next highest priority.

3. Setting the parameter values in a text file. Parameter values specified using this method have the lowest priority.

% mpirun --mca param-name value

In this example, param-name stands for the name of the MCA parameter you want to set, and value stands for the new value you want to specify for the parameter. For example, the following command sets the value of the mpi_show_handle_leaks parameter to 1 for the specified job:

% mpirun --mca mpi_show_handle_leaks 1 -np 4 a.out

This sets the value of MCA parameter mpi_show_handle_leaks to 1 before running the program a.out with four processes.

Using MCA Parameters As Environment Variables

As with other types of environment variables, the syntax for setting MCA parameters as environment variables varies with the type of command shell.

% OMPI_MCA_param-name=value

where param-name is the name of the MCA parameter you want to set, and value is the desired value for the parameter. For example, the following command sets the mpi_show_handle_leaks parameter to 1:

% OMPI_MCA_mpi_show_handle_leaks=1

% export OMPI_MCA_param-name

For example, an export command using the parameter used in the previous step would look like this:

% export OMPI_MCA_mpi_show_handle_leaks

% mpirun -np 4 a.out

% setenv OMPI_MCA_param-name value

where param-name is the name of the MCA parameter you want to set, and value is the desired value for the parameter. The following example shows how to set the mpi_show_handle_leaks parameter to 1.

% setenv OMPI_MCA_mpi_show_handle_leaks 1

% mpirun -np 4 a.out

1. Create a text file, specifying each parameter/value pair on a separate line. Comments are allowed. For example:

# This is a comment
# Set the same MCA parameter as in previous examples
mpi_show_handle_leaks = 1
 
# Default to rsh always
plm_rsh_agent = rsh
 
mpi_preconnect_all = 1
mpi_param_check = 0
#
# udapl parameters - comment or uncomment as needed
#
#btl = self,tcp,sm
#btl = self,udapl,sm
btl = ^tcp

You can save the file either to your home directory under $HOME/.openmpi/mca-params.conf, where the parameter values in the file will only affect your jobs, or you can save it to /opt/SUNWhpc/HPC8.2.1c/sun/etc/openmpi-mca-params.conf, where the parameter values in the file affect all users.

The following example shows the output from the ompi_info command for mca_param_files.

% ompi_info --param mca mca_param_files
MCA mca: parameter "mca_param_files" (current value:
"/home/joeuser/.openmpi/mca-params.conf:
/opt/SUNWhpc/HPC8.2.1c/sun/etc/openmpi-mca-params.conf")
Path for MCA configuration files containing default parameter values
MCA mca: parameter "mca_component_path" (current value:
"/opt/SUNWhpc/HPC8.2.1c/sun/etc/openmpi:/home/joeuser/.openmpi/components")
Path where to look for Open MPI and ORTE components
MCA mca: parameter "mca_verbose" (current value: <none>)
Top-level verbosity parameter
MCA mca: parameter "mca_component_show_load_errors" (current value: "1")
Whether to show errors for components that failed to load or not
MCA mca: parameter "mca_component_disable_dlopen" (current value: "0")
Whether to attempt to disable opening dynamic components or not

The MCA parameter mca_param_files specifies a colon-delimited path of files to search for MCA parameters. Files to the left of the colon have lower precedence; files to the right of the colon have higher precedence. At runtime, mpirun searches the following two files in order when the mca_param_files parameter is set:

1. $HOME/.openmpi/mca-params.conf: The user-supplied set of values takes the highest precedence.

2. $prefix/etc/openmpi-mca-params.conf: The system-supplied set of values has a lower precedence.

In the above example, Open MPI first searches /home/joeuser/.openmpi/mca-params.conf for MCA parameters, and then searches /opt/SUNWhpc/HPC8.2.1c/sun/etc/openmpi-mca-params.conf. If a parameter appears in both locations, the value set in the second file (the file to the right of the colon) is used.

Including and Excluding Components

Each MCA framework has a top-level MCA parameter that you can use to select which components are to be used at runtime. In other words, there is an MCA parameter of the same name as each MCA framework (for example, btl) that you can use to include or exclude components from a given run.

You can use top-level parameters in the same way you would use other MCA parameters (for example, you can set them from the command line, as environment variables, or in text files).

For example, the btl MCA parameter is used to control which byte transfer layer (BTL) components are used with mpirun. The value for the btl parameter is a list of components separated by commas, with the optional prefix ^ (caret symbol).

% mpirun --mca framework comp1, comp2 ^comp3

In this example, the components comp1 and comp2 are included for the framework specified by --mca framework. Component comp3 is excluded, since it is preceded by the ^ (caret) symbol.

For example, the following command excludes the tcp and openib components from the BTL framework, and implicitly includes all the other components:

% mpirun --mca btl ^tcp,openib ...

The use of the caret followed by the ellipsis in the command means “Perform the opposite action with the rest of the components.” When the mpirun -- mca command specifies components to be excluded, the caret followed by the ellipsis ellipsis implicitly includes the rest of the components in that framework. When the mpirun --mca command specifically includes components, the caret followed by the ellipsis means “and exclude the components not specified.”

For example, the following command includes only the self, sm, and gm components of btl and implicitly excludes the rest:

% mpirun --mca btl self,sm,gm ...

Using MCA Parameters With Sun Grid Engine

The ras_gridengine parameters enable you to specify output from the Open MPI RAS (Resource Allocation Subsystem). The rsh PLM (Process Launch Module) contains the gridengine parameters.

The following example shows the mpirun command line being used to set the MCA parameter plm_gridengine_debug value to 100.

% mpirun -np 4 -mca plm_gridengine_debug 100 connectivity.sparc -v

To view a list of the RAS parameters from the command line, use the ompi_info command. The following example shows how to specify the RAS parameters.

% ompi_info -param ras gridengine

Changing the Default Values in MCA Parameters

There are options available in the MCA PLM and RAS components and modules to allow changes of the default values.

For more information about how to change the values in MCA parameters, see the General Run-time Tuning FAQ on the Open MPi Web site at:

For More Information

For more information about the Modular Component Architecture and MCA parameters, refer to the following sources:

About the Modular Component Architecture

Open MPI Frameworks

The `ompi_info` Command

Using the `ompi_info` Command With MCA Parameters