8.4 Setting Up the Appropriate PSE Environment

When running a parallel application, you should select PSE characteristics that match the application's execution requirements as closely as possible to ensure the best performance. In the worst case scenario, applications may not execute if environment characteristics such as the amount of physical memory or other operational constraints are not met. For example, a parallel program that has been compiled to run on eight peers must be run on a partition with at least eight available CPUs, or it exits with an error message.

You can define a default PSE environment by setting PSE-related environment variables. Chosen defaults should apply to most parallel applications of interest. The simplest case is to set the PSE_FARM environment variable to select a base set of resources that are used by PSE applications. PSE command- line options are used to fine-tune the environment defined by the existing PSE environment variables for each particular application run.

The general procedure for defining your PSE environment is:

Set up your default environment by defining PSE- related environment variables. This is typically done in your .login or your shell's startup (.cshrc for the C shell). Users should typically set PSE_FARM , PSE_PARTITION , and PSE_LOADSERVER .
Determine which characteristics are defined for each partition in the selected PSE cluster using psemon or lspart , that is, how many hosts are in the partition, the default execution priority and any priority limits, the list of authorized users and groups, and so on.
Based on the majority of your applications' general requirements, select a partition that comes closest to meeting those requirements.
On an application-by-application basis, command-line options can be used to override the environment defined by PSE-related environment variables.

The following sections describe PSE environment fine-tuning concepts. Although both environment variables and command-line options are discussed for each topic, in most cases the examples are given using the command-line option. Unless specifically stated otherwise, you can use the environment variable equivalent to create the environment condition globally (for your user process) in the same way as command-line options define values for the specific application instance. Very few options are accessible only using an environment variable definition.

For More Information:

On PSE_FARM, see Section 8.5.1.6.
On PSE_PARTITION, see Sections 8.2.3 and 8.5.1.10.
On PSE_LOADSERVER, see Sections 8.2.4 and 8.5.1.7.

8.4.1 Specifying a PSE Cluster

Defining a PSE cluster environment under which applications can be run is generally the first step you undertake before actually running HPF applications. A set of available PSE clusters is identified by executing the lspart -search command. You must also know if you have execution privileges on specific PSE clusters, in addition to what file systems are available for your use. The lspart or psemon commands can be used to examine the details of a PSE cluster configuration.

8.4.2 Selecting the Number of Peers

Applications are compiled to execute using a certain number of peer processes. In most cases this number is preset and embedded within the program executable; in other cases the number of peer processes is determined based on the number of available CPUs or the number specified using the -peers n command-line option. This is a direct result of the manner in which the program was compiled:

If the program was compiled with the -wsf option without an argument, the number of peer processes is determined at run time, based on the number of available CPUs or based on the -peers n command-line option.
If the program was compiled with an argument n to the -wsf option, the program must run using n peer processes.

When an application was compiled with the -wsf option without an argument, you can use the -peers n command-line option to set the number of peer processes.

The -peers n run-time option is ignored if the application was compiled with an argument n to the -wsf option.

You can use the -virtual run-time option to control whether execution fails when an insufficient number of CPUs are available. Virtual mode allows more than one peer process per CPU, at the expense of performance degradation.

For More Information:

On the -peers n command-line option, see Section 8.5.1.13.
On virtual mode and the -virtual run-time option, see Sections 8.2.5 and 8.5.1.11.

8.4.3 Specifying PSE Cluster Members in a Partition

As an alternative to the default load-balanced placement of peers, the user can specify peer placement and peer ordering using the -on member-list command-line option.

The -on command-line option assigns sequential peer numbers to the selected list of partition members. The hosts listed must be members of the current partition. SMP computers are assigned one peer for each CPU, and single-processor workstations are assigned one peer each. For example:

% my_program -peers 7 -on h1,h2,h6,h7

The list of hosts is separated by commas, with no spaces.

Instead of using the default method of assigning peers based on system load, PSE will assign peers based on the order in which the host names are specified to the -on option.

In this example, if the host named h2 is a four-processor SMP computer, and the other hosts are single-processor workstations, then the peers will be assigned as follows:

h1  -  peer 0
h2  -  peers 1, 4, 5, and 6
h6  -  peer 2
h7  -  peer 3

There are two reasons to use the -on option:

To select specific PSE cluster members for a program execution
To select the PSE cluster member on which that Peer 0 resides

If the application was compiled with -wsf option without a numerical argument, and the -peers n option is not specified, PSE defines the number of application peers to be the number of CPUs available on the hosts defined in the -on argument-list .

The -on option may be used together with the -virtual option.

8.4.3.1 Selecting Peer 0

When using the -on option, PSE software creates peers on the PSE cluster members you specify in the order you specify them. This may be both tedious and have an adverse performance impact if all you want is to place Peer 0 on a specific processor.

Selection of Peer 0 may be important because of such factors as I/O, graphics that may be displayed, memory that may be required, and so on. The -on command-line option offers a special syntax to support placement of a subset of nodes:

% a.out -peers 4 -on h1,...

The last member of the argument list is three periods (no spaces are allowed before the periods or in between them). This signifies to PSE that it should select the remaining PSE cluster members based on load-balancing considerations.

For More Information:

On the -on command-line option, see Section 8.5.1.12.
On the -peers command-line option, see Sections 8.4.2 and 8.5.1.13.
On the -virtual command-line option, see Sections 8.2.5 and 8.5.1.11.

8.4.4 Specifying the Execution Priority

PSE applications differ from normal processes in several ways. One very important dimension is that while PSE users are presented the image of a single system, functionally their applications are a loosely coordinated set of distributed processes. The coordinated sheduling of such processes is commonly called gang-scheduling.

Although true gang scheduling is not currently provided, PSE applications are able to execute at elevated execution priority using the -priority command-line option and gain a scheduling advantage over other processes in the system. In this fashion, PSE applications are able to gain most of the advantages of gang-scheduling.

Consider the following example:

% a.out -partition bigmem -peers 4 -priority 25

This example shows a user application being run at POSIX level 25, which is usually reserved for system level processes. The default execution priority for PSE applications is SCHED_PRIO_USER_MAX (see /usr/include/sched.h ) unless specified otherwise by the PSE administrator (see PSE_DEFAULT_PRIORITY ) and never exceeds that defined by PSE_PRIORITY_LIMIT .

Note: Digital recommends using priorities of 30 or less to prevent conflicts with the operating system software.