8.2 PSE User Environment Overview

This section describes the PSE user environment.

The Digital Parallel Software Environment provides the necessary components for the development and execution of parallel High Performance Fortran (HPF) programs on Digital Alpha Systems. PSE provides the optimized message passing that makes the execution of parallel HPF programs possible.

PSE extends the standard user execution environment to allow HPF applications to execute in parallel across a cluster of Alpha processors. You set up this extended environment, called a PSE cluster, by installing PSE software on one or more computers that share a single system image.

8.2.1 UNIX Semantics

PSE preserves the look and feel of the UNIX operating system. PSE can be thought of as an extension of Digital UNIX for parallel applications. With few exceptions, parallel application development and execution under PSE follow standard UNIX conventions, as shown in the following sequence:

% vi my_program.f90
% f90 -wsf 4 -o my_program my_program.f90
% my_program -farm sdscfarm -partition workstation -peers 4

All PSE parallel applications can:

Execute with redirected or piped I/O
Be placed in the background or run as a batch job
Open files for reading and writing
Read using stdin and write to stdout and stderr . Note, stderr is shared with stdout .

The application environment is controlled by a set of matching run- time options and default environment variables. For example, the -partition option and the PSE_PARTITION environment variable control the partition on which an application will run.

The concept of these options, discussed fully later in this chapter, is like the X-windows environment where X-applications are defined to accept standard run-time options and default environment variables such as -display and DISPLAY .

8.2.2 PSE Clusters

When a PSE cluster environment is defined, a collection of systems is configured by the PSE administrator into a database to share a common set of operational characteristics. Each characteristic governs a particular aspect of the execution of a parallel program, for example, the hosts used, the network used to communicate between peers, the default execution priority of the program, and so forth.

You are able to select among the defined characteristics best suited for the execution of your application by either using application command line options or setting the PSE environment variables. Some characteristics, such as which hosts define the PSE cluster membership and which communications options are available for users, can be defined only by a PSE administrator.

In simple hardware terms, a PSE cluster is a set of machines. Each machine is connected to each other using one or more networks (see Figure 8-1). From a software view, each machine should be configured similarly; this includes the running of an appropriate version of the Digital UNIX operating system and PSE software on each PSE cluster member.

Figure 8-1 Eight-Member PSE Cluster Connected by Ethernet and FDDI

PSE has a daemon, known as farmd , which is responsible for enforcing PSE cluster configuration rules, and for providing user applications with load balancing information.

8.2.3 Partitions

Partitions are PSE's principal grouping mechanism. Partitions are designed to make it easy to specify a pre-defined set of PSE cluster members for executing a parallel program. Partitions typically group together machines that are similar in some respect, such as hardware configuration or availability of software licenses and mounted file systems. PSE clusters always include a default partition to be used by those applications not requesting the use of a specific partition.

Typical installations usually define a partition which includes all PSE cluster members. Such an all-hosts partition can be useful both for general PSE cluster administration and for giving end users maximum flexibility to use the command-line options -use or -on to choose a set of machines for program execution (see Section 8.5).

In default PSE installations, the only partition available is an all-hosts partition named all-hosts . Customized PSE clusters allow additional partitions to be defined.

Partitions can overlap, i.e., any given PSE cluster member can belong to several partitions.

In the sample PSE cluster defined in Figure 8-1, the PSE administrator could, for example, define several partitions:

all-hosts-Specifies all PSE cluster members
ether-Consists of all eight members connected using the Ethernet network.
fddi-Consists of all eight members connected using the FDDI network. This partition is identified for this example as the default PSE partition.
bigmem-Composed solely from PSE cluster members having similar large real physical memories. The partition name is arbitrary, but grouping machines by available memory is a common practice.
chemistry - Many user sites may have certain applications which are constrained by license or by other issues; this particular partition has been defined to be available to users of a particular chemical analysis application.

Once a PSE administrator has installed and configured a PSE cluster, you can use the psemon(1) or lspart(1) utilities to examine the set of partitions and the other PSE cluster characteristics. These utilities can also be used in making decisions about setting default PSE environment variables or choosing options to maximize their parallel application performance.

You can execute your application in a particular partition either by using the command-line option -partition, or by setting the environment variable PSE_PARTITION.

For example, if you want to execute an application within a partition named fddi, just add fddi as an argument to the command-line option -partition, like this:

% my_program -partition fddi

Alternatively, you can give the partition name as an argument to the environment variable PSE_PARTITION, like this:

% setenv PSE_PARTITION fddi

When the environment variable is already set to the partition you want, it is not necessary to specify any partition on the command line:

% my_program

In these examples, PSE starts the user application (my_ program ) on the least-loaded members of the partition.

You can examine your existing PSE environment by issuing the UNIX command env(1). For example:

% env | grep PSE
PSE_FARM=sdscfarm
PSE_PARTITION=all-hosts

You can override environment variable definitions with command line switches. For example, to execute on the two least-loaded members of the partition named bigmem, your command line would be:

% my_program -partition bigmem -peers 2

8.2.4 Load Balancing

Normal peer distribution in PSE is load balanced; it allocates the required n process instances across the n least loaded machines in the selected partition. PSE farm daemons provide system load information to the load servers on a periodic basis.

The frequency of load-average sampling can be adjusted. Also, a limit on the total number of PSE jobs allowed can be assigned on a per-machine basis in order to control the impact of PSE jobs on the normal use of the machine.

For More Information:

On adjusting the frequency of load-balance sampling, see Section 16.3.3.12.
On assigning the PSE job-slot limit for each machine, see Section 17.2.3.2.

8.2.5 Physical and Virtual Modes

Physical mode and virtual mode refer to the behavior of PSE when an application needs more peer processes than the number of available CPUs. In SMP systems, each CPU is counted separately.

Physical and virtual mode are controlled by the -physical and -virtual run-time options, and the corresponding PSE_MACHINE environment variable.

8.2.5.1 Definition and Performance Considerations

In physical mode, PSE attempts to assign at most a single application peer to each CPU (Central Processing Unit) in the partition. Physical mode is the default.

In physical mode, the application fails at startup when the number of available CPUs is less than the number of peer processes required by the application.

In virtual mode, indicated by the use of the -virtual command-line option or by setting the PSE_MACHINE environment variable to virtual , PSE assigns more than one application peer to a CPU when necessary, as shown in Figure 8-2.

The -virtual command-line option causes performance degradation when the number of peers needed is greater than the number of available CPUs. When a sufficient number of CPUs are available, application performance is identical with or without the -virtual option.

8.2.5.2 Uses of -virtual

Application failure control: The -virtual command-line option can be used to specify how applications behave in case of insufficient physical resources. When you want the result to be failure at startup, run without -virtual . When you want the result to be a virtual-mode execution at the cost of reduced performance, use -virtual .
For example, if you attempt to run an application requiring eight peers in a partition with only four available CPUs, the result will depend upon the mode you use. In physical mode (the default), the application fails at startup and exits abnormally. In virtual mode, the application runs successfully, at reduced performance.
Simulating a larger PSE cluster: You can also use virtual mode to simulate a larger PSE cluster than is currently available or desired. For example, even if you have access to a large cluster, you may want to limit execution to a few CPUs during the development and debugging stages of an application, when you are more concerned with program correctness than with performance. For example:
```
      % my_program -peers 4 -debug -virtual -on myalpha.xyz.edu
```