1.2 What Is PSE?

PSE provides system software and tools to define and manage a cluster of Alpha systems. A PSE cluster is a collection of one or more Alpha workstations and SMP machines that are integrated to form a single processing entity. Each host in the cluster is referred to as a PSE cluster member. PSE manages the distribution of a parallel program among the cluster members while preserving UNIX syntax and semantics. PSE can be thought of as the operational glue between processors and HPF program execution. PSE also supplies development tools including a debugger and a profiler. PSE is layered upon a number of hardware, system, network, and configuration components.

1.2.1 PSE Cluster Definition

A PSE cluster is a collection of one or more Alpha workstations or SMPs that are integrated to form a single processing entity. Individual hosts belonging to the cluster are referred to as PSE cluster members.

A PSE cluster is more strictly defined as a collection of Alpha systems that run PSE cluster daemons and conform to a number of other requirements such as having a consistent file system name space. Each PSE cluster configuration is site specific. System managers have considerable control over the topology and configuration of their PSE cluster.

1.2.2 Baseline PSE Cluster Configuration

A baseline PSE cluster consists of a group of Alpha worksystems connected to a network with PSE installed and running, as illustrated in Figure 11-1.

1.2.3 Hardware Overview

The objective of creating a PSE cluster is to employ it in order to run parallel programs written in HPF. These programs will execute most efficiently when nearly identical machines with consistent system environment characteristics are connected using a high performance network. Some important considerations are:

• Basic Hardware - Basic hardware characteristics that must be considered when building a PSE cluster include processor type, amounts of memory, type of network, storage, as well as swap and /tmp space. By ensuring that hardware characteristics are similar between cluster members, parallel program execution occurs at a steady, even rate if the processing is evenly distributed. Consistent hardware configurations maximize overall performance of the cluster and reduce idle time.
• Interconnects and Protocols - Because communications overhead is a critical factor in parallel processing performance, the ideal interconnect between cluster members offers high bandwidth and low latency. Typical interconnects include Fiber Distributed Data Interface (FDDI) and Memory Channel.

  Another factor in cluster performance is the amount of common system communications (for example, Network File System (NFS)) that an application must contend with on a network during execution. By using a dedicated PSE cluster interconnect in addition to a common system network, contention can reduced.

  PSE supports the use of common network protocols such as Transmission Control Protocol/Internet Protocol (TCP/IP) or User Datagram Protocol/Internet Protocol (UDP/IP) and shared memory to implement its internal message passing. A fast UDP/IP called UDP Prime can be configured into the kernel to improve performance by lowering latency and increasing bandwidth.

1.2.4 Systems Software Overview

PSE defines several concepts, and requires a consistent software environment. These concepts are:

• Digital UNIX Operating System - All cluster members must be running the Digital UNIX operating system. Digital recommends that all cluster members run the same versions of all software.
• Single System Image - The PSE is designed to operate in an environment where machines are configured consistently so that it does not matter where an application is executed. In other words, the machines should be formally organized using good administrative practices such that user access, file name space, hostnames, and so on are consistent. PSE refers to this logical cluster configuration as the single system image. A single system image configuration allows a cluster of hosts to appear like a single host, increases efficiency, and simplifies trouble shooting. A cluster with a single system image environment offers the user the ability to log in to any cluster member and see the same system.

  To accomplish this, cluster members must have the following environment characteristics:

  • A common file name space - File systems that contain user files, and files referenced during application execution must all be accessible to each cluster member using a consistent name.
  • Consistent user access - Users have the same user name, password, user ID (uid ), and group ID (gid ) on all cluster members.
  • Consistent system data - The consistency of host, services, files, and so on is critical to produce the single system image required by Digital Parallel Software Environment.
  • Consistent clocks
  • Same or similar /tmp size
  • Same or similar swap space
  • Same or similar kernel configuration

  The methods recommended by Digital for creating a single system image are described in Chapter 12.

1.2.5 PSE Cluster Entities and Conventions

This section lists and defines the terminology associated with PSE clusters (PSE cluster database, servers, partitions, peers, and so on) and the conventions governing their use.

• PSE cluster Database - PSE can be customized to provide additional features beyond the default functionality. Customized PSE clusters require a database of member and configuration information. The database can reside in a file that is accessible to all cluster members. Alternatively, the database can be based on a Domain Name System (DNS), which distributes the data to the cluster members. PSE includes a PSE Database editor psedbedit that simplifies the creation and modification of a DNS-based PSE cluster database. It recognizes PSE tokens that are used to define a PSE cluster. When the PSE cluster database is changed it is automatically distributed using DNS, ensuring that the PSE cluster definition is consistent and current.

  A PSE cluster database includes:

  • Configuration information
  • Names of PSE cluster members
  • List of available networks
  • Sets of PSE cluster members that are grouped into partitions
• PSE cluster Database Servers - When a DNS-based database is used, DNS is used to distribute PSE cluster configuration information using a set of primary and secondary DNS servers. Using DNS to distribute data other than hostname information is nontraditional, yet no special steps need be taken because the PSE cluster database is just another subdomain. A primary DNS server, the PSE cluster database server, sources data to cluster members and to secondary servers that provide failover protection. When a DNS-based database is used, each PSE cluster member is required to be a DNS client.

  Usual role assignments for the primary PSE cluster database server include:

  • Primary DNS server for the PSE cluster database
  • Recommended Install server
  • Machine on which psedbedit is executed

  The PSE cluster database server does not need to be a PSE cluster member.

• Install Server - The Install server is the PSE cluster member on which the PSE binaries are installed first. The pse_ remote_install utility is then used to propagate the software to other members of the PSE cluster.
• Load Server - A Load server is a PSE cluster member that receives system load information from all other members and makes the information available to applications at startup time to allow load balancing of jobs across the cluster. Each PSE cluster requires that at least one member act as a load server. Multiple Load servers are recommended to increase load information availability.
• PSE cluster Membership, farmd , and Job Slots - Although potential PSE cluster members are defined using the PSE cluster database, actual membership and availability is controlled by the execution of a daemon process (farmd ). The farmd daemon responds to various requests from applications, PSE utilities, and other PSE cluster daemons. When the farmd daemon is started, it determines which PSE cluster it joins and the number of PSE cluster jobs (job slots) this machine allows to execute at a time. Limiting the number of job slots restricts the impact PSE cluster jobs have on a machine.

  The farmd daemon also communicates system load information to the Load servers.

• Partitions - In customized PSE clusters, members may be grouped arbitrarily into named sets called partitions that can be used to bound the machines on which an application runs. The goal in creating partitions is to create tailored, efficient parallel processing environments.

  For example, a partition grouping all members with large amounts of physical memory might be called BIGMEM. A partition called FIBER could have as its members all machines that are connected using FDDI and so forth.

• Peers - When a parallel program is executed under PSE, n program instances are created. The variable n is the number of processes required by the application. Each program instance is assigned a number ranging from 0 to n-1 and is referred to as a peer process or peer for short. For example, if an application requires four processes, the processes are known as Peer 0 through Peer 3.

  Peer 0 is a significant assignment because this peer becomes responsible for all the input/output (I/O) of the parallel program.

1.2.6 Parallel Application Execution

PSE is designed to support distributed parallel execution of HPF applications. Digital Fortran 90 supports the HPF extensions that are used to specify the distribution of data onto PSE cluster members during execution as shown in Figure 1-1.

Figure 1-1 Parallel Program Creation and Execution

A goal of PSE is allow the user to work with parallel applications in a manner as similar as possible to nonparallel applications. For example, setting the environment variables PSE_FARM to CHEMCLUSTER and PSE_PARTITION to BIGMEM targets parallel application execution onto the PSE cluster named CHEMCLUSTER on the partition BIGMEM.

A running application under PSE consists of a controlling process and n peer processes. When an application is started, it is the controlling process that communicates with the PSE cluster daemon to request load information to determine where to request the creation of peer processes. Next, the controlling process sends requests to the identified members to start the peer processes. Each peer process then creates communications channels to all other peers. Each peer communicates its stdio stream to the controlling process using an I/O Manager process.

1.2.7 Developing Parallel Applications

HPF is the language currently supported for developing parallel programs for execution under the PSE. The following facilities support HPF programming:

• Digital Fortran 90 Compiler - Includes support for HPF. The compiler is not included as a component of the PSE software. It is provided as a separate product.

  Digital Fortran 90 supports full HPF (High Performance Fortran Language Specification V 1.1), with the exception of, nested FORALLs, and WHERE in FORALLS. Digital Fortran 90 also supports most Fortran 95 features, and many of the optional approved extensions of High Performance Fortran Language Specification V 2.0, including shadow widths, mapped components of derived types, and certain forms of the ON directive.

  ---

  Note

  The PSE software and its environment are not required for either the creation or compilation of HPF programs. HPF programs can be compiled and debugged in scalar mode (nonparallel) on a single processor. However, programs compiled for parallel execution (with the -wsf flag) can be linked, executed, and profiled only in the PSE environment.

  ---

• PSE Tools and Libraries - Support the creation, parallel debugging, tuning, and execution of HPF programs. PSE includes:
  • Two debuggers:
    • dbx in n windows
    • Ladebug in n windows
  • A parallel profiler, pprof , for HPF programs
  • HPF parallel run-time libraries (RTL)
• Parallel Debugger (Ladebug or dbx) In n Windows - PSE supports modified versions of Ladebug and dbx that use the X Window System to generate an xterm per peer process for debugging HPF applications that are fully debugged on a single node but not in parallel. Distributed HPF arrays can be examined and debugging actions can be applied to all peers in an application (broadcast) or targeted to a single peer.
• Parallel Profiling Tool - The pprof profiler is used to analyze the execution of parallel HPF programs. The information in pprof reports can be used to locate hotspots and to identify parts of a program that might benefit from optimization. The pprof report also provides information about PSE system performance.

  When a Digital Fortran 90 program is compiled with the -pprof command line option, the compiler instruments the code to enable the generation of profiling data. There are two categories of performance profiling:

  • Interval profiling (event begin/end timing)
  • Program counter (PC) sampling

  Running the resulting executable on a PSE cluster creates the profiling data that is analyzed by the pprof utility. The profiler can be used to produce reports on numerous event types using different analysis options. Overall application performance or performance on individual peer processes can be examined.

• HPF Run-Time Library - Provides runtime support for message- passing code generated by the compiler, and provides serial and parallel versions of a variety of useful computational functions, including array reduction functions, array scatter (generalized array reduction) functions, prefix and suffix functions, sorting functions, and some bit manipulation functions.

1.2.8 PSE Utility Commands

PSE offers a set of utilities to monitor and manage the environment. They are lspart(1), pspart(1), and psemon(1).

• lspart -The list partition command is used to display PSE cluster database information, current system load, and job slot data for members of a partition. The lspart command can also be used to check rsh access to the members of the PSE cluster.
• pspart - The process status partition command is used to display the current status of processes running across a partition. The ps command is the basis for this command.
• psemon - This utility is a graphical user interface (GUI) supporting many of the features of lspart . It displays:
  • PSE cluster member names
  • system load average statistics
  • processor job slot values
  • partition configuration information

  Pull-down menus provide navigation through various Domain Name servers, PSE clusters and partitions. A resource file, PSEMon is used to customize psemon .