6.6 Extrinsic Procedures

HPF provides a mechanism by which HPF programs may call procedures written in other parallel programming styles or other programming languages. Because such procedures are themselves outside HPF, they are called extrinsic procedures.

In Digital Fortran 90, there are three kinds of EXTRINSIC procedures:

1. EXTRINSIC(HPF) (data parallel),
2. EXTRINSIC(HPF_LOCAL) (explicit SPMD), and
3. EXTRINSIC(HPF_SERIAL) (single processor).

These three kinds are explained in Section 6.6.1.

6.6.1 Programming Models and How They Are Specified

The following list describes programming models:

• Data Parallel Model [EXTRINSIC(HPF)] - This is a regular HPF procedure. It is the default programming model, producing the same result as if the EXTRINSIC prefix is not used at all. The language contains directives which you may use to tell the compiler how data is distributed across processors. Parallelism is attained by having processors operate simultaneously on different portions of this data. From the program's viewpoint, however, there is only a single thread of control.

  The source code does not ordinarily contain SEND or RECEIVE calls. If they are present in the source, they refer to communication with some exterior program (operating on another set of processors).

  Procedures written in this fashion are referred to as global HPF (or simply "HPF") procedures.

• Explicit SPMD Model [EXTRINSIC(HPF_LOCAL)] - A program is written containing explicit SENDs and RECEIVEs and references to GET_HPF_MYNODE (see the online man page for GET_HPF_MYNODE), and is loaded onto all processors and run in parallel. Storage for all arrays and scalar variables in the program is privatized; identical storage for each such data object exists on each processor. In general, the storage for an array A on distinct processors represents distinct slices through a global array with which you are really concerned (only you are aware of these global arrays, however; they have no representation in the source code).

  This is a multithreaded programming model, where the same program runs on each processor. Different processors execute different instructions, because the program on each processor is parameterized by references to the processor number, and because parallel data on different processors in general has different values. Since the data on each processor consists of different slices of global arrays, this programming model is referred to as single-program, multiple-data or SPMD.

  Each array element really has two kinds of addresses: a global address and a two-part address which consists of a particular processor and a local memory address in that processor. It is up to you to handle the translation between these two forms of addressing. Similarly, it is up to you to insert whatever SENDs and RECEIVEs are necessary. This involves figuring out which array elements have to be sent or received and where they have to go or come from. This requires you to explicitly translate between global and local addressing.

  Within the context of Digital Fortran 90, the explicit SPMD model is supported by EXTRINSIC(HPF_ LOCAL) procedures. These are procedures coded as shown previously, but they also have inquiry library routines available to them. These enable you to retrieve information about global arrays corresponding to dummy arrays in the procedure.

• Single Processor Model [EXTRINSIC(HPF_SERIAL)] - This is the conventional Fortran programming model. A program is written to execute on one processor which has a single linear memory. Sequence and storage association holds in this model, since it implements conventional Fortran 90.

  In this implementation, a single processor procedure may execute on any processor, with two exceptional cases:

  • If it is the main program (a Digital Fortran 90 program compiled without the -wsf option), it always executes on processor 0.
  • It may additionally be declared to be EXTRINSIC(HPF_ SERIAL). This is just an indication that it may be called from a global HPF procedure, and that the compiler generates code in the global procedure to move all the arguments to a single processor before the call, and back afterwards.

The compiler can be invoked in several ways, corresponding to these different programming models:

• When invoked with the -wsf switch but without an EXTRINSIC declaration, the compiler expects a global HPF source program, and generates the correct addressing and message-passing in the emitted object code.

  The compiler produces code containing explicit SENDs and RECEIVEs and references to the processor number. In effect, the compiler accepts a global HPF program and emits explicit SPMD object code.

  This implements the data parallel programming model.

• Without the -wsf switch and without an EXTRINSIC declaration, the compiler generates addressing for a single linear memory. Such a procedure can be used to implement the explicit SPMD programming model, by containing programmer-written message passing as needed. It can also be used to implement the single processor model. There is no way to discover by inspecting the source code of such a procedure which programming model is intended. The programming model is determined by how the procedure is invoked: on one processor or simultaneously on many.
• If the procedure is declared as EXTRINSIC(HPF_LOCAL) (if this procedure is called from a global HPF procedure, this declaration must also be visible in an explicit interface in the calling procedure), the procedure becomes an explicit SPMD procedure. The HPF_LOCAL declaration also has two other effects:
  • If the calling procedure is global HPF, it is a signal to the compiler when compiling the calling procedure to localize the passed parameters. That is, only the part of each passed parameter which lives on a given processor is passed as the actual parameter to the instance of the HPF_LOCAL procedure which is called on that processor.
  • It makes available to the HPF_LOCAL procedure a library of inquiry routines (the HPF Local Routine Library) which enable the local procedure to extract information about the global HPF data object corresponding to a given local dummy parameter. This information can be used (explicitly, by the programmer) to set up interprocessor communication as needed.
• If the procedure is declared as EXTRINSIC(HPF_SERIAL) the compiler processes it as any other single-processor Fortran 90 procedure. If the procedure is called from a global HPF procedure, the EXTRINSIC(HPF_SERIAL) declaration must be visible to the calling procedure in an explicit interface. The compiler generates code in the calling procedure that (before the call) moves all arguments to the processor on which the subprogram executes, and copies them back after the call if necessary.

An EXTRINSIC(HPF_LOCAL) or an EXTRINSIC(HPF_SERIAL) declaration in a procedure overrides the -wsf switch - as if that switch were not invoked for the procedure. This makes it possible to have an HPF_ LOCAL or HPF_SERIAL subprogram in the same file as that procedure. The compiler is invoked with the -wsf switch, but that switch has no effect on the compilation of the HPF_LOCAL or HPF_SERIAL subprocedure.

6.6.2 Who Can Call Whom

A single processor procedure which is the main program always executes on processor 0. Other than that, there is no restriction on where a single processor procedure executes.

A single processor procedure can call a single processor procedure.

A global HPF procedure can call:

• A global HPF procedure
• An HPF_LOCAL procedure
• An HPF_SERIAL procedure

An HPF_LOCAL procedure can call:

• An explicit SPMD procedure, which might be another HPF_LOCAL procedure
• A single processor procedure; such a procedure runs on the processor from which it is called (an example of this could be a scalar library procedure)

An explicit SPMD procedure which is not an HPF_LOCAL procedure (such as subroutine bar in Section 6.6.2.1) can call:

• Another explicit SPMD procedure; this procedure, however, cannot be an HPF_LOCAL procedure
• A single processor procedure; such a procedure runs on the processor from which it is called

This relaxes some of the restrictions in Annex A of the High Performance Fortran Language Specification.

6.6.2.1 Calling Non-HPF Subprograms from EXTRINSIC(HPF_LOCAL) Routines

According to the High Performance Fortran Language Specification, EXTRINSIC(HPF_LOCAL) routines are only allowed to call other EXTRINSIC(HPF_LOCAL) routines, EXTRINSIC(F90_LOCAL) routines, or other extrinsic routines that preserve EXTRINSIC(HPF) semantics.

Digital Fortran 90 does not currently support the optional extrinsic prefix EXTRINSIC(F90_LOCAL). However, Digital relaxes the restriction given in the High Performance Fortran Language Specification and allows (non-HPF) Fortran routines to be called from EXTRINSIC(HPF_ LOCAL) routines. This is done by calling the non-HPF subprogram without an EXTRINSIC prefix, as in the following example:

! The main program is an EXTRINSIC(HPF) routine
      PROGRAM MAIN
      INTERFACE
         EXTRINSIC(HPF_LOCAL) SUBROUTINE foo
         END SUBROUTINE foo
      END INTERFACE

      CALL foo()
      END

! foo is an EXTRINSIC(HPF_LOCAL) routine

      EXTRINSIC(HPF_LOCAL) SUBROUTINE foo
      INTERFACE
        SUBROUTINE bar(B)
        REAL B(100)
        END SUBROUTINE bar
      END INTERFACE
      REAL A(100)

      CALL bar(A)
      PRINT *, A(1)
      END SUBROUTINE foo

! bar is declared without an EXTRINSIC prefix

      SUBROUTINE bar(B)
      REAL B(100)

      B = 1.0
      END SUBROUTINE bar

The non-HPF routine bar is called from an EXTRINSIC(HPF_LOCAL) routine. It is declared without using an EXTRINSIC prefix. This is the only method of calling existing routines with non-assumed-shape arguments from EXTRINSIC(HPF_LOCAL) routines. This can be useful, for example, if you wish to call an existing routine written in Fortran 77, or in C.

6.6.3 Requirements on the Called EXTRINSIC Procedure

HPF requires a called EXTRINSIC(HPF_LOCAL) or EXTRINSIC(HPF_SERIAL) procedure to satisfy the following behavioral requirements:

• The overall implementation must behave as if all actions of the caller preceding the subprogram invocation are completed before any action of the subprogram is executed; and as if all actions of the subprogram are completed before any action of the caller following the subprogram invocation is executed.
• IN/OUT intent restrictions declared in the interface for the extrinsic subroutine must be obeyed.
• Replicated variables, if updated, must be updated consistently. If a variable accessible to a local subprogram has a replicated representation and is updated by (one or more copies of) the local subroutine, all copies of the replicated data must have identical values when the last processor returns from the local procedure.
• No HPF variable is modified unless it could be modified by an EXTRINSIC(HPF) procedure with the same explicit interface.
• When a subprogram returns and the caller resumes execution, all objects accessible to the caller after the call are mapped exactly as they were before the call. As with an ordinary HPF subprogram, actual arguments may be copied or remapped in any way, as long as the effect is undone on return from the subprogram.
• Exactly the same set of processors is visible to the HPF environment before and after the subprogram call.

For More Information:

• On the EXTRINSIC prefix, see the High Performance Fortran Language Specification.

6.6.4 Calling C Subprograms from HPF Programs

To write EXTRINSIC routines in C (or other non-HPF languages), you must make your subprogram conform to Fortran calling conventions. In particular, the subprogram may have to access information passed through dope vectors.

For More Information:

• On calling C subprograms from an HPF program, see Chapter 4.
• On calling parallel HPF programs from C-language main programs, see the following section (6.7).
• On mixed-language programming in general, see the Fortran User Manual.