7.8 Avoiding Unnecessary Communications Setup for Allocatable Arrays

One of the important optimizations in Digital's HPF compiler is that set-up for data communications is eliminated when it can be proven at compile time that communications will not be necessary. Eliminating communications set-up can provide a significant performance improvement.

Although the removal of communications set-up is a Digital-specific optimization, the proofs that communications set-up is unnecessary are general proofs based on the rules of the HPF language.

When allocatable arrays are used, the sizes and lower bounds of each array dimension are not known at compile time. Nevertheless, it is often possible to write ALIGN directives that give enough information to allow the compiler to prove when communication is not necessary.

The key is to know whether to write an ALIGN directive with an align subscript, like this: 

!HPF$ ALIGN B(i) WITH C(i)

or without an align subscript, like this: 

!HPF$ ALIGN B WITH C

These two forms have slightly different semantics. When an align subscript is used, the align target is permitted to be larger than the alignee. Also, elements whose subscripts are equal are aligned, regardless of what the lower bound of each array happens to be.

When an align subscript is not used, the alignee and the align target must be exactly the same size. Corresponding elements are aligned beginning with the lower bound of each array, regardless of whether the subscripts of the corresponding elements are equal.

The rule of thumb is: When allocatable arrays are used in a FORALL assignment, use an ALIGN directive with an align subscript. When allocatable arrays are used in a whole array assignment, use an ALIGN directive without an align subscript.

Example 7-1 illustrates this rule of thumb by comparing the two forms of the ALIGN directive:

Example 7-1 Avoiding Communication Set-up with Allocatable Arrays

              SUBROUTINE NO_SUBSCRIPT(n)    |       SUBROUTINE SUBSCRIPT(n)
                                            |
              INTEGER i                     |       INTEGER i
              REAL         :: B, A, C       |       REAL         :: A, B, C
              ALLOCATABLE  :: B, A, C       |       ALLOCATABLE  :: A, B, C
              DIMENSION(:) :: B, A, C       |       DIMENSION(:) :: A, B, C
                                            |
        !HPF$ DISTRIBUTE C(BLOCK)           | !HPF$ DISTRIBUTE C(BLOCK)
        !HPF$ ALIGN A WITH C                | !HPF$ ALIGN A(i) WITH C(i)
        !HPF$ ALIGN B WITH C                | !HPF$ ALIGN B(i) WITH C(i)
                                            |
        ! Local                             | ! May require communication
              A = B                         |        A = B
                                            |
        ! May require communication         | ! Local
              FORALL (i=1:n) A(i) = B(i)    |        FORALL (i=1:n) A(i) = B(i)
                                            |
        ! Local                             | ! Local
              A = C                         |        A = C
                                            |
              END SUBROUTINE NO_SUBSCRIPT   |        END SUBROUTINE SUBSCRIPT

The statements commented as "Local" can be proven to require no communication, and the compiler will eliminate communications set- up. Communications set-up cannot be removed for the statements commented as "May require communication". Of course, even when communications set-up is performed, no superfluous data motion will occur if communication turns out at run time to be unnecessary.

Table 7-1 explains why communication may be (or is not) needed for each statement in Example 7-1:

Table 7-1 Explanation of Example 7-1

Statement:  Routine: NO_SUBSCRIPT  Routine: SUBSCRIPT 
A = B  The "no subscript" form of the ALIGN directives requires both A and B to be the same size as C. A and B are therefore aligned with each other.  The "subscript" form of the align directives allows C to be larger than A or B. Since the lower bounds of the three arrays are unknown and can potentially be different from one another, it is possible that A is aligned with a different part of C than B is. 
FORALL (i=1:n) A(i)=B(i)  Even though A and B must both be the same size, their lower bounds may be different. If n is smaller than the extent of the arrays, and A and B have different lower bounds, then A(1:n) is not aligned with B(1:n).  The "subscript" form of the ALIGN directive guarantees that elements whose subscripts are equal are aligned. Even if C is larger than A and B, and even if A is aligned with a different part of C than B is, the sections A(1:n) and B(1:n) are both aligned with C(1:n), and are therefore aligned with each other. 
A = C  The "no subscript" form of the ALIGN directive requires A to be the same size as C. The whole-array assignment syntax also requires this. Corresponding elements are therefore aligned, whether or not the two arrays have the same lower bound.  The "subscript" form of the ALIGN directive states that all elements whose subscripts are equal are aligned. The whole-array assignment syntax requires A to be the same size as C. Therefore, A and C are aligned, and have the same lower bound. 

For More Information: