4.0 Fortran

Fortran is the most widely used programming language for high-performance scientific computing applications. In the high-performance computing community it has received more attention than any other single language. Indeed, many parallel Fortran dialects exist, ranging from Fortran M to Fortran 90 and HPF. In the past the Legion project has not attempted to support Fortran in its native form, opting instead to concentrate on object-oriented languages more closely matched to the Legion programming model. The effort to provide similar compiler support for Fortran that we provide for C++ was beyond the scope of the project.

An alternative to full-fledged compiler support has been under discussion for some time and is now available. This alternative, which we call Basic Fortran Support or BFS, is a compromise between full-fledged compiler support and essentially no support. The basic idea is simple and harks back to the pseudo-comments used in early parallelizing Fortran compilers such as Paraphrase [20]. Rather than creating an extended Fortran dialect that includes coarse-grain objects and attempting to parse and perform data-dependence analysis on Fortran programs, we provide a set of Legion directives that can be embedded in Fortran code in the form of pseudo-comment lines. The programmer uses these embedded pseudo-comments to call member functions on Legion objects and to obtain the results of these member functions. The pseudo-comment syntax is intentionally similar to Fortran, and is easy for Fortran programmers to learn.

The pseudo-comments are translated by a simple preprocessor, legion_bfs, into calls to a BFS support library, which in turn directly invokes the Legion run-time system. The pseudo-comments direct the preprocessor to invoke member functions on Legion objects. Keywords in the pseudo-comments indicate whether the invocation is a subroutine-style invocation or a function, whether the call should be made synchronously or asynchronously, and whether the call is to a stateless object or to stateful object. The execution model is that of Legion--each "remote" member function invocation executes in a separate address space. Thus, all communication between the caller and callee is via parameters and returns values--there is no global memory. In the case of stateless objects, the back-end run-time system will create new object instances to service function invocations as needed.

In addition to invocations on Legion objects, the Fortran program can define Fortran-based Legion objects using a Fortran-like interface description language (IDL). The IDL is translated by the legion_bfs filter, which generates Fortran skeletons that can be linked to Fortran code in order to implement the object interface. The Fortran implementation code may itself be the output of the legion_bfs preprocessor, allowing Fortran code to act as both client and server in a Legion system.

4.1 Programming interface

Figure 4: Creation of a Legion client program using BFS. The client Fortran code, with embedded pseudo-comments describing Legion method invocations, is translated to standard Fortran by legion_bfs. This translated code is then compiled and linked against the Legion library and a BFS support library to create the executable client program.

The Legion BFS programming interface consists of two basic parts: client-side method invocation through pseudo-comment directives, and server-side object development through the BFS IDL. In "client" code (i.e. Fortran code that will invoke methods on Legion objects) pseudo-comments are used to describe method invocations. These pseudo-comments are translated into calls to a BFS support library, which in turn uses the services provided by the Legion system to support method invocation (see Figure 4). Fortran code that will be used to implement Legion objects (i.e. Fortran code that will be called through a Legion member function interface) must be described by programmer-supplied BFS IDL interfaces. These IDL interfaces are translated by the legion_bfs filter, which generates Mentat Programming Language (MPL) [11] skeleton code. This skeleton code is compiled using the legion_mplc compiler and linked to Fortran implementation code (Figure 5).

Figure 5: Creation of a Legion object using BFS. The object's interface, described in an IDL file, is translated by the legion_bfs filter to create MPL skeleton code. This skeleton code, together with the Fortran implementation code, is compiled and linked against the Legion library to create an executable objects.

It is certainly possible that "server" code--code that will be called through Legion member functions--may need to call the methods of other Legion objects.In such cases, the server implementation code includes the appropriate BFS pseudo-comment directives, is translated by legion_bfs, and is then linked to the appropriate IDL (also translated by BFS), as depicted in Figure 6. Thus, a single Fortran subroutine or function might both be called through Legion, and call other objects through Legion. In the following two sections, we examine the server and client programming interfaces in more detail. The combination of these two (as depicted in Figure 6) is straightforward, and is thus left to the reader.

Figure 6: Creation of a Legion object the performs Legion method invocations. Creation of such objects involves a combination of the "client" and "server" mechanism.

4.1.1 Server-side programming

We first consider the problem of implementing Legion objects in Fortran--that is, the process of taking subroutines and functions defined in Fortran and making them available as Legion methods. Fortran code wrapped in this manner can be called in parallel from anywhere in a Legion network. In Section 4.1.2 we will examine the programming interface used to invoke functions on Legion objects--i.e. the client side interface.

4.1.1.1 Class styles

BFS supports the definition of two basic kinds of Legion objects: stateless and stateful. Stateless objects are useful for making purely functional services available. Stateless objects support methods whose outputs are pure functions of their inputs--the operation of a stateless object is not affected by previous method invocations or evolving object state. Because of this, methods invoked on stateless objects are automatically load-balanced among all available instances of the given stateless class, thus allowing good performance in variably loaded or heterogeneous networks. Despite this advantage, it is often more natural to deal with objects that maintain state between method invocations. To enable this programming style, BFS supports stateful classes. Instances of these classes have normal method invocation semantics: methods are invoked on a specific object, and the object "remembers" its state from one invocation to the next.

4.1.1.2 Mechanics

To make Fortran subroutines and functions available in the form of a Legion object, we must "wrap" the Fortran code that implements the functions in a C++/MPL skeleton, invoke the MPL compiler, and link the generated MPL objects with the Fortran implementation code provided by the user. This process is automated through the use of a simple IDL that follows the same form as the caller-side specification of the function (discussed in Section 4.1.2). The user specifies the name and parameters of all functions in an interface specification (i.e. IDL) file. The legion_bfs filter parses this interface specification file and generates an MPL program that matches the interface. The user then compiles and links the program using legion_mplc. No changes to the Fortran code, nor any C++ or MPL programming on the part of the user is required.

Suppose we wish to make a Fortran module containing a function called function1 and a subroutine called subroutn1 available in the form of a stateless Legion object. The following IDL interface specification file would be used:

C Comments allowed
LEGION STATELESS CLASS my_class
     real function1(integer)
     subroutn1(INOUT integer I, INOUT real dimension (*,*))
CLASS END

The syntax for stateful objects is similar. For example:

LEGION CLASS my_class
     mysub(inout integer I)
     initialize(integer I, character path)
CLASS END

There should be exactly one class specification in each input file. BFS IDL files require a ".bfs" suffix.

4.1.1.3 Compilation

To compile an object, first translate the IDL file using legion_bfs. Given a file with a ".bfs" suffix, legion_bfs will generate a ".c" and a ".h" MPL file. The resulting MPL file should be compiled using legion_mplc. The object file containing the Fortran implementation code should be specified on the legion_mplc link line. For example:

$ legion_bfs my_class.bfs
  Parsing class my_class
$ f77 -c my_class_impl.f
$ legion_mplc my_class.c my_class_impl.o -o my_class

To specify the object's context path, use the -LegionOut option:

$ legion_mplc my_class.c my_class_impl.o -o my_class \
  -LegionOut /home/user_name/my_class

Without the -LegionOut option, stateful objects are placed in the current working context and stateless objects are placed in the /class/StatelessClasses context.

4.1.2 Client-side programming

In this section, we examine the client-side (i.e. method invocation) programming interface. The client-side programming interface is based on the use of pseudo-comments. The programmer embeds BFS directives into normal Fortran code using comments that begin with the prefix LEGION. This "annotated" Fortran code is then translated by the legion_bfs filter, which generates standard Fortran code. This resulting code is then compiled and linked against a BFS support library and the Legion library.

4.1.2.1 Include Files

Before methods can be invoked on objects of a given class, the BFS translator must have a description of the class's interface. This interface description will be used by legion_bfs to generate the appropriate calls to the Legion run-time system. The mechanism for communicating object interfaces to the translator is the INCLUDE directive, which is used to "include" BFS IDL files in BFS Fortran code. For example, the following statement

C LEGION INCLUDE my_class.bfs

should precede any methods called on instances of the class my_class.

4.1.2.2 Method invocation

In this section we cover the BFS syntax used to invoke methods. We start with a very simple example: a blocking remote method invocation that takes a single parameter and returns a single result. To call remote method function1 defined by stateless class my_class, which takes an integer parameter and returns a real result, the following syntax could be used:

C LEGION SYNCH FUNCTION real X = (my_class)function1(integer I)

This would perform a blocking remote procedure call (RPC) on an instance of my_class: calling function1, passing the integer parameter I as an input, and placing the real return value in the variable X. Note that the complete method invocation is contained within a single Fortran comment line. Note also that this method is invoked on a stateless Legion class, whose instances are therefore pure functional units. Thus, this method will be serviced not by a specific named object but by any available (or newly created) instance of my_class.

The above example performs a standard, blocking RPC on a Legion object. To achieve parallelism between remote method invocations, a non-blocking RPC mechanism is necessary. In BFS, non-blocking RPCs are achieved through the use of the ASYNCH specifier.

C LEGION ASYNCH FUNCTION real X = (my_class)function1(integer I)

This statement causes the same function to be executed but does not wait for the return value to be placed in X. When this statement is executed, the method function1 begins to be processed at the remote object, and the caller immediately proceeds. The caller can later block for the result using the statement:

C LEGION BLOCK real X

Since the variable X was previously named as the result of an asynchronous method invocation, when this statement is encountered the caller blocks for the result and assigns it to X. The use of asynchronous method calls allows methods to execute in parallel. For example, if function1 is time consuming, a caller of this function might perform other work in parallel before blocking for the result. This parallel work could include the invocation of other Legion methods, such as additional calls to function1.

Unlike Fortran, which uses call-by-reference parameter passing, the default BFS parameter passing convention is call-by-value. So, if the variable I in the above example is modified by function1, that change would not be propagated back to the caller. Other parameter-passing semantics are supported through the use of the key words IN, OUT, and INOUT. For example:

C LEGION ASYNCH FUNCTION real X = (my_class)function2(INOUT integer I)

This specifies that the variable I is call-by-value-result. The value of I is passed to the function, and when the function terminates the new value of I will be copied from the callee to the caller.

The above examples demonstrate functions--methods that produce a return value. Standard Fortran subroutines are also supported. For example:

c LEGION SUBROUTINE (my_class)subroutn1(INOUT integer I)

The analogous asynchronous call is:

C LEGION ASYNCH SUBROUTINE (my_class)subroutn1(INOUT integer I)
C LEGION BLOCK integer I

So far, we have only considered scalar parameters. Routines of interest often deal with arrays, however. To pass the array A into a method subroutn2, the following syntax is used:

C LEGION SUBROUTINE (my_class)subroutn2(INOUT integer 
     dimension(10,10) A)

The effect is to pass the array A into subroutn2, and, when the method completes, to copy the array back to the caller. The maximum number of dimensions is not fixed, though the user should be aware that there must be adequate memory to copy parameters, and that large parameters require more communication time.

To this point, we have only demonstrated calls on stateless objects. As described in Section 4.1.1, BFS also supports stateful objects--objects that maintain state between method invocations. Unlike stateless invocations, which are performed on any object of a given class, stateful invocations must be performed on a specific, named object instance. In BFS, Legion objects are identified by integer OIDs (Object IDentifiers) that are only valid in the local address space. OIDs are obtained in two ways: as the result of object creation requests, and as the result of looking up object names in Legion context space.

Object creation is supported through the CREATE directive. For example:

C LEGION CREATE OBJ = NEW my_stateful_class

This statement will cause the creation of a new object my_stateful_class. An OID that refers to the new object is stored into the integer variable OBJ. In addition to creating new objects, BFS programs can bind to existing objects. An OID for an existing object, which is named in Legion context space, can be obtained using the LOOKUP directive:

C LEGION LOOKUP OLDOBJ = my_object

This statement will look up the existing object named my_object in Legion context space, and store an OID that refers to my_object in OLDOBJ.

The syntax for invoking methods on stateful objects is similar to that used with stateless objects, but introduces the need to specify a target object using an OID. For example:

C Legion SUBROUTINE OBJ->subroutn1(parameter list)

If your program is using multiple classes with methods of the same name, you must specify the class explicitly to disambiguate method invocations. For example:

C Legion SUBROUTINE (my_stateful_class)OBJ->subroutn1(parameter list)

This more verbose syntax is always acceptable, and will generally lead to easier to read, safer code.

Unlike memory that is declared within a program, stateful objects can persist indefinitely beyond the lifetime of the program that creates them. To avoid creating "garbage" objects--objects that are no longer needed by any programs but are still consuming system resources--BFS programs must clean up its stateful objects before terminating. Stateful objects may be deleted using the DESTROY directive.

C LEGION DESTROY OBJ

This statement will destroy the object referred to by the OID OBJ.

4.1.2.3 Compilation

Fortran code containing the pseudo-comment directives discussed in this section must first be preprocessed by the legion_bfs translator. Given a file with a ".f" suffix, the translator will produce a new Fortran source file with the suffix ".trans.f". This resulting file should be compiled by a standard Fortran compiler, and linked against the Legion libraries using the following flags:

-L$LEGION/lib/$LEGION_ARCH/g++ -lLegion1 -lLegion2 
-lLegionBFS

Note: to specify a search path for locating objects, users can set the Unix environment variable LEGION_OBJECT_PATH:

LEGION_OBJECT_PATH=/home/username/directoryA:/home/username/directoryB

4.2 Example program: stateful objects

In this section, we present a simple but complete program using stateful objects. Consider the Fortran code depicted in Figure 7. To make this code available in the form of a Legion object, we define an object interface for it using the IDL depicted in Figure 8. Note that since the Fortran code in Figure 7 depends on state that is maintained between method invocations (i.e. the common block variable A), we use a stateful Legion object to wrap the code.

Once the IDL depicted in Figure 8 has been translated by legion_bfs, compiled by legion_mplc, and linked to the object code resulting from a Fortran compilation of the code in Figure 7, programs can create objects of the class "my_class" and invoke methods on them. In Figure 9 (below) we depict a simple BFS Fortran main program that demonstrates object creation, a remote subroutine call, a remote function call, and object deletion. Note that the integer variable OID is used as a local reference to the Legion object created in the program. Also note that in this simple example we use synchronous method invocation, since no parallelism was possible between the two method invocations. In section 4.3 we consider the use of asynchronous methods and stateless objects, both of which have the potential to offer improved performance.

The output of the program depicted in Figure 9 will be:

1 + 2 + 2 = 5

4.3 Example program: stateless objects

The previous example used stateful objects, since the wrapped Fortran code maintained state from one method invocation to the next. Consider, however, the code depicted in Figure 10.

Both of the functions defined in this file are purely functional--they are free of side effects, and do not rely on state set by previous methods in producing their results. Because of these features, this Fortran module can be wrapped in a stateless Legion object, and thus benefit from improved performance through automatic load balancing. The BFS IDL required to wrap this code is depicted in Figure 11, below.

Figure 12, also below, depicts a simple Fortran main program that uses the stateless class dprod_object. This example performs two dot product operations and computes the sum of their results.Note that the two dot product operations are completely independent of one another--they operate on entirely disjoint data. Thus, we use ASYNCH methods to allow the functions to proceed in parallel. A further benefit of using ASYNCH methods comes from data dependence analysis performed by BFS: since the results of the dot product operations are needed to perform the sum operation, the results of these methods are forwarded directly to the object that will perform the sum operation. Direct forwarding of results, as afforded by ASYNCH methods, improves performance by reducing communication.

In Figure 12, instead of the results D1 and D2 being sent first to the main program and then to the sum operation, the parameters skip the middle hop and go directly to the sum operation. This optimization is especially important when array parameters are used, as they consume the most communication resources. The output of the program depicted in Figure 12 is:

SUM = 12.

4.4 Current limitations

The BFS type system is currently very limited: only REAL, INTEGER, LOGICAL, COMPLEX, and CHARACTER are supported. Types such as DOUBLE PRECISION and sized types such as INTEGER*8 are not yet available.

4.5 BFS syntax

This grammar is provided as an aid to understanding the BFS syntax, and as a guide to implementing the syntax. It is neither complete nor suitable for automatic processing.

legion@Virginia.edu
http://legion.virginia.edu/