Legion MPI cookbook Quick Guide:
Running a Legion MPI code with the fewest changes

MPI (Message Passing Interface) is a standard for writing parallel programs in message-passing environments. For more information please see the MPI web site at http://www.epm.ornl.gov/~walker/mpi/.

The current Legion implementation supports a core MPI interface, which includes messages passing, data marshaling and heterogeneous conversion. Legion supports legacy (native) MPI codes and provides an enhanced MPI environment using Legion features such as security and placement services. A link-in replacement MPI library uses the primitive services provided by Legion to support the MPI interface. MPI processes map directly to Legion objects.

There are two different ways to run MPI in a Legion system: legion MPI and native MPI. Legion MPI programs have been adapted to run in Legion, are linked to Legion libraries, and can only be run on machines that have the Legion binaries installed.

This guide discusses Legion MPI. Click here for information about native MPI support. For more information about MPI in Legion, please see the Basic User manual.

Table of Contents
Modifying the Program
Running this Legion-MPI Program
Limitations of this "fewest changes" approach
Modifying the program for the Typed Binary Interface
Other on-line tutorials & documentation

Click on the to move to the selected text.


Depending on how your system is set up, you may need to set up your access to your system before you can run Legion commands. This will probably involve running a command such as this:
$ . ~legion/setup.sh
     or
$ source ~legion/setup.csh
The exact syntax will depend on what kind of shell you are using and where your Legion files are installed (i.e., the value of ~legion will depend on your individual Legion net). Consult your system administrator for more information.

The following style conventions are used in these tutorials:


Modifying the Program

Let's say that you have the following MPI program:

      program foo

      implicit none
      include 'mpif.h'
      integer ierr, nodenum, numprocs
      integer i, j, k

      call mpi_init( ierr )
      call mpi_comm_rank( mpi_comm_world, nodenum, ierr )
      call mpi_comm_size( mpi_comm_world, numprocs, ierr )

      if (nodenum .eq. 0) then
         open (10, file = 'input', status = 'old')
         read (10,*) i, j, k
         close (10)
      endif

      call do_work (i, j, k)

      if (nodenum .eq. 0) then
         open (11, file = 'output', status = 'new' )
         write (11, *) i, j, k
         close (11)
      endif

      call mpi_finalize(ierr)      
      stop
      end
The subroutine do_work() represents the computational part of the program.

In order to run this program over Legion and get the benefit of remote I/O, it is necessary to insert extra I/O calls. The MPI calls do not change. Here is the result, in which the changes are in all upper-case:

      program foo

      implicit none
      include 'mpif.h'
      integer ierr, nodenum, numprocs
      integer i, j, k
      CHARACTER*256 INPUT, OUTPUT

      call mpi_init( ierr )
      call mpi_comm_rank( mpi_comm_world, nodenum, ierr )
      call mpi_comm_size( mpi_comm_world, numprocs, ierr )

      if (nodenum .eq. 0) then
         call LIOF_LEGION_TO_TEMPFILE ('input', INPUT, ierr)
         open (10, file = INPUT, status = 'old')
         read (10,*) i, j, k
         close (10)
      endif

      call do_work (i, j, k)

      if (nodenum .eq. 0) then
         call LIOF_CREATE_TEMPFILE (OUTPUT, IERR)
         open (11, file = OUTPUT, status = 'new' )
         write (11, *) i, j, k
         close (11)
         call LIOF_TEMPFILE_TO_LEGION (OUTPUT, 'output', IERR)
      endif

      call mpi_finalize(ierr)      
      stop
      end
We added four lines and changed two.

         CHARACTER*256 INPUT, OUTPUT
allocates space where we will store the local names of files we will read and write remotely.

         call LIOF_LEGION_TO_TEMPFILE ('input', INPUT, ierr)
copies the file with the Legion filename input into some local file, storing the name of that local file into the variable INPUT.

         open (10, file = INPUT, status = 'old')
opens this local copy of the file. We may then do normal Fortran I/O on this local file.

         call LIOF_CREATE_TEMPFILE (OUTPUT, IERR)
creates the local output file, storing the name of the file in the variable OUTPUT.

         open (11, file = OUTPUT, status = 'new' )
opens this local copy of the file. We may now do normal Fortran I/O to this local file.

         call LIOF_TEMPFILE_TO_LEGION (OUTPUT, 'output', IERR)
copies the local file OUTPUT into the Legion file with the name output.

Running this Legion-MPI Program

In order to run this program under Legion MPI, we need to:

  1. Compiling and linking

    Assuming that our program is in the file example.f, we would compile and link using:

    % f77 -c example.f -I$LEGION/include/MPI
    % legion_link -Fortran -mpi -o example example.o
    
  2. Setting up access to the Legion system

    Before you issue Legion commands, you may need to execute an extra command:

    % . /home/appnet/setup.sh
    or
    % source /home/appnet/setup.csh
    Also, before you run a Legion program, it is useful to have a Legion tty, which is an object which allows you to observe the output of your program.

    % legion_tty my_tty
    This creates a tty with the name my_tty in Legion space, and send the output from this tty object to your current terminal.

  3. Tell Legion where the executable and input file are:

    • executable
         % legion_mpi_register example ./example $LEGION_ARCH
      This command registers the unix file ./example as the MPI Class example. Legion will create a class object, named /mpi/programs/example, in Legion space.

    • input file
         % legion_cp -localsource ./input input
      This command creates a Legion FileObject with the name input, and copies the Unix file with the name ./input into it.

  4. Run the program

    Now you may run the program:

    % legion_mpi_run -n 4 /mpi/programs/example
    This command will run your program on four nodes somewhere in the Legion system. Any output sent to the screen will be sent to your tty object.

  5. Copy the output file back to our workstation

    Finally, we might want to copy the output file named output in Legion space back to our local workstation.

    legion_cp -localdest output ./output
Limitations of this "fewest changes" approach

While this approach allows you to run MPI programs and transparently read and write files remotely, it does have one limitation: it does not support heterogeneous conversions of data. If you run this program on several machines which have different formats for an integer, such as Intel PC's (little-endian) and IBM RS/6000's (big-endian), the result of using unformatted I/O will be surprising. If you want to use such a heterogeneous system, you will have to either use formatted I/O (all files are text) or use the "typed binary" I/O calls instead of Fortran READ and WRITE statements. These "typed binary" I/O calls are discussed in "Buffered I/O Library, low impact interface," in the Legion Developer Manual.

Modifying the program for the Typed Binary Interface

Here's how the program would change if we used the typed binary interface:

      program foo

      implicit none
      include 'mpif.h'
      integer ierr, nodenum, numprocs
      integer i, j, k
      integer fd, ierr

      call mpi_init( ierr )
      call mpi_comm_rank( mpi_comm_world, nodenum, ierr )
      call mpi_comm_size( mpi_comm_world, numprocs, ierr )

      if (nodenum .eq. 0) then
         call LIOF_OPEN ('input', 0, FD)
         call LIOF_READ_INTS (FD, I, 1, IERR)
         call LIOF_READ_INTS (FD, J, 1, IERR)
         call LIOF_READ_INTS (FD, K, 1, IERR)
         call LIOF_CLOSE (FD, IERR)
      endif

      call do_work (i, j, k)

      if (nodenum .eq. 0) then
         call LIOF_OPEN ('output', 0, FD)
         call LIOF_WRITE_INTS (FD, I, 1, IERR)
         call LIOF_WRITE_INTS (FD, J, 1, IERR)
         call LIOF_WRITE_INTS (FD, K, 1, IERR)
         call LIOF_CLOSE (FD, IERR)
      endif

      call mpi_finalize(ierr)      
      stop
      end
Running the program would be the same.


Other relevant on-line documents:
Click on the to go to the page.
Logging in to a running Legion system
Introduction to Legion context space
Context-related commands
Legion tty objects
Running a PVM code in Legion
Running a Legion MPI code
Running native MPI code
Quick list of all 1.7 Legion commands
Usage of all 1.7 Legion commands
FAQs for running programs in Legion
Starting a new Legion system
Legion security
Legion host and vault objects
Adding host and vault objects
Brief descriptions of all on-line tutorials

Last modified: Thu Jun 15 16:19:52 2000

 

[Home] [General] [Documentation] [Software]
[Testbeds] [Et Cetera] [Map/Search]

legion@Virginia.edu
http://legion.virginia.edu/