10.0 MPI

MPI (Message Passing Interface) is a standard for writing parallel programs in message-passing environments. For more information please see the MPI web site, <http://www-unix.mcs.anl.gov/mpi/>.

The current Legion implementation supports a core MPI interface, which includes messages passing, data marshaling and heterogeneous conversion. Legion supports legacy (native) MPI codes and provides an enhanced MPI environment using Legion features such as security and placement services. A link-in replacement MPI library uses the primitive services provided by Legion to support the MPI interface. MPI processes map directly to Legion objects.

The remainder of this section is divided into two sections: legion MPI (section 10.1, below) and native MPI (section 10.2). Legion MPI programs have been adapted to run in Legion, are linked to Legion libraries, and can only be run on machines that have the Legion binaries installed. Native MPI programs do not need to fulfill any of these requirements (although they may: a Legion MPI program can be run as a native MPI program but not vice versa).

10.1 Legion MPI

10.1.1 Task classes

MPI implementations generally require that MPI executables reside in a given place on disk. Legion's implementation objects serve a similar role. We have provided a tool, legion_mpi_register, to register executables of different architectures for use with Legion MPI.

10.1.2 Installing Legion MPI

The Legion MPI library can be found in $LEGION/src/ServiceObjects/MPI.

10.1.3 Compilation

MPI code may be compiled as before. Link it against libLegionMPI and the basic Legion libraries. The final linkage must be performed using a C++ compiler or by passing C++ appropriate flags to ld. A sample Legion MPI makefile is shown in Figure 6 (below).

Figure 6: Sample Legion MPI makefile

C       =       gcc
CC      =       g++
F77     =       g77
OPT     =       -g
INC_DIR =       /home/appnet/Legion/include/MPI
FFLAGS  =       -I$(INC_DIR) -ffixed-line-length-none 

mpitest: mpitest.o
        legion_link -Fortran -mpi mpitest.o -o mpitest

mpitest.o: mpitest.f
        $(F77) -c mpitest.f -I$(INC_DIR) $(FFLAGS)

mpitest_c: mpitest_c.o
        legion_link -mpi mpitest_c.o -o mpitest_c

mpitest_c: mpitest_c.c
        $(C) $(CFLAGS) mpitest_c.c -o mpitest_c $(LDFLAGS) $(LIBS)

10.1.4 Register compiled tasks

Run legion_mpi_register. Usage of this tool is:

legion_mpi_register <class name>
    <binary path> <platform type>
    [-help]

The example below registers /myMPIprograms/vdelay (the binary path) as using a Linux architecture.

$ legion_mpi_register vdelay /myMPIprograms/vdelay linux

If necessary, this will create the MPI-specific contexts (/mpi, /mpi/programs, and /mpi/instances), a class object for this program, an implementation for this program, and registers the name in Legion context space. In a secure system (a Legion system with security turned on), these contexts will be placed in /home/<user_name>. If the user is logged in as admin or the net is running without security then the mpi contexts will be placed in the / (root) context.

The legion_mpi_register command may be executed several times if you have compiled your program on several architectures.

10.1.5 Running the MPI application

MPI programs are started using the program legion_mpi_run:

legion_mpi_run {-f <options file> [<flags>]} |
    {-n <number of hosts> [<flags>] <command> [<args>]}
    [-help]

There are number of parameters and flags for this command. Please the Reference Manual for a full list and explanation of each flag's use.

To control scheduling, you can create a list of hosts and the number of Legion objects that can be placed on each host. This can be a local file or a Legion file object and should contain one host (use a context path) and one optional integer (indicating the number of objects the host can create -- default is 1) per line. A host can be listed multiple times in one file, in which case the integer values accumulate. E.g.:

/hosts/BootstrapHost	5
/hosts/slowMachine1
/hosts/bigPBSqueue 	100
/hosts/slowMachine2 	1

Alternatively, you can use the legion_make_schedule command to produce a file. You can then use the -hf or -HF flag to use this list when running a program.

You can start multiple MPI applications with the -f flag and an options file. All of the applications must have been registered with legion_mpi_register and use a common MPI_COMM_WORLD. The legion_mpi_run utility expects one application per line in the options file, including any necessary arguments as they would appear on the command line. Each line can also contain any of the legion_mpi_run flags except the -f flag. For example:

-n 2 /mpi/programs/mpitest
-n 3 /mpi/programs/mpitest_c

This would start a run of five instances (two of mpitest and three of mpitest_c). All five instances would share a single MPI_COMM_WORLD.

Note that if the -f flag is used, the -n flag, program name, and any arguments will be ignored. Any <flags> used with -f will be treated as defaults and applied to all processes executed by this command, unless otherwise specified in the options file.

If the -f flag is not used, the MPI application named in the <command> argument will then be started with the given flags.

For example:

$ legion_mpi_run -n 2 /mpi/programs/vdelay

Note that the application name here is the full context space path for the class created by legion_mpi_register in section 10.1.4. In this example the default path /mpi/programs is used with the class name vdelay.

You can examine the running objects of your application using legion_ls:

$ legion_ls /mpi/instances/vdelay

The /mpi/instances/<program_name> context has an entry for each object in the specified application. If you run multiple versions of an application simultaneously, you can use the -p flag to specify an alternate context in which to put the list of objects.

To view all instances of the program's class, use legion_list_instances:

$ legion_list_instances /mpi/programs/<program_name>

To destroy the program class's instances, use legion_destroy_instances command. Be aware, though that this command will destroy all of the class's instances.

MPI programs cannot be deactivated.

10.1.6 Example

There is a sample MPI program included in a new Legion system, written in C (mpitest_c.c) and Fortran (mpitest.f). Go to the directory containing the program's binary files:

$ cd $LEGION/src/ServiceObjects/MPI/examples

You must then register whichever version you wish to use with the legion_mpi_register command. The sample output below uses mpitest_c and shows Legion creating three new contexts (mpi, programs, and instances, the last two being subcontexts of mpi), creating and tagging a task class, and registering the implementation of the mpitest_c program.

$ legion_mpi_register mpitest_c \
      $LEGION/bin/$LEGION_ARCH/mpitest_c linux
"/mpi" does not exist - creating it
"/mpi/programs" does not exist - creating it
"/mpi/instances" does not exist - creating it
Task class "/mpi/programs/mpitest_c" does not exist -
	creating it
"/mpi/instances/mpitest_c" does not exist - creating it
Set the class tag to: mpitest_c
Registering implementation of mpitest_c
$

In order to view the output of the program, you will need to create and set a tty object if you have not already done so (see page 65).

$ legion_tty /context_path/mytty

Having done all of this, you can now run the program, with the legion_mpi_run command to run the program. You must specify how many processes the program should use to run the program with the -n flag.

If you do not specify which hosts the program should run its processes on, Legion will arbitrarily choose the hosts from the host objects that are in your system. Here it runs three processes on three different hosts.

$ legion_mpi_run -n 3 /mpi/programs/mpitest_c
Hello from node 0 of 3 hostname Host2
Hello from node 1 of 3 hostname Host3
Node 0 has a secret, and it's 42
Node 1 thinks the secret is 42
Hello from node 2 of 3 hostname BootstrapHost
Node 2 thinks the secret is 42
Node 0 thinks the secret is 42
Node 0 exits barrier and exits.
Node 1 exits barrier and exits.
Node 2 exits barrier and exits.
$

Another time it might run the three processes on two hosts.

$ legion_mpi_run -n 3 /mpi/programs/mpitest_c
Hello from node 1 of 3 hostname BootstrapHost
Hello from node 0 of 3 hostname Host3
Node 0 has a secret, and it's 42
Node 1 thinks the secret is 42
Hello from node 2 of 3 hostname BootstrapHost
Node 2 thinks the secret is 42
Node 0 thinks the secret is 42
Node 0 exits barrier and exits.
Node 1 exits barrier and exits.
Node 2 exits barrier and exits.
$

10.1.7 Accessing files in programs using MPI

When you run MPI, normally, you run on a single machine and all of your input files are locally placed. However, when you run an MPI program in Legion you will need to be able to transparently access other files for input or output. Legion has three subroutines that allow your program to read and write files in Legion context space: LIOF_LEGION_TO_TEMPFILE, LIOF_CREATE_TEMPFILE, and LIOF_TEMPFILE_TO_LEGION.

The first, LIOF_LEGION_TO_TEMPFILE, lets you copy a file from Legion context space into a local file. For example,

call LIOF_LEGION_TO_TEMPFILE ('input', INPUT, ierr)
open (10, file = INPUT, status = 'old')

will copy a file with the Legion filename input into a local file and store the name of that local file in the variable INPUT. The second line opens the local copy of the file.

The other two subroutines can be used to create and write to a local file. For example:

call LIOF_CREATE_TEMPFILE (OUTPUT, IERR)
open (11, file = OUTPUT, status = 'new')
call LIOF_TEMPFILE_TO_LEGION (OUTPUT, 'output', IERR)

The first line creates a local file and stores the file's name in the variable OUTPUT. The second line opens the local copy of the file. The third line copies the local file OUTPUT into a Legion file with the name output.

While this approach allows you to run MPI programs and transparently read and write files remotely, it does have one limitation: it does not support heterogeneous conversions of data. If you run this program on several machines which have different formats for an integer, such as Intel PC's (little-endian) and IBM RS/6000's (big-endian), using unformatted I/O may produce surprising results. If you want to use such a heterogeneous system, you will have to either use formatted I/O (all files are text) or use the "typed binary" I/O calls instead of Fortran READ and WRITE statements. These "typed binary" I/O calls are discussed in section 6.2.4 in the Legion Developer Manual.

10.1.8 Scheduling MPI processes

As noted above, Legion may not choose the same number of hosts objects as processes specified in the -n parameter. If you specify three processes, Legion will arbitrarily choose to run them on one, two, or three hosts.

If you wish to specify which hosts are used to run your processor, use the -h flag. In the example below, we will run mpitest_c on four hosts, which we will assume are already part of the system (see Host and vault objects for information about adding hosts to a Legion system) and which, in this example, are named in a single context called mpi-hosts.

First, you must create the mpi-hosts context with the legion_context_create command:

$ legion_context_create mpi-hosts

We can now assign new context names to the four hosts that we wish to use for this program, using the legion_ln command. These new names map to the host objects' LOIDs, and provide a convenient way to specify different hosts for different MPI programs or to execute the same program on different groups of hosts.

$ legion_ln /hosts/Host1 /mpi-hosts/mpitest_Host1
$ legion_ln /hosts/Host2 /mpi-hosts/mpitest_Host2
$ legion_ln /hosts/Host3 /mpi-hosts/mpitest_Host3
$ legion_ln /hosts/Host4 /mpi-hosts/mpitest_Host4

The program can now be run with legion_mpi_run, using the -h flag to specify that the four processes should be run on the hosts in mpi-hosts. Note that the nodes are placed on the hosts in order.

$ legion_mpi_run -h /mpi-hosts -n 4 /mpi/programs/mpitest
Hello from node 0 of 4 hostname Host1.xxx.xxx
Hello from node 1 of 4 hostname Host2.xxx.xxx
Node 0 has a secret, and it's 42
Hello from node 2 of 4 hostname Host3.xxx.xxx
Node 1 thinks the secret is 42
Node 2 thinks the secret is 42
Hello from node 3 of 4 hostname Host4.xxx.xxx
Node 3 thinks the secret is 42
Node 0 thinks the secret is 42
Node 0 exits barrier and exits.
Node 1 exits barrier and exits.
Node 2 exits barrier and exits.
Node 3 exits barrier and exits.
$

Note also that the -n parameter is still used, even though there are only four host objects named in the mpi-hosts context. The number of processes and number of host object do not have to match: if you wanted to run only two processes on the hosts named in mpi-hosts you could use -n 2. The program would then run on the first two hosts listed in the mpi-hosts context.1

$ legion_mpi_run -h /mpi-hosts -n 2 /mpi/programs/mpitest
Hello from node 0 of 2 hostname Host1.xxx.xxx
Node 0 has a secret, and it's 42
Hello from node 1 of 2 hostname Host2.xxx.xxx
Node 1 thinks the secret is 42
Node 0 thinks the secret is 42
Node 0 exits barrier and exits.
Node 1 exits barrier and exits.
$

10.1.9 Debugging support

A utility program, legion_mpi_debug, allows the user to examine the state of MPI objects and print out their message queues. This is a handy way to debug deadlock. Usage is:

legion_mpi_debug [-q]
    {[-c] <instance context path>}
    [-help]

The -q flag will list the contents of the queues. For example:

$ legion_mpi_debug -q /mpi/instances/vdelay
Process 0 state: spinning in MPI_Recv source 1 tag 0 comm?
Process 0 queue:

(queue empty)

Process 1 state: spinning in MPI_Recv source 0 tag 0 comm?
Process 1 queue:

MPI_message source 0 tag 0 len 8
$

Do not be alarmed that process 1 hasn't picked up the matching message in its queue: it will be picked up after the command has finished running.

There are a few limitations in legion_mpi_debug: If an MPI object doesn't respond, it will hang, and it won't go on to query additional objects. An MPI object can respond only when it enters the MPI library; if it is in an endless computational loop, it will never reply.

Output from a Legion MPI object to standard output or Fortran unit 6 is automatically directed to a tty object (see page 65). Note, though, that this output will be mixed with other output in your tty object so you may wish to segregate MPI output. Legion MPI supports a special set of library routines to print on the Legion tty, which is created using the legion_tty command. This will cause the MPI object's output to be printed in segregated blocks.

LMPI_Console_Output (string)
Buffers a string to be output later.

LMPI_Console_Output_Flush ()
Flushes all buffered strings to the tty.

LMPI_Console_Output_And_Flush (string)
Buffers and flushes in one call.

The Fortran version of these routines adds a carriage return at the end of each string, and omits trailing spaces. The C version does not.

10.1.10 Checkpointing support

As you increase the number of objects or the length of time for which an MPI application runs, you increase the probability that the application will crash due to a host or network failure.

To tolerate such failures, we provide a checkpointing library for SPMD-style (Single Program Multiple Data) applications.2 SPMD-style applications are characterized by a regular communication pattern. Typically, each task is responsible for a region of data and periodically exchanges boundary information with a neighboring task.

We exploit the regularity of SPMD applications to provide an easy-to-use checkpointing library. Users are responsible for (1) deciding when to take a checkpoint and (2) writing functions to save and restore checkpoint state. The library provides checkpoint management support, i.e. fault detection, checkpoint storage management, and recovery of applications. Furthermore, users are responsible for taking checkpoints at a consistent point in the program. In the general case, this requirement would overwhelm most programmers. However, in SPMD applications, there are natural places to take checkpoints, i.e. at the top or bottom of the main loop [2].

10.1.10.1 Example

To use the checkpoint library, users are responsible for the following tasks: (1) recovery of checkpoint data, (2) saving the checkpoint data, and (3) deciding how often to checkpoint.

The following example is included in the standard Legion distribution ($LEGION/src/ServiceObjects/MPI/ft_examples) and consists of tasks that perform some work and exchange boundary information at each iteration.

Example: Sample uses of the checkpoint library

//
// Save state. State consists of the iteration count.
//
do_checkpoint(int iteration) {
        // pack state into a buffer
                MPI_Pack(&iteration, 1, MPI_INT, user_buffer,
                     1000, &position, MPI_COMM_WORLD);
        
        // save buffer into checkpoint
                MPI_FT_Save(user_buffer, 20);

        // done with the checkpoint
                MPI_FT_Save_Done();
}


//
// State consists of the iteration count
//
do_recovery(int rank, int &start_iteration) {
        // Restore buffer from checkpoint
                size = MPI_FT_Restore(user_buffer, 1000);

        // Extract state from buffer
                MPI_Unpack(user_buffer, 64, &position,
                 &start_iteration, 1,
                MPI_INT, MPI_COMM_WORLD);
}

//
// C pseudo-code
// Simple SPMD example
//
// mpi_stencil_demo.c
//
main(int argc, char **argv) {
        // declaration of variables omitted...

        // MPI Initialization
        MPI_Init (&argc, &argv);
        MPI_Comm_rank (MPI_COMM_WORLD, &nodenum);
        MPI_Comm_size (MPI_COMM_WORLD, &nprocs);

        // Determine if we are in recovery mode
        recovery = MPI_FT_Init(nodenum, start_iteration);
        if (recovery)
            do_recovery(nodenum, start_iteration);
        else
            start_iteration = 0;

        for (iteration=start_iteration; 
             iteration < NUM_ITERATIONS; ++iteration) {

        exchange_boundary();

        // ...do some work here...

           // take a checkpoint every 10th iteration
           if (iteration%10==0)
                do_checkpoint(iteration);
        }
}

The first function called is MPI_FT_Init(). MPI_FT_init() initializes the checkpointing library and returns TRUE if the object is in recovery mode. If in recovery mode, we restore the last consistent checkpoint. Otherwise, we proceed normally and periodically take a checkpoint.

In do_checkpoint() we save state by packing variables into a buffer using MPI_Pack() and then calling MPI_FT_Save(). To save more than one data item, we can pack several items into a buffer (by repeatedly calling MPI_Pack) and then call MPI_FT_Save(). When the state to be saved is very large, we can break it down into smaller chunks and call MPI_FT_Save() for each chunk. When all data items for the checkpoints have been saved, we call MPI_FT_Save_Done().

In do_recovery() we recover the state. Note that the sequences of MPI_Unpack() and MPI_FT_Restore() must be mirror images of the sequences of MPI_Pack() and MPI_FT_Save() in do_checkpoint().

10.1.10.2 API (C & Fortran)

C API

Fortran API

Description

int MPI_FT_Init(int rank)

MPI_FT_INIT(RANK, RECOVERY)
INTEGER RANK, RECOVERY

Initializes the checkpointing library. Returns TRUE if in recovery mode

int MPI_FT_ON()

MPI_FT_ON(FT_IS_ON)
INTEGER FT_IS_ON

Returns TRUE if this application is running with checkpointing turned on.

void MPI_FT_SAVE(
char *buffer, int size)

MPI_FT_SAVE(
BUFFER, SIZE, IERR)
<type> BUFFER(*)
INTEGER SIZE, RET_SIZE

Saves checkpoint onto Checkpoint Server.

void MPI_FT_SAVE_DONE()

MPI_FT_SAVE_DONE(IERR)
INTEGER IERR

Done with this checkpoint.

int MPI_FT_RESTORE(
char *buffer, int size)

MPI_FT_RESTORE(
BUFFER, SIZE, RET_SIZE)
<type> BUFFER(*)
INTEGER SIZE, RET_SIZE

Restore data from current checkpoint.

10.1.10.3 Running the above example

First, run the legion_ft_initialize script on the command line. You should see the following output:

$ legion_ft_initialize
legion_ft_initialize: Could not find a CheckpointStorage object ...
Attempting to register one
Stateful class "/CheckpointStorage" does not exist - creating it.
Registering implementation for class "/CheckpointStorage"
legion_ft_initialize: success initializing FT SPMD Checkpointing support
$

Note that you only need to run this command once.

Compile the application and register the program with Legion:

$ cd $LEGION/src/ServiceObjects/MPI/ft_examples
$ make reg_stencil

Create an instance of the CheckpointStorage server and place it in context space:

$ legion_create_object /CheckpointStorage /home/username/ss

Run the application and specify a ping interval and reconfiguration time:

$ legion_mpi_run -n 2 -ft -s /home/username/ss -g 200 \
   -r 500 mpi/programs/mpi_stencil_demo

There is a set of special legion_mpi_run flags that can be used when running in a fault tolerant mode. These flags are used in specific combinations. You must use the -ft in all cases and you must use either -s or -R. The -g and -r flags are optional.

-ft

Turn on fault tolerance.

-s <checkpoint server>

Specifies the checkpoint server to use.

-g <pingInterval>

Ping interval. Default is 90 seconds.

-r <reconfigurationInterval>

When failure is detected (i.e. an object has not responded in the last <reconfigurationInterval> seconds) restart the application from the last consistent checkpoint. Default value is 360 seconds.

-R

Recovery mode.

10.1.10.4 Recovering from failure

If an object fails (i.e. has not responded in the last <reconfigur-ationInterval> seconds) the application automatically restarts itself.

10.1.10.5 Restarting application

Under catastrophic circumstances (e.g., prolonged network outages), the recovery protocol described above may not work. In this case users can restart (i.e., rerun) the entire application with the -R flag.3 Continuing the above example:

$ legion_mpi_run -n 2 -ft -R /home/username/ss -g 200 \ 
   -r 500 mpi/programs/mpi_stencil_demo

The application will restart from the last consistent checkpoint. Note that the number of processes and checkpoint server name must match that of the original run.

Once the application has successfully completed you should delete the Checkpoint Server by typing:

$ legion_rm -f /home/username/ss

If you would like to test the restart mechanism, you can simulate failure then restart the application. Kill the application by typing ^C (Control-C).

10.1.10.6 Compiling/makefile

To compile C applications with the checkpointing library, link your application with the -lLegionMPIFT -lLegionFT libraries. For Fortran applications, link with the -lLegionMPIFTF library also.

A sample makefile and application is provided in the standard Legion distribution package, in the $LEGION/src/ServiceObjects/MPI/ft_ex-amples directory.

10.1.10.7 Another example

For a more complicated example, please look at the application in $LEGION/src/Applications/ocean. This Fortran code contains the skeleton for an ocean modeling program.

10.1.10.8 Limitations

We use pings and timeout values to determine whether an object is alive or dead. If the timeout values are set too low, we may declare an object dead falsely. We recommend setting the ping interval and reconfiguration time to conservatively high values.

We do not handle file recovery. When running applications that write data to files, users are responsible for recovering the file pointer.

10.1.11 Functions supported

Please note that data packed/unpacked using the MPI_Pack and MPI_Unpack functions will not be properly converted in heterogeneous networks.

All MPI 1.1 functions are currently supported in Legion MPI. Some 2.0 functions are supported: please contact us at legion-help@virginia.edu for more information.

10.1.12 Running a Legion MPI code with the fewest changes

Please see the Legion web site's documentation (<legion.virginia.edu/documentation.html>) for an on-line tutorial called "Running an MPI code in Legion with the fewest changes." The tutorial shows how to adapt a sample MPI program to run in Legion. Please note, however, that while this "fewest changes" approach allows you to run the program and to transparently read and write files remotely, it does not support heterogeneous conversions of data. If you run this program on several machines which have different integer formats, such as Intel PCs (little-endian) and IBM RS/6000s (big-endian), using unformatted I/O will produce surprising results. If you want to use such a heterogeneous system, you will have to either use formatted I/O (all files are text) or use the "typed binary" I/O calls instead of Fortran READ and WRITE statements. These "typed binary" I/O calls are discussed in "Buffered I/O library, low impact interface" in the Legion Developer Manual.

10.2 Native MPI

Native MPI code is code written for an MPI implementation. Legion supports running native MPI programs without any changes. You only need the binary and a host to run it on. You can, if you wish, adapt your program to make Legion calls.

You will need a Legion host object with native MPI properties set to run these programs. Please see Setting up a native MPI host in the System Administrator manual for more information.

10.2.1 Task classes

Native MPI implementations generally require that MPI executables reside in a given place on disk. We have provided a tool, legion_native_mpi_register, to register executables of different architectures for use with native MPI.

10.2.2 Compilation

Native MPI code may be compiled independently of Legion, unless your code makes Legion calls (see Making Legion calls from native MPI programs). In that case, you must link your program to the Legion libraries. A sample makefile for this situation is below.

Figure 7: Sample makefile for native MPI code that makes Legion calls

CC      =       mpiCC
MPI_INC =  /usr/local/mpich/include

mpimixed:       mpimixed.c
        $(CC) -g -I$(MPI_INC) -I$(LEGION)/include -D$(LEGION_ARCH) -DGNU \
             $(LEGION)/lib/$(LEGION_ARCH)/$(CC)/libLegion1.$(LEGION_LIBRARY_VERSION).so \
             $(LEGION)/lib/$(LEGION_ARCH)/$(CC)/libLegion2.$(LEGION_LIBRARY_VERSION).so \
             $(LEGION)/lib/$(LEGION_ARCH)/$(CC)/libLegion1.$(LEGION_LIBRARY_VERSION).so \
             $(LEGION)/lib/$(LEGION_ARCH)/$(CC)/libBasicFiles.so $< -o $@

10.2.3 Register compiled tasks

Run legion_native_mpi_register. Usage of this tool is:

legion_native_mpi_register <class name>
    <binary path> <architecture>
    [-help]

The example below registers /myMPIprograms/charmm (the binary path) as using a Linux architecture.

$ legion_native_ mpi_register charmm \ 
    /myMPIprograms/charmm linux

You can run register a program multiple times, perhaps with different architectures or different platforms. If you have not registered this program before, this will create a context in the current context path (the context's name will be the program's class name -- in this case, charmm) and registers the name in Legion context space.

10.2.4 Running a native MPI application

MPI programs are started using the program legion_native_mpi_run. Usage is:

legion_native_mpi_run
    [-v] [-a <architecture>]
    [-h <host context path>]
    [-IN <local input file>]
    [-OUT <local result file>]
    [-in <Legion input file>]
    [-out <Legion result file>]
    [-n <nodes>] [-t <minutes>]
    [-legion] [-help]
    <program context path>
    [<arg 1> <arg 2> ... <argn>]

The following parameters are used for this command:

-h <host context path>

Specify the host on which the program will run. The default setting is the system's default placement, which is to pick a compatible host and try to run the object. If the host fails, the system will try another compatible host.

-v

Verbose option.

-a <architecture>

Specify the architecture on which to run.

-IN <local input file>

Specify an input file that should be copied to the remote run from the local file system.

-OUT <local result file>

Specify a result file that should be copied back from the remote run to the local file system.

-in <Legion input file>

Specify an input file that should be copied to the remote run from the Legion context space.

-out <Legion result file>

Specify a result file that should be copied out from the remote run into Legion context space.

-n <nodes>

Specify the number of nodes for the remote MPI run.

-t <minutes>

Specify the amount of time requested for the remote MPI run. This option is only meaningful if the host selected to run the remote program enforces time limits for jobs.

-legion

Indicate whether the application makes Legion calls (see section 10.2.5, below).

-help

Print command syntax and exit.

<arg1> <arg2> ... <argn>

Arguments to be passed to the remote MPI program.

You can examine the running objects of your application using

$ legion_ls program_name

This context will have an entry for each object in this MPI application.

If you need to kill the program and its implementations, run:

$ legion_rm program_name

10.2.5 Making Legion calls from native MPI programs

The -legion flag indicates that your MPI code makes Legion calls (e.g., BasicFile calls). If you do not use this flag and your code attempts to make Legion calls, your program may not run correctly. If your program makes Legion calls you must:

  • link your program with the Legion libraries (see Compilation)
  • specify the -legion flag when you run the code

10.2.6 Example

We have provided two sample native MPI programs, available in $LEGION/src/ServiceObjects/MPI/examples/. The first, nativempi.c, produces exactly the same output as the mpitest_c.c program discussed in Example. The second, mixedmpi.c, is the same code but with Legion calls.

Please note two important adaptations that were made to mixedmpi.c in order to access Legion files. There are two new include files:

#include "legion/Legion_libBasicFiles.h"
#include "legion/LegionNativeMPIUtils.h"

and a new function call:

LegionNativeMPI_init(&argc, &argv);

These lines must be added to a native MPI code that makes any kind of Legion call.

10.2.7 Scheduling native MPI processes

Legion does not schedule native MPI processes. When a program is run with legion_native_mpi_run, the local mpirun utility on the destination host schedules the processes.


1. If you use the -h flag in this way, be sure to double-check the order in which the host objects are listed in your context. Back

2. Support for generic communication patterns will be provided in a future release of Legion. Back

3. The restart mechanism can also be used to migrate applications by specifying a different set of hosts with the -h flag. Back

Directory of Legion 1.6.4 Manuals
[Home] [General] [Documentation] [Software]
[Testbeds] [Et Cetera] [Map/Search]

Free JavaScripts provided by The JavaScript Source

legion@Virginia.edu
http://legion.virginia.edu/