8.0 Executing programs remotely

Remote execution allows you to run remotely executed programs from the Legion command line, taking advantage of Legion's distributed resources. Your workload can be spread out on several processors, improving job turn-around time and performance. You can execute multiple instances of a single program or execute several different programs in parallel.

A remote program is an executable process that can be started with a command-line utility. Note that a remote program is not part of the Legion system, but is an independent program that may or may not be linked to Legion. Remote programs might be shell scripts, compilers, existing executables, etc. The term "remote" here means that the program is not or may not be executed on the local host.

8.1 Linked and independent programs

Legion permits two kinds of programs to be run remotely: Legion-linked and independent. A Legion-linked program is linked to the Legion libraries. An independent program is not linked to the Legion libraries. For example, a shell script or a program written and compiled on non-Legion systems are independent programs.

8.2 Registering independent programs

Before either a linked or independent remote program can be run from Legion it must be registered . The legion_register_runnable command registers Legion-linked programs. The legion_register_program command registers independent programs. Both of these commands require parameters containing a context path name, the program's binary path name, and the binary's architecture. You can register a program multiple times to run on different architectures.

The legion_run command runs registered programs.

8.2.1 Independent programs

Independent remote programs are registered with legion_register_program. This includes information about the program's architecture (e.g., linux, sgi, solaris, etc.), its binary path name, and a program class. Once the program has been registered, it can be executed with legion_run.

Usage is:

legion_register_program
<program class>
<Unix executable path>
<legion arch> [-debug] [-help]

The <program class> parameter is a context path name for a class that will manage the program's Legion objects. The path name can be whatever you find most convenient. You might want to use the program's name so that it will be easier to remember (e.g., if I'm registering the program MyProgram I will use MyProgram as the <program class> parameter). The context path name can be either new or a previously created path name (if you are registering multiple versions of the same program). It will refer to a (new or previously created) Legion object that permits an independent remote program to execute.

If multiple programs are registered with the same program class and architecture, the most recently registered version will be used when the program is run on that particular architecture. Be sure to reregister the program if you recompile it: Legion makes a copy of your executable for context space only when you register it and will not update its copy when you update the original.

An example of this command might look like this:

$ legion_register_program myProgram /bin/myProgram linux
Program class "myProgram" does not exist.
Creating class "myProgram".
Registering implementation for class "myProgram"
$

This output shows Legion creating the program class myProgram, as the command requested. If this class had been previously created, the output would look like this:

$ legion_register_program myProgram /bin/myProgram linux
Registering implementation for class "myProgram"
$

8.2.2 Legion-linked programs

The legion_register_runnable command registers programs that are linked to the Legion libraries, and exports a runnable object interface. The command creates an interface between the Legion system and the program. Like legion_register_program, this command requires the remote program's program class, executable binary path, and the binary's architecture as parameters.

Its usage is:

legion_register_runnable
<program class> <executable path>
<legion arch> [debug] [-help]

The <program class> parameter is a context path name for the Legion objects that will handle a particular program. You can choose a context path that suits your organizational scheme. If you reregister a program and use the same program class name, Legion will use the most recently registration information when running the program. Once the program has been registered, you can run it with legion_run. You must reregister the program if you recompile it (Legion copies your executable to context space when you register it, so the most recently registered copy will be run).

$ legion_register_runnable myProgram /bin/myProgram linux
Program class "myProgram" does not exist.
Creating class "myProgram".
Registering implementation for class "myProgram"
$

This output shows Legion creating the program class myProgram, as the command requested. If this class had been previously created, Legion will simply register the new information:

$ legion_register_runnable myProgram /bin/myProgram linux
Registering implementation for class "myProgram"
$

8.3 Running a program remotely

Use legion_run to start a single instance of a program on a remote host. If you are running a serial program with many input files and/or multiple executions you may prefer to use the legion_run_multi command (see page 104 in the Reference Manual). The program must be registered with either legion_register_program (for independent programs not linked to the Legion libraries) or legion_register_runnable (for Legion-linked programs) before you run it.

There are a number of optional parameters associated with legion_run. To try to keep things simple, we will not discuss all of them here: please see page 100 for the complete syntax and explanation of all options.

The legion_probe_run command is a useful aid when remote programs. It lets you pass input files to the remote host after the program has started running, pick up output files, check the job's status, and clean up the remote host after the job has finished.

8.3.1 Choosing a remote host

Legion will choose a host with an appropriate architecture if you do not specify one with the -h flag. The host must be part of your system (i.e., have a host object and be listed in the /hosts context: if necessary, see "Adding a new host" in the System Administrator Manual).

8.3.2 Command-line arguments

If your program requires command-line arguments, you must include them as the final parameters to legion_run. They can not be put in an option file.

8.3.3 Getting input files to the remote host

Your program may expect to find certain files in its local directory at run time, so when you execute it on a remote host you must pass copies of any expected input files to that host. If you know the name of the remote host, you can pass the files by hand, but otherwise you'll find it much easier to tell Legion the name of the files that need to be copied. You can do this before or after the program starts.

8.3.3.1 Before the program starts

You can use the -in and -IN flags to move files onto the remote host before the program starts. The -in flag gets files from context space and the -IN file gets them from the local host. Or, use legion_probe_run to move files to the remote host after the program starts.

8.3.3.2 After the program has started

Use legion_probe_run to move input files to the remote host after the program has started running (see section 8.3.8). If you wish to use this method, please note that you must have started the program with a probe file (see section 8.3.6).

You can use the -in or -IN flags to pass input files from context space or local file space to the remote job's current working directory. These flags are identical to legion_run's -in and-IN flags, described above.

8.3.4 Getting output files from the remote host

Once the program has finished, you need to retrieve the output files. If you know the name of the remote host and you are running in nonblocking mode you can get the files by hand (see section 8.3.7). Otherwise, you will need tell Legion the names of the output files that you wish to retrieve and whether you wish to put them on your local host or in context space. You can do this before or after the program starts.

8.3.4.1 Before the program starts

You can use legion_run's -out and -OUT flags to specify which files you want copied from the remote host. The -out flag copies files into context space and the -OUT file copies them onto your local host.

If the program has terminated because of a crash, Legion may not be able to copy all of the results.

Into context space

• The -out flag tells Legion to copy the contents of a specified output file from the program's current working directory (on the remote host) into a new Legion file object in your context space once the program is terminated. The new file object will have the same name as the specified output file. E.g., if you tell Legion to copy outputFoo:

-output outputFoo 

• Legion will look for a file called outputFoo in the remote directory and copy its contents into a new file object in your context space called outputFoo.

Onto your local host

• The -OUT flag tells Legion to copy the remote output file into a new file on your local host. The new file will have the same name as remote output file.

8.3.4.2 After the program has started

You can use the -out and -OUT flags with legion_probe_run to specify which files you want copied from the remote host after the program has started running. If you wish to use this method, please note that you must have started the program with a probe file (see section 8.3.8).

The -out flag copies files into context space. The -OUT file copies them onto your local host. These flags are identical to legion_run's -out and-OUT flags.

8.3.5 Option file

If you find that you are trying to keep track of too many options when you start legion_run, you may wish to use an option file. This is a text file that holds a list of all of your flags and optional settings for that run. The file can be delimited with tabs, spaces, or new lines and can contain any of the legion_run flags except for the program class name and any command-line arguments: those must appear on the command line, along with the -f flag and the option file name. Please note that you can only use legion_run flags in this file.

8.3.6 Creating and using a probe file

A probe file is a Unix file on your local machine. The legion_probe_run command uses it to contact a remote job. To create one, simply use the -p flag when you start legion_run.

If you run in blocking mode, legion_run automatically will remove a job's probe file when the job is finished. If you run in nonblocking mode, legion_probe_run's -kill option will delete it as part of its cleaning-up operation. Otherwise, the file will remain in your local file space.The probe file is good for only one remote job, so if you reuse the name Legion will simply write over the previous file if it still exists.

8.3.7 Blocking vs. nonblocking

You can run the legion_run command in blocking or nonblocking mode.

The default mode is blocking. This means that legion_run will continue to run on your command line until the remote job has finished. All output files will be collected from the remote host, the remote directory that was used to run the job will be cleaned up, and the command will exit.

Blocking

The basic steps in running a blocking remote job are shown below in Figure 6.

Figure 6: A blocking remote run.

In this example, a user starts a remote job on his local host in blocking mode. The following events occur:

1. The legion_run tool sends a copy of local input files (marked with the -IN flag) to the remote host.
2. The legion_run asks the remote host to start the job. It then blocks and waits for the job to finish.
3. The remote job gets a copy of context input files (marked with the -in flag).
4. The job starts running.
5. The remote job finishes and creates its output files. Any files that were marked with the -out flag are copied into context space.
6. The remote job tells legion_run that it is finished. legion_run copies any files marked with the -OUT flag into the user's local file space and cleans up the remote job's working directory.

You can use ^C to kill the job prematurely.

Nonblocking

If you start legion_run with the -nonblock flag, on the other hand, the command will start the program on the remote host, wait ten seconds, verify that the program can start, and then die. When the remote host has finished executing the program, the job will copy any files named with the -out flag into context space but ignore any files named with the -OUT flag. The job's working directory will remain on the remote host. The basic steps are shown below in Figure 7.

Figure 7: A nonblocking remote run.

In this example, a user starts a remote job on his local host using the -nonblock flag. The following events occur:

1. The legion_run tool sends a copy of local input files (marked with the -IN flag) to the remote host.
2. The legion_run asks the remote host to start the job. It waits ten second, verifies that the job can start and exits.
3. The remote job gets a copy of context input files (marked with the -in flag).
4. The job starts running.
5. The remote job finishes and creates its output files. Any files that were marked with the -out flag are copied into context space.

Since this is nonblocking mode, the remote job's working directory does not get cleaned up. The remote host will hold on to the job's working directory for six hours. During this period, the user can remove any remaining output files by hand. If he started a probe file, he can remove copy output files to his local host with legion_probe_run's -OUT flag.¹ If the user does not take clean up the remote host in that period, the entire directory will be tarred and moved into the user's context scratch space (see section 8.3.9).

8.3.8 About legion_probe_run

The legion_probe_run command checks a remote job that was started with legion_run. You must know the name of the job's probe file (see above). You can use this command to pass input files, pick up output files, find what host is running your program, check the job's status, see what files are in the job's current working directory, kill the job, and clean up the remote host when the job is finished. An example is shown in Figure 8, below.

Figure 8: A nonblocking remote run with legion_probe_run.

In this example, a user starts a job using -nonblock to run it in nonblocking mode and -p to create a probe file. The following events occur:

1. The legion_run tool sends a copy of local input files (marked with the -IN flag) to the remote host
2. The legion_run asks the remote host to start the job.
3. It also creates and stores a probe file on the local host. It then waits ten second, verifies that the job can start and exits.
4. The remote job gets a copy of context input files (marked with the -in flag).
5. The job starts running.
6. The remote job finishes and creates its output files. Any files that were marked with the -out flag are copied into context space.
7. The user runs legion_probe_run with the -statjob flag to check on the job. The tool looks for the probe file in its local execution environment.
8. The legion_probe_run tool contacts the remote job and sees that it is finished.
9. The user runs legion_probe_run with -OUT flag, telling Legion which files he wants copied onto his local host. (In this scenario it makes no difference whether or not the user ran legion_run with -OUT flags: if legion_run is set to nonblocking it will not stick around to move files back to the local host.)

The user can then run legion_probe_run with the -kill flag. This destroys the remote job's working directory and the probe file. Please note that if you use this flag before the job is finished you will terminate the job and lose all data.

Please see page 96 in the Reference Manual for more information about this command.

8.3.9 Context scratch space

Legion uses context space as backup storage (scratch space) for any remote jobs that finish and are not cleaned up by legion_run or legion_probe_run. If the job finishes and is not picked up or checked for six hours, the remote host will tar, compress, and move the job's working directory into your context scratch space. The default scratch space is /tmp but you can set it to anywhere in your context space.² There are several ways to do this.

1. Set the LEGION_SCRATCH_CONTEXT variable from the command line:

$ LEGION_SCRATCH_CONTEXT=/home/myContext/scratch/

2. Use -setscratch or -D when you run legion_run.
3. Use -setscratch when you run legion_probe_run.

If you set it multiple times, the most recent setting will be used.

8.3.10 Retrieving files from context scratch space

If your remote job's working directory was sent to context scratch space, you'll need to copy it to your local host then uncompress and untar it in order to use it. First, run legion_cp:

$ legion_cp -localdest <tar file> <local_file_name.tar.Z>

Be sure to give the local file name a *tar.Z suffix. You then need to run uncompress and tar:³

$ uncompress <local_file_name.tar.Z>
$ tar xvf <local_file_name.tar>

8.4 Example

Suppose that you have a program called Doris that you wish to run on a remote linux host. The first step is to determine whether it is linked to the Legion libraries (Legion-linked) or not (independent). In this case, we'll suppose that Doris is not linked to the Legion libraries.

1. The first step is to register the program with Legion. Since it's independent, use legion_register_program. To keep things simple, call the program class doris.

   $ legion_register_program doris /bin/Doris linux
   Program class "doris" does not exist.
   Creating class "doris".
   legion_update_attributes: Added 1 attributes(s) to object
   legion_update_attributes: Added 1 attributes(s) to object
   Registering implementation for class "doris"
   $

2. If you haven't already set a tty object for your shell, do so now.
3. You can now run the program. You have two options: to run it as in blocking mode (the default setting) or in nonblocking mode. Doris only takes a few minutes to run, so there's no reason not to use the default setting. If it took a long time to run and you wanted to free up the command line, you might choose to run in nonblocking mode. You should, though, make a probe file called dorisProbe to check on Doris's progress.

   Furthermore, Doris expects two input files, input1 and input2, and produces two output files, output1 and output2, so you need to pass their names to Legion when you start legion_run.

   $ legion_run -IN input1 -IN input2 -OUT output1 \
      -OUT output2 -p dorisProbe doris 

4. You didn't ask for a specific host, but you can get the remote host's name and the remote job's working directory with legion_probe_run.
5. You didn't ask for a specific host, but you can get the remote host's name and the remote job's working directory with legion_probe_run.

   $ legion_probe_run -hostname -pwd -p dorisProbe
   gander.cs.virginia.edu
   /myDir/OPR/BootstrapVaultOPR/ReservedOPR-1.80
   $

   Note that you can use legion_run's -h and -d flags to specify a remote host and directory when you start the program.

6. Once the program finishes, legion_run copies the -OUT files to the local host, deletes the probe file and cleans up the remote working directory. If you had run this in nonblocking mode you would have to pick up the -OUT files and then clean up, either with legion_probe_run or by hand.

8.5 Converting a C/C++ program

Any C or C++ program can be made into a Legion runnable object using the following steps:

1. The program should export a C-linkable legion_main function in place of its main function.
2. The program should be linked to the Legion libraries. Add the following to your link line:

1. If you run legion_probe_run on a terminated job, the clock will restart and give you another six hours.

2. You may need a different scratch space if security is enabled. You may not have permission to work in the /tmp context or your system administrator may have removed the context altogether.

3. These are common Unix tools that should be available on all Unix platforms. If you do not have access to these tools, please contact us at legion-help@virginia.edu.

Table of Contents 1.0 Introduction    1.1 About this manual    1.2 Style conventions    1.3 About Legion 2.0 Setting up and logging in    2.1 Preparing your Legion environment    2.2 Logging in       2.2.1 Logging in as a user       2.2.2 Changing your password       2.2.3 About object permissions       2.2.4 Checking your log in status       2.2.5 Logging out       2.2.6 Using Legion in a Kerberos environment    2.3 Changing your profile 3.0 An introduction to context space 4.0 Context space    4.1 Legion object names    4.2 About the LOID    4.3 Organizing context space 5.0 Working in context space    5.1 Viewing contexts       5.1.1 Read a context's contents       5.1.2 Look up LOIDs       5.1.3 Create a new context       5.1.4 Changing your current context       5.1.5 Check your current context    5.2 Naming objects       5.2.1 Assign a context name to a LOID       5.2.2 Give an object extra context names       5.2.3 Change an object's context path       5.2.4 List context names       5.2.5 Remove names and objects       5.2.6 Remove an entire context       5.2.7 Use the same name in different contexts    5.3 Moving between local files and contexts       5.3.1 Copying          5.3.1.1 Copy a local file to Legion          5.3.1.2 Copy from Legion to a local file          5.3.1.3 Copy from Legion to another Legion file object          5.3.1.4 Using wildcards when copying       5.3.2 Look at a file object's contents       5.3.3 Importing a local Unix tree       5.3.4 File sharing: temporarily link files to Legion 6.0 Host and vault objects    6.1 Host/vault versus host/vault object    6.2 About the bootstrap host/vault    6.3 Creating objects on new hosts    6.4 Look up an object's host    6.5 Instance placement on hosts and vaults    6.6 Backup vaults       6.6.1 WORM objects       6.6.2 Assigning and synchronizing backup vaults       6.6.3 Using replication 7.0 Running a Legion application 8.0 Executing programs remotely    8.1 Linked and independent programs    8.2 Registering independent programs       8.2.1 Independent programs       8.2.2 Legion-linked programs    8.3 Running a program remotely       8.3.1 Choosing a remote host       8.3.2 Command-line arguments       8.3.3 Getting input files to the remote host          8.3.3.1 Before the program starts          8.3.3.2 After the program has started       8.3.4 Getting output files from the remote host          8.3.4.1 Before the program starts          8.3.4.2 After the program has started       8.3.5 Option file       8.3.6 Creating and using a probe file       8.3.7 Blocking vs. nonblocking       8.3.8 About legion_probe_run       8.3.9 Context scratch space       8.3.10 Retrieving files from context scratch space    8.4 Example    8.5 Converting a C/C++ program 9.0 PVM    9.1 Core PVM interface    9.2 Tids &apm; LOIDs    9.3 Task classes       9.3.1 Legion-PVM library    9.4 Compilation    9.5 Registering compiled tasks    9.6 Examples    9.7 Running PVM code with the fewest changes 10.0 MPI    10.1 Legion MPI       10.1.1 Task classes       10.1.2 Legion MPI libraries       10.1.3 Compilation       10.1.4 Register compiled tasks       10.1.5 Running the MPI application       10.1.6 Example       10.1.7 Input and output files          10.1.7.1 Input and output flags          10.1.7.2 Subroutines       10.1.8 Scheduling MPI processes       10.1.9 Debugging support       10.1.10 Checkpointing support          10.1.10.1 Example          10.1.10.2 API (C & Fortran)          10.1.10.3 Running the above example          10.1.10.4 Recovering from failure          10.1.10.5 Restarting application          10.1.10.6 Compiling/makefile          10.1.10.7 Limitations          10.1.11 Functions supported          10.1.12 Running a Legion MPI code with the fewest changes    10.2 Native MPI       10.2.1 Task classes       10.2.2 Compilation       10.2.3 Register compiled tasks       10.2.4 Running a native MPI application       10.2.5 Making Legion calls from native MPI programs       10.2.6 Example       10.2.7 Scheduling native MPI processes 11.0 Executing remote programs    11.1 Sample record and replay A-1 Sample makefile A-2 About Legion tty objects    A-2.1 Simple tty management    A-2.2 Complex tty management A-3 Alphabetical list of Legion commands A-4 Subject listing of Legion commands    A-4.1 Calls on objects    A-4.2 Calls on class objects    A-4.3 Calls on LegionClass    A-4.4 Calls on file and context objects    A-4.5 Start-up and shutdown functions    A-4.6 Scheduling support    A-4.7 General functions about the state of the system    A-4.8 Security    A-4.9 Application development    A-4.10 Program support Getting help References Index

Directory of Legion 1.8 Manuals [Home] [General] [Documentation] [Software] [Testbeds] [Et Cetera] [Map/Search] Free JavaScripts provided by The JavaScript Source

legion@Virginia.edu
http://legion.virginia.edu/