3.0 Starting a new system
A summary of the start-up procedure is on page 28.
3.1 Before you start
Before you start a new Legion system, consider what type of set-up will best suit your needs. Primary considerations include:
- What kind of system do you need? How many machines do you anticipate using? Do you only use local hosts or do you also use remote machines? Possible configurations might include:
-
a. A single host system: one Legion machine with one or more host objects (Figure 1). This is the simplest system.
b. A multihost system: multiple Legion hosts linked together and sharing local resources (Figure 2).
This can include homogeneous or heterogeneous platforms, as well as non-Legion machines. The machines do not need to be in physical proximity.
Figure 2: Multihost system
c. A multidomain system: multiple Legion domains connected together and sharing each others' resources (Figure 3).
Figure 3: Multidomain system
- Will you be using Legion security? This is an important consideration, since a secure Legion system cannot be cleanly shut down and restarted. If you are using security, you should decide who will be the "admin," the system administrator. The system should be started up by the admin, since he or she will have special privileges on the core objects created in the new system. (These privileges do not extend to other users' objects, however.)
- What kind of host objects will you be using? Options include:
- a. Basic host object: it resides on its host and manages and guards its host's resources. This is the template for the other host objects. See Host and vault objects in the Basic User Manual for information on basic host objects. In this manual, see section 11.0 for a discussion of host-vault pairings and adding new hosts.
b. PCD host object: it resides on its host, manages and guards its host's resources, and uses a process control daemon to regulate ownership of all Legion processes executed on that host. If you use a PCD host as your bootstrap host, the start-up process will be slightly different. For more information, please see Process control daemon host objects.
The daemon requires root privileges to start and to run. The PCD host object is useful if outside users will be running processes on your host, but can only be used if Legion security is enabled. Each user's processes will be tracked and accounted for.
c. Batch queue host object: it resides on its host, manages and guards its host's resources, and submits Legion jobs to the local queueing system.
This is the best choice for hosts that use a queue management system, although the PCD host object is more secure and has better accounting. For more information, please see Batch queue host objects.
d. Virtual host object: it resides on a different host, represents and guards its host's resources, but does not run normal Legion objects. A virtual host cannot be used as a bootstrap host: it is added to an already running system.
A virtual host object is used for running Legion jobs on unsupported platforms. The host object resides on a supported platform and runs native jobs with standard Legion tools on the target host machine. It can be used for scheduling, resource selection, and transparent execution on the target machine. For more information, please see Virtual hosts.
- a. Basic host object: it resides on its host and manages and guards its host's resources. This is the template for the other host objects. See Host and vault objects in the Basic User Manual for information on basic host objects. In this manual, see section 11.0 for a discussion of host-vault pairings and adding new hosts.
3.2 Set up the environment
A properly set-up environment is crucial for working in the Legion system. The start-up process uses certain Legion-specific environment variables, which must be correctly set before starting applications and running command-line utility programs. You must set these variables each time you starting working in Legion. Without a properly set environment, programs cannot communicate with other objects in the system, and the program may terminate with an error, never return a value, or fail in a more spectacular fashion. If this occurs, try setting your environment properly and starting over.
You must have /bin/ksh installed in your system. There are a number of Legion scripts that will look for ksh, and if it is not installed in your system you will get error messages.
If you have not yet done so, set $OPENSSL_INC and $OPENSSL_LIB (see page 10). You must also set $LEGION_HOME and $LEGION_OPR and run the legion_profile.[c]sh script.1 The environment must be properly set in each shell in which you plan to run Legion commands. Check to be sure that environment variables are properly set.
export LEGION_HOME=<Legion root dir path> export LEGION_OPR=<Legion OPR root dir path> export OPENSSL_INC=<OpenSSL installation directory>/include export OPENSSL_LIB=<OpenSSL installation directory>/lib . $LEGION_HOME/legion_profile.sh
setenv LEGION_HOME <Legion root dir path> setenv LEGION_OPR <Legion OPR root dir path> setenv OPENSSL_INC <OpenSSL installation directory>/include setenv OPENSSL_LIB <OpenSSL installation directory>/lib source $LEGION_HOME/legion_profile.csh
We suggest $LEGION_HOME/../OPR for the OPR root directory path)
3.3 Starting a single host system
You must have the Legion module on your bootstrap host in order to start a new system. Once you've untarred it (see page 10) you'll need to configure, start, and then initialize the new system. You'll use three commands:
legion_setup_state
legion_startup
legion_initialize
3.3.1 Choose a bootstrap host
The bootstrap host is where you start and shut down your system.2 It must be able to hold:
- The LegionClass object (the root of the Legion binding mechanism and the parent class of many metaclasses), and
- the $LEGION_OPR/LegionClass.config file.
This file will be created on this host and will contain the LegionClass object address, which must be globally known to all Legion objects. The file must be available when other objects are started in the system.
If you choose a PCD host as your bootstrap, the start-up procedure is slightly different than for a basic host object.
3.3.2 Set up and configure
You must first set up the initial state for core Legion system objects. Legion system objects are persistent, and can save and restore their own state. Some of these objects must have their state initialized before they run for the first time. After the initial start-up, these objects will manage their own state and configuration, if the system is properly maintained.
to configure the system. This program will return your start-up host name, a port number for the LegionClass object, and a time. If you do not want to use the default settings, use the -i flag to run the command in an interactive mode. Your output will look something like this:
$ legion_setup_state Creating OPR directory /home/xxx/OPR/. Saving LegionClass configuration file: /home/xxx/OPR/LegionClass.config LegionClass host name = your.startup.host.name LegionClass port number = 7899 LegionClass timestamp = 898198093 $
The script creates the $LEGION_OPR directory and several sub-directories, populating them with initial states for several core system objects. The timestamp sets the starting time for the system: Legion objects use a timestamp to guarantee each object's unique identity. The current time is measured in seconds since January 1, 1970.
3.3.3 Start up
The legion_startup script provides prompts asking whether or not to start each component. It's best to answer "yes to all" (Y). The verbose option allows you to see more detailed information, as the script works, about debugging. (This can be large amounts of information, so use this option only if you are searching for a problem.)
To start the main core system objects, enter
$ legion_startup
Legion will start several classes on your host. The output shows major class objects starting up.
$ legion_startup Starting meta-class object: LegionClass Continue (startup) (y=yes, Y=yes to all, n=no, N=no to all, v=verbose, V=verbose all)? Y Starting meta-class object: BootstrapMetaClass Starting class object: DefaultBindingAgentClass Starting class object: CommandLineClass Starting class object: UnixHostClass Starting class object: UnixVaultClass Starting class object: DefaultImplementationClass Starting class object: DefaultImplementationCacheClass Starting class object: DefaultContextClass (SKCC enabled) Done with DefaultContextClass Legion first-time system startup complete $
The first object created, LegionClass, is the highest-level metaclass (the meta-metaclass) and the parent of every other object in the system. The next object, BootstrapMetaClass, is the class object for bootstrap class objects (i.e., class objects whose instances must be created in the initialization phase).
The bootstrap class objects are:
UnixVaultClass
UnixHostClass
UnixImplementationClass
UnixImplementationCacheClass
These are started up a bit further down, as the output shows.
The next new classes also start instances in the new system but are not bootstrap class objects: BindingAgentClass parents binding objects, CommandLineClass parents command-line objects, etc.
3.3.4 Initialize
Like legion_startup, the legion_initialize script will provide prompts asking whether or not to perform each task, and it is generally best to use the "yes to all" option (Y). To initialize Legion, enter:
$ legion_initialize
The output shows the system creating and tagging the key ingredients of a new system. It is too long to reproduce in full here, but we'll look at some selected actions.
Creating host object BootstrapHostObject on "your.current.host.name" Continue (y=yes, Y=yes to all, n=no, N=no to all, v=verbose, V=verbose all)? Y Configuring wellknown binaries for host "1.01.07.0100..."
The first line shows the system creating a bootstrap host object on your current host (a host object manages a host, so the bootstrap host object manages the bootstrap host).
Creating vault object BootstrapVaultObject on "your.bootstrap.host.name" Setting BootstrapHost and BootstrapVault restrictions Added 1 host(s) to vault's compatibility set Added 1 vault(s) to host's compatibility set
A bootstrap vault object is automatically created on your current host (a vault object manages a vault, which stores Legion object's permanent states). This guarantees that the bootstrap host object has a compatible vault object. All host objects must be paired with at least one compatible vault object (i.e., a vault that it can "see").3
Creating an ImplementationCache Creating an implementation (ContextObject) for ContextClass Creating the root context object
Implementation objects represent and manage the implementation cache (used to allow Legion processes to take place in different architectures) and context space.
Adding "BootstrapHost" to the hosts context Adding the alias "your.bootstrap.host.name" for BootstrapHost to the hosts context Adding "BootstrapVault" to the vaults context
More implementation objects are created as the process creates new object classes.
Two context names are added to the /hosts context, BootstrapHost and your.bootstrap.host.name. Both names refer to the Bootstrap host object but only one is added to the /vaults context (Figure 4).
Figure 4: Context paths for the bootstrap host and vault objects  |
This object will manage a portion of the persistent storage mechanism for the Bootstrap host. The /impls context contains names of the default implementation objects (see Implementation model). These can all be viewed with context-related commands or the GUI once the system has been completely started.
Before finishing, legion_initialize generates a set-up script for the new system. This script is placed in the $LEGION_OPR directory. Please see section 3.5 for more information on using this script.
The basic Legion system is now ready to go. If you wish to start security, follow the steps in section 3.3.5, below. If you have other modules, follow the steps in section 3.3.6.
3.3.5 Set security
If you wish enable Legion security, run the legion_init_security command. You'll have to decide now whether or not you want security, since the command will not run properly unless you run it immediately after initializing the system. If you don't wish to use it, just skip over this section. However, none of your processes will be protected and you won't be able to create individual accounts.
$ legion_init_security
Please note that this command is for the Legion module and will not work with all classes in the other modules. For best results, start each module's security when you initialize it (see section 3.3.6).
Several events take place when you run this command.
$ legion_init_security Creating the context "/users" to contain user-objects Creating the initial system-admin user object, "/users/admin" Please select a Legion password for "/users/admin": New Legion password: xxxx Retype password: xxxx 1.399b330d.6f000000.01000000.000001fc0cd... Please enter the Legion password for "/users/admin" to continue: Enter Password: xxxx You have successfully logged in.
First, Legion creates a /users context. This context contains all Legion user ids. Since you need a user id to work in a secure system, you are automatically assigned a system administrator user id called admin. Anyone logged in as admin has root privileges in the system and can create new users, modify security settings, etc. The admin user also has ownership of all existing objects in the new system but not any future objects that other users create.
You must create a password for admin. You'll be asked to enter it three times during the legion_init_security process.
Once you're logged in, Legion gives you ownership of all existing objects in the system.
Changing ownership of all objects to "/users/admin" 1.3622260c.01..000001fc0cbe1846763f895a... 1.3622260c.02..000001fc0b3b16eb8b2dde29... [...etc.] Changed ownership of 63 objects.
After this point, any new objects created will belong to whoever created them.
Legion then configures security for the new system's resources. It creates access control lists (ACLs) for all existing core classes and their instances.
Configuring security for the default collection Creating initial ACLs files for all core objects in /home/spw4s/OPR/init_acls Creating ACL for /class/AuthenticationObjectClass class Creating ACL for /class/BasicFileClass class Creating ACL for /class/BasicSchedulerClass class Creating ACL for /class/BatchQueueMetaClass class [...etc.] Creating ACLs for instances of /class/BasicSchedulerClass Creating ACL for 1.399b330d.68000000.01000000.000001fc0b3 da560eff6580b840f3e7a76b4c82beb9b421ce47ff465557c914bc4bb 3ba85140b3444091bdf45dca6e50deac309b02d420b631b886619ea27 6de1372260b Creating ACLs for instances of /class/BatchQueueMetaClass Creating ACL for 1.399b330d.73000000..000001fc0cd9a1202af c0753365c8441c69ffcbcd9ccd235c5c72603707b855aaf543dc66327 31d93228618948049c13dba35de9727993c1e4abe7467c232cb16c05c 831 ... already done Creating ACLs for instances of /class/BootstrapMetaClass Creating ACL for 1.399b330d.04..000001fc0e608816a569dbc1 d0503434eedcd9d7cbb97112871ff09e32482308466f094531d7b332 007536be821a598bb4aa4cbdbd9731592bdc06167c028403f5ab8945 ... already done [...etc.]
The access control lists (ACLs) protect objects against unauthorized use. Only an object's creator can use the object, unless the creator specifies otherwise. The initial ACL files allow only the admin to use the core objects.
Setting ACL for /class/AuthenticationObjectClass class Setting ACL for /class/BasicFileClass class Setting ACL for /class/BasicSchedulerClass class [...etc.] All acls set. You have successfully logged out. $
When all necessary ACLs have been set, you are logged out.
At this point, security has been enabled and is running. You must now log in as /users/admin.
Legion's security is now enabled. For more information about security see About Legion security; and Using security features.
3.3.6 Starting up other packages
The steps outlined in 3.3.2 - 3.3.5 initialized only the Legion package. If you are using any others, you must initialize them by hand by running the appropriate legion_init_<module name> tool.
If have run legion_init_security, Legion security will automatically be enabled in each package when you initialize it.
Due to internal dependencies, you need to initialize in a specific order: HPC, Extra, and Apps. So, if you have all of the packages, you would run the following tools, in this order:
3.4 Starting a multihost system
This is a two-part process. First, you have to have a running single host system, as laid out in section 3.3. Second, you add new host objects on the desired machines. Since you will be starting processes on the target hosts from the bootstrap host be sure that you can run rsh/ssh on the bootstrap host as well as on the target hosts from the bootstrap host without having to enter a password. You can set up a .rhosts file for rsh or an authorized_keys files for ssh to accomplish this (see the rsh and ssh man pages for more information).
3.4.1 Set up
You'll need to set the proper environment variable on the bootstrap host and the remote host(s) so that you can run Legion commands on a remote host using rsh or ssh.
LEGION_RSH=<rsh|ssh> LEGION_RCP=<rcp|scp> export LEGION_RSH LEGION_RCP
setenv LEGION_RSH <rsh|ssh> setenv LEGION_RCP <rcp|scp>
Set these variables on the bootstrap host before you start the new system (i.e., before you run legion_startup). Please note that you only need to follow these steps on the bootstrap host: you will need to install the Legion binaries on any other machines that you add to your system, but you do not need to start more Legion systems.
To add additional hosts and users, copy the $LEGION_OPR/setup.[sh|csh] scripts to a globally accessible location. Hosts can share an NFS-mounted Legion tree,4 but for best results you should place the OPRs on a local disk.
3.4.2 Create a new vault
If the new host will not be compatible with your existing vaults, create a new vault object with the legion_startvault command.
$legion_startvault /hosts/Bootstrap
There are several flags that you can use to set $LEGION, $LEGION_OPR, architectures, etc. Please see page 43 in the Reference Manual for more information about this command.
See page 56 for more information on new vaults.
3.4.3 Create a new host
Use the legion_starthost command to create a new host object on the desired host. For example, to start a new host object on MyNewHost, you would enter:
The same command, with -B, will start a new PCD host object.
$ legion_starthost -B PCDUnixHost \ MyNewPCDHost.DNS.name /vaults/BootstrapVault \ hosts/myNewPCDHost
See also page 59 for more information on adding PCD hosts.
The same flag can be used to start a new batch queue host object. You'll need to update the host object's attributes to include the queue type.
$ legion_starthost -B BatchQueueHost \ MyNewBQHost.DNS.name /vaults/BootstrapVault \ hosts/myNewBQHost $ legion_update_attributes /hosts/myNewBQHost \ -a "host_queue_type(`<Queue_type>')"
See page 66 for more information on adding batch queue hosts.
There are several other flags and options with this command described on page 42 in the Reference Manual. See page 52 in this manual for more information on new hosts.
If the new host has a new architecture, you now need to add implementations of the core objects for the new architecture. Log in to the new machine and run the following:
$ source <path_to_globally_visable_setup_script>/setup.[sh|csh] $ legion_login /users/admin
$ legion_init_arch
(Run legion_login only if you have enabled Legion security.)
Repeat these steps for each additional host. We suggest that you customize these steps and write a script to simplify the process, especially if you need to bring up a big net.
3.4.4 Adding new users to a secure net
If you have not initialized security, there are no user accounts.
Only admin can add users to a secure net. Run the legion_create_user command with the new user's name. We suggest that you put all users in the /users context. I.e.,
$ legion_create_user /users/<new user name>
New user accounts are available immediately after creation.
If you are working on a PCD host, follow up with these steps:
|
3.4.5 Working in the new net
Users can work in an insecure net by entering:
$ source <path_to_globally_visible_setup_script>/setup.[sh|csh]
Users can work in a secure net by entering:
Please note that neither of these procedures will open a separate shell.
3.5 Making a set-up script for users
We strongly suggest that Legion system administrators use a set-up script for users to source when starting work in Legion. In version 1.8 and forward, Legion will automatically generate a setup script (called setup.[c]sh) in your $LEGION_OPR directory when you run legion_initialize. It contains information about your Legion environment variables and should be run before you start working in a new shell. You can edit the script as necessary and distribute it to other users in your system.
You can also run the legion_make_setup_script command to generate a set-up script. The usage is:
legion_make_setup_script
[-o <script basename>]
[-OPR <OPR dir name>]
[-L <$LEGION dir name>]
[-debug] [-help]
1. Bourne Shell is not directly supported by our implementation of Legion, due to the use of alias to implement some Legion commands. Bash, however, is supported.
2. Once fully operational, Legion does not automatically shut down and restart: the system is intended to stay up.
3. For more about hosts and vaults, see section 6.0 in the Basic User Manual. For more about host-vault pairs, see page 49 in this manual.
4. See rpc.lmountd and rpc.lnfsd in the Reference Manual on page 109 for more information about NFS-mounted Legion systems.
[Home] [General] [Documentation] [Software] [Testbeds] [Et Cetera] [Map/Search]
|
legion@Virginia.edu
http://legion.virginia.edu/