Australian BADG Testbed

Installation Guide

New node installation. Having trouble? Look at the following link.

Step 0 - Planning

You should decide how best to install you Grid software to utilise your existing resources. You will probably require a service for the submission of jobs (Globus gatekeeper), a service for publishing your resource information (GRIS - Grid Resource Information Service), and/or a service for access to disk/tape storage (GSIFTP). The usual set up is to have all 3 installed on the one host but this is not required. Whatever you decide, a GRIS is recommended on any host running a gatekeeper or GSIFTP service.

If you are intending to provide access to a job manager or scheduler for the submission of jobs a gatekeeper service is required. (Check Globus compatibility with job managers.) This must be installed on a node on which has access to the job manager and has the job manager's tools installed. Often this is installed on the job manager server.

You need to decide where to put your Globus software installation. If you are installing Grid software on a cluster, the best place is one that is accessible to all machines in the cluster via the same path. This saves having to install the client software and security settings on each machine.

Step 1 - Download

Log in to your installation host as some user. This user will be able to manage the Globus installation but does not have to be the root user. Create the installation path on this host:

    cd /pathto/
    mkdir -p globus2.0/packages
    ln -s globus2.0 globus
    cd globus/packages

A symbolic link may be created to help manage multiple Globus version in future. You should download all necessary binary or source packages into the packages directory, for safe keeping. For i686 Linux binary installations download:

For source installations download:

The globus_all-*-src.tar.gz are tar balls of individual packages and can not be installed using globus-build without unpacking. ???The two software development kit (SDK) bundles are required for building the replica catalog bundle.???

Step 2 - GPT (Globus Package Tool) Installation

The GPT tool is used to install and build Globus packages.

	### for CSH installation
    setenv GPT_LOCATION /pathto/globus
    setenv GLOBUS_LOCATION /pathto/globus
	### for SH installation
    export GPT_LOCATION=/pathto/globus
    export GLOBUS_LOCATION=/pathto/globus

    cd $GLOBUS_LOCATION/packages
    mkdir -p ../bin
    gunzip -c gpt-1.0.tar.gz | tar xvf -
    cd gpt-1.0
    ./build_gpt

Step 3.1 - Globus Installation (i686 Linux binaries)

This step is not required if you are installing from source packages, however, it is recommended if you have i686 Linux architecture.

The following commands will install the default client and server binaries, the default replica catalog binaries, and the BADG patches to these. You should explicity specify the full and primary hostname with the environment variable GLOBUS_HOSTNAME. Don't worry about running setup/globus/setup-gsi yet.


	### for CSH installation
    setenv PATH "${PATH}:${GPT_LOCATION}/sbin"
    setenv GLOBUS_HOSTNAME "hostname.domain"
	### for SH installation
    export PATH="${PATH}:${GPT_LOCATION}/sbin"
    export GLOBUS_HOSTNAME="hostname.domain"

    cd $GLOBUS_LOCATION/packages
    globus-install globus_all_bundle-server-linux-i686-gcc32.tar.gz
    globus-install globus_all_bundle-client-linux-i686-gcc32.tar.gz
    gpt-postinstall
    globus-install globus_replica_bundle-linux-i686-gcc32pthr.tar.gz
    globus-install --force globus_badg_patch-1.0-i686-pc-linux.tar.gz
    gpt-postinstall

After installing the binary BADG patch you will need to reconfigure MDS/GRIS. You should only do do this after a fresh installation as your GRIS settings will be reset.

    cd $GLOBUS_LOCATION/setup/globus/
    ./setup-globus-mds-common
    ./setup-globus-mds-gris
    ./setup-globus-gram-reporter

Step 3.2 - Globus Installation (from source)

This step is not required if you are installing i686 Linux binary packages.

???? Under Construction ???? Don't worry about running setup/globus/setup-gsi yet.

Step 4 - Globus Setup

It is recommended that you create a Globus environment setup script that can be sourced when you login or before running Globus commands.

    vi globus_cshrc_general
        >>>> create this new C/TC-Shell file
        #!/bin/csh -f
        setenv GLOBUS_LOCATION /pathto/globus
        setenv GLOBUS_INSTALL_PATH $GLOBUS_LOCATION
        setenv GPT_LOCATION ${GLOBUS_LOCATION}
        set path = ( $GLOBUS_LOCATION/bin $path )
        source $GLOBUS_LOCATION/etc/globus-user-env.csh

    vi globus_shrc_general
        >>>> create this new Bourne/Bash/K-shell file
        #!/bin/sh
        export GLOBUS_LOCATION=/pathto/globus
        export GLOBUS_INSTALL_PATH=$GLOBUS_LOCATION
        export GPT_LOCATION=$GLOBUS_LOCATION
        export PATH=$GLOBUS_LOCATION/bin:$PATH
        . $GLOBUS_LOCATION/etc/globus-user-env.sh

The Globus security setup must continue as the root user on the host that runs the Grid services.

    su root
    $GLOBUS_LOCATION/setup/globus/setup-gsi

If you wish to use the BADG certificate authority for user and host certificates you will need to modify the displayed values. If you wish to use the Globus certificate authority, leave the values as they are, type (q) then (Enter) to save and exit, then move on to the next step.

To use the BADG CA, type (1) then (Enter) to answer the "Base DN for user certificates" question, then (2) then (Enter) etc. Do not add any extra spaces to your answers.

    (1) Base DN for user certificates
           ou=People,o=BelleTestbed,o=Grid
    (2) Base DN for host certificates
           ou=Hosts,o=BelleTestbed,o=Grid
    (3) Certificate Authority name
           Melbourne EPP CA
    (4) Certificate Authority email address
           ra@epp-ca.ph.unimelb.edu.au

Then type (q) then (Enter) to save these changes. You should also make the BADG CA certificate available to Globus.

    cp $GLOBUS_LOCATION/setup/5e24fe33.*  \
       /etc/grid-security/certificates/

If the Globus installation directory is shared by other hosts you should copy these security settings to the following shared location. This should be done as the install user and NOT the root user.

    cp /etc/grid-security/globus-*-ssl.conf $GLOBUS_LOCATION/etc
    cp /etc/grid-security/grid-security.conf $GLOBUS_LOCATION/etc
    mkdir -p $GLOBUS_LOCATION/share/certificates
    cp /etc/grid-security/certificates/*  \
       $GLOBUS_LOCATION/share/certificates/

Step 5 - Certificate Applications

In order to start Grid services you will need to apply for a host certificate. To start the recommended MDS/GRIS service you will also need to apply for an LDAP service certificate. To test any of these services a user certificate will also be required.

NOTE: All certificate hostnames must be the primary/first name of that host as reflected in both local /etc/hosts files and in the DNS.

To apply for a host certificate for a gatekeeper or GSIFTP service type the following as root user on the service host:

    su root
    . /pathto/globus_shrc_general
    grid-cert-request -gatekeeper hostname.domain  \
       -key /etc/grid-security/hostkey.pem  \
       -cert /etc/grid-security/hostcert.pem  \
       -req /etc/grid-security/host.req

To apply for an MDS/GRIS certificate type the following as the installation user on the service host:

    source /pathto/globus_cshrc_general
    grid-cert-request -int -nopw \
       -cert $GLOBUS_LOCATION/etc/server.cert  \
       -key $GLOBUS_LOCATION/etc/server.key  \
       -req $GLOBUS_LOCATION/etc/server.request  \
       -dir $GLOBUS_LOCATION/etc

If you are using the Globus CA answer blank (default) for all but the "Name" question. If you are using the BADG CA answer the first two question blank (default), answer the third question "OrgUnit" with "Services". The "Name" question should always be answered with the full hostname.domain prefixed with "ldap/". Do not add any extra spaces to your answers.

    Level 0 Organization [Grid]:
    Level 1 Organization [BelleTestbed]:
    Level 0 Organizational Unit [People]:  Services
    Name (e.g., John M. Smith) []:  ldap/hostname.domain

To apply for a user certificate log in and type the following commands. Use the interactive option "-int" if you wish to specify a different fullname than the one associated with your account.

    source /pathto/globus_cshrc_general
    grid-cert-request

A user's certificate subject must be added to a servers /etc/grid-security/grid-mapfile before they can access services on that host. This subject is displayed as a result of the grid-cert-request process. Once you have a recieved your certificate from the certificate authority you may also retrieve this subject by the grid-cert-info command.

    grid-cert-info -subject
    su root
    vi /etc/grid-security/grid-mapfile
        >>>>> Add one line for each allowed user
        "Users Full Certificate Subject"             localusername
        "/O=Grid/O=BelleTestbed/OU=People/CN=Lyle Winton"   winton

For each certificate request you should follow the instructions on sending the request file to your certificate authority. (For some CA's you may need to send supporting documentation.) In a few days you will be notified or sent the new certificate. The following request files should result in certificates stored in the following locations:

request /etc/grid-security/host.req
certificate /etc/grid-security/hostcert.pem
request $GLOBUS_LOCATION/etc/server.request
certificate $GLOBUS_LOCATION/etc/server.cert
request ~/.globus/usercert_request.pem
certificate ~/.globus/usercert.pem

Each time you install a new host certificate you must reconfigure the Globus job manager settings. If you have an existing Globus installation and have removed the fork jobmanager or set another default jobmanager, this will reinstall fork as the default jobmanager.

    $GLOBUS_LOCATION/setup/globus/setup-globus-gram-job-manager

Step 6.1 - Testing the Globus Gatekeeper

The following will test the execution of a personal gatekeeper:

    source /pathto/globus_cshrc_general
    grid-proxy-init
    globus-personal-gatekeeper -start
       >>>>> Take the printed GRAM contact & paste into the following
    globus-job-run "[GRAM contact]" /bin/date
    globus-personal-gatekeeper -killall
    grid-proxy-destroy

Step 6.2 - Testing the GASS Service

The following will test the execution of a personal GASS service. A GASS service may be required by job submitted to your resource for the staging or serving of files.

    source /pathto/globus_cshrc_general
    grid-proxy-init
    globus-gass-server -r -p 46562 -c &
    globus-url-copy https://localhost:46562/etc/passwd  \
       file:///tmp/file1
    cat /tmp/file1
    rm /tmp/file1
    globus-gass-server-shutdown https://localhost:46562
    grid-proxy-destroy

Step 6.3 - Testing the GSIFTP Service

The following will test the execution of a personal GSIFTP service:

    source /pathto/globus_cshrc_general
    grid-proxy-init
    $GLOBUS_LOCATION/sbin/in.ftpd -s -p 2811 &
    globus-url-copy -s "`grid-cert-info -subject`"   \
       gsiftp://localhost/etc/passwd file:///tmp/file1
    cat /tmp/file1
    rm /tmp/file1
    kill %1
    grid-proxy-destroy

Step 6.4 - Testing the MDS/GRIS Service

The following will test the execution of a MDS service:

    source /pathto/globus_cshrc_general
    $GLOBUS_LOCATION/sbin/SXXgris start
    grid-info-search -x -h localhost
    $GLOBUS_LOCATION/sbin/SXXgris stop

You may need to ensure that the GRIS service has stopped completely. On some operating systems "SXXgris stop" appears not to work.

Step 7 - Installing Root Services

You may install any of the Grid services as root services that startup at system boot. You will find INIT.D scripts in the $GLOBUS_LOCATION/share/init.d directory. You will need to edit the GLOBUS_LOCATION line within these scripts to reflect your Globus install directory. Copy these to your system init directory (usually /etc/init.d) and link these to the approriate run levels. In Debian Linux you can do this by running the command:

    update-rc.d scriptname defaults 80 20

In Red Hat Linux the following command is used:

    /sbin/chkconfig --add scriptname

To start/stop these services you can run the following commands as root:

    /etc/init.d/scriptname start
    /etc/init.d/scriptname stop

You must also make sure that the host/service certificate and key files have the correct permissions. Certificates should be readable by everyone. Private key files should be owned and readable only by the service user (in this case root). By default this is not the case so you will need to run the following commands:

    chown root:root $GLOBUS_LOCATION/etc/server.key

It is also advisable to add the appropriate ports to /etc/services :

    vi /etc/services
        >>>>> add the following lines
        globus-gatekeeper  2119/tcp        # Globus Gatekeeper
        gsiftp             2811/tcp        # Globus GSIFTP

Step 8 - Configuring a Job Manager or Queuing System

You need to configure Globus to make your installed job manager or queuing system available as a Grid resource. As the installation user on the gatekeeper host follow the commands listed below. The first command gives you a list of compatible job managers that can be specified using the "-type=?" option.

    $GLOBUS_LOCATION/setup/globus/setup-globus-gram-job-manager -help
    $GLOBUS_LOCATION/setup/globus/setup-globus-gram-job-manager -type=pbs

The new job manager then needs to be registered with the MDS/GRIS. Restart the GRIS service once this is done.

    $GLOBUS_LOCATION/setup/globus/setup-globus-gram-reporter -help
    $GLOBUS_LOCATION/setup/globus/setup-globus-gram-reporter -type=pbs

To test this new resource you can query the status of the host and see records for the new job manager. You can also submit a test job as a user (provided they have a certificate and are in the /etc/grid-security/grid-mapfile). Remember to start the gatekeeper service if not already running.

    source /pathto/globus_cshrc_general
    grid-proxy-init
    grid-info-search -x -h hostname
    globusrun -a -r hostname/jobmanager-pbs
    globus-job-run hostname/jobmanager-pbs /bin/date
    grid-proxy-destroy

Step 9 - Additional Client Tools

There are several user tools that you may wish to install. The GSINCFTP package provides command line a command line GSIFTP tool. The GSI OpenSSH bundle provides secure shell server, a secure shell client utility (ssh), and secure copy (scp) with globus authentication. The Grid-RC-Tools provide command line navigation, querying, and management of the replica catalog. Download all required packages into the $GLOBUS_LOCATION/packages directory for safe keeping. For i686 Linux binary installations download:

For source installations download:

To install GSINCFTP from the i686 Linux binary package type the following commands as the installation user:

    source /pathto/globus_cshrc_general
    cd $GLOBUS_LOCATION/packages
    globus-install globus_gsincftp-0.2-i686-pc-linux-gnu-gcc32-pgm.tar.gz
    gpt-postinstall

To install GSINCFTP from source type the following:

    source /pathto/globus_cshrc_general
    cd $GLOBUS_LOCATION/packages
    globus-build -force gcc32 globus_gsincftp-0.2.tar.gz
    gpt-postinstall

To test the GSINCFTP installation run the tool as a user (provided they have a certificate and are in the /etc/grid-security/grid-mapfile). Remember to start the GSIFTP service if not already running it and you are connecting to your own host.

    source /pathto/globus_cshrc_general
    grid-proxy-init
    gsincftp hostname
    gsincftp gsiftp://hostname/
    gsincftpls gsiftp://hostname/mydir/
    grid-proxy-destroy

To install GSI OpenSSH from the i686 Linux binary package type the following commands as the installation user:

    source /pathto/globus_cshrc_general
    cd $GLOBUS_LOCATION/packages
    globus-install gsi_openssh_bundle-1.7-badg-i686-pc-linux-gnu-gcc32.tar.gz
    gpt-postinstall

Read the instructions that are displayed as a result of the gpt-postinstall. If you already have OpenSSH installed on your system there is probably little you need to do. For the GSI OpenSSH server you will find the INIT.D script in the $GLOBUS_LOCATION/share/init.d directory (grid-gsisshd). You will need to edit the GLOBUS_LOCATION line within this script to reflect your Globus install directory. Copy this file to your system init directory (usually /etc/init.d) and link these to the approriate run levels (see previous step 7).

To test the GSI OpenSSH installation run the tool as a user (provided they have a certificate and are in the /etc/grid-security/grid-mapfile). Remember to start the service if not already running it and you are connecting to your own host.

    source /pathto/globus_cshrc_general
    grid-proxy-init
    gsissh hostname
    grid-proxy-destroy

To install Grid-RC-Tools type the following commands as the installation user:

    source /pathto/globus_cshrc_general
    cd $GLOBUS_LOCATION/packages
    globus-install grid_rc_tools-1.0-noflavor-pgm.tar.gz
    gpt-postinstall

You may wish to configure a default replica catalog by setting a new variable in the Globus environment scripts. The environment variable GLOBUS_REPLICA_CATALOG_HOST can hold an LDAP URI which points to your default replica catalog. Individual users can also configure an environment variable GLOBUS_REPLICA_CATALOG_MANAGER to hold their username for replica catalog management (format is an LDAP BindDN).

    vi /pathto/globus_cshrc_general
       >>>>> add new line
       setenv GLOBUS_REPLICA_CATALOG_HOST  \
         "ldap://belle-rc.ph.unimelb.edu.au/rc=users,ou=Catalogs,o=Belle,o=Grid"
    vi /pathto/globus_shrc_general
       >>>>> add new line
       export GLOBUS_REPLICA_CATALOG_HOST=\
         "ldap://belle-rc.ph.unimelb.edu.au/rc=users,ou=Catalogs,o=Belle,o=Grid"

To test the Grid-RC-Tools installation run the following commands:

    source /pathto/globus_cshrc_general
    grc-ls -l
    grc-cd
    grc-cd somedir
    grc-ls -o

Step 10 - BADG Virtual Organisation Setup

To allow BADG Testbed users to access your Grid resources you must configure your host to query the BADG virtual organisation server. Download the BADG Testbed VO Setup package into the $GLOBUS_LOCATION/packages directory for safe keeping.

badgtest_vo_setup-1.0-noflavor-pgm.tar.gz

To install the BADG VO Setup type the following commands on the Grid services host as the installation user:

    source /pathto/globus_cshrc_general
    cd $GLOBUS_LOCATION/packages
    globus-install badgtest_vo_setup-1.0-noflavor-pgm.tar.gz
    gpt-postinstall

You will then need to complete the setup as the root user on the service host.

    su root
    . /pathto/globus_shrc_general
    $GLOBUS_LOCATION/setup/globus/badgtest-vo-setup

This final step has installed the GenGridMap utility into the /etc/grid-security directory. This utility accesses a remote virtual organisation (VO) server and constructs an appropriate /etc/grid-security/grid-mapfile , allowing users from the VO to access local service. The GenGridMap configuration file gengridmap.conf determines the address of the queried VO servers, the inclusion or exclusion of users and groups, and the local usernames used for VO users. By default, your orginal user list is copied to local-grid-mapfile which contains an additional list of allowed users, and all memebers of the BADG VO are allowed access as the local user bellegrid. You will need to create this local username or change the configuration file to point to an appropriate username.

The GenGridMap configuration and connection to the BADG VO can be tested via the following commands:

    /etc/grid-security/gengridmap -t

The grid-mapfile (and hence Grid user permissions) can be updated manually as the root user on the service host:

    /etc/grid-security/gengridmap

You should configure your system to update the grid-mapfile regularly with a CRON table entry. The following CRON table entry attempts to update the grid-mapfile every 15 minutes. The entry (non-commented line) should be all on one line.

    vi /etc/crontab
       >>>> add the following lines, the comment is optional
       #M          H  DM MON DW USER     COMMAND
       0,15,30,45  *  *  *   *  root     /etc/grid-security/gengridmap

Created: 14 September 2007
Last modified: 22 August 2003
Maintained by: Dr Lyle Winton winton#physics.unimelb.edu.au