ARSC T3D Users' Newsletter 98, August 2, 1996

Porting T3D PVM Codes to Network PVM

[ Don Morton, visiting ARSC from Carmeron University, contributes this article. See his contact information, provided below. ]

Last week, Richard Barrett of LANL discussed numerous issues in porting Network PVM codes to the T3D. I have been interested in this area, and in the reverse scenario - writing T3D codes so that they'll easily port to a network PVM environment. Clearly, the issues mentioned by Richard, such as sending messages to TID's rather than PE's, are important when writing portable code. I have found that the most difficult task in writing portable PVM code is that of dealing with "spawning" vs. "non-spawning" environments. Although Richard touched upon this, I'd like to go a little deeper.

If we are writing code for the T3D, we are restricted to a Single-Program Multiple-Data (SPMD) paradigm - that is, we have the same executable running on each processor. We tend to address other processes by a PE number, or at the very least, the PVM TID of a PE. We don't have an inherent master-slave relationship between processes, although traditionally we view PE0 as a "master" process. If we want to run our T3D SPMD code in a network PVM environment, where spawning is necessary, then it's necessary to simulate the T3D behavior of a non-spawning environment. The following function is designed to do just that. It has been utilized in a T3D environment AND in a cluster of PC's running Linux.

First, the function prototype (in Fortran) looks like


       subroutine startup(numprocs, mype, tidlist, icode)

       integer numprocs
       integer mype
       integer tidlist(0:numprocs-1)
       integer icode
The behavior of the function is as follows:
  1. 
       We assume that the "master" process (the initial process in a
       network environment, PE0 on the T3D) somehow knows how many
       processes are to run.  This might be read in from a file, come
       from a command-line argument, etc.  The "master" process regards
       "numprocs" as an (in) parameter.  All other processes regard
       this as an (out) parameter.  Additionally, "mype", "tidlist",
       and "icode" are all (out) parameters.
    
  2. 
       From each process' point of view, startup() is called with
       uninitialized parameters (with the exception of "numprocs" on
       the master process), and upon return from the function, every
       process has a value for each of the parameters.  Thus, each
       process knows how many total processes are in the virtual
       machine, they each know their logical position in the system,
       and they each have an identical list of TID's, so they can
        easily communicate with each other by referencing the
       appropriate logical PE in "tidlist" - if Process 3 wants to
       communicate with Process 7, it simply references tidlist(7).
    
  3. 
       "icode" is an error flag.  If the call to startup() was
       successful, icode stores the number of total processes,
       otherwise it stores some negative value (note - this aspect
       hasn't been fully implemented in the following code).
    

The code (see below), "startup.F", is preprocessed with gpp on the T3D and cpp in other environments, and is conditionally compiled depending on values of preprocessing macro _CRAYMPP. _CRAYMPP is automatically defined if you're using the Cray MPP compiler.

The entire function is based on processes joining a global group, then obtaining information about themselves and other processes within the global group. Conditional compilation enables/disables code for spawning and T3D environments. The following differences are addressed through conditional compilation:

  1. 
       The global group name, "ALLGROUPNAME."
    
  2. 
       Variables declared for use in spawning environments are not used
       in the T3D environment.
    
  3. 
       In T3D environments, we're not able to join the global group 
       (via pvmfjoingroup()), or obtain the instance within the global
       group (via pvmfgetinst()).  Thus, the T3D-specific function,
       pvmfgetpe() is used.  In spawning environments, we join the
       global group and get our instance in the group with
       pvmfjoingroup().
    
  4. 
       A good portion of the code in the function is only used in a
       spawning environment.  After the spawn, we force a
       synchronization by having child processes send a message to the
       parent.  Without this forced synchronization, future group
       operations (e.g.  pvmfbcast(), pvmfbarrier(), pvmfgettid()) may
       fail, as the parent process may attempt to call these functions
       before others have even joined the group.  Trust me, I've
       encountered this way too many times!
    

After the spawning code, there is no difference in T3D and spawning implementations. The rest of the code is centered on insuring that every process knows how many total processes there are, then each process obtains the TID's of every other process. One might be tempted to use pvmfgsize() to find the number of total processes, but, as above, this "forced" synchronization is necessary. Without it, it is possible for one process to reach this point before others have joined the global group, producing different values of "numprocs" for different processes! Again, trust me! Been there, done that!

The source code for the function is below, followed by the Makefile, which in turn is followed by "test.f," a simple program which calls the startup() subroutine. Note that in the main program there is no dependence on the architecture. This is all handled in startup(), and as long as basic PVM operations are utilized, it will be portable.


ccccccccccccccccccccccccc   startup.F  ccccccccccccccccccccccccccccccccc

#ifdef _CRAYMPP
#define ALLGROUPNAME PVMALL
#else
#define ALLGROUPNAME "alltasks" 
#endif

      subroutine startup(numprocs,
     &                   mype,
     &                   tidlist,
     &                   icode)

      implicit none

ccc  Header files
      include 'fpvm3.h'

ccc-----------------------------------------------------------
ccc   Arguments

      integer numprocs               ! number of processes to start up 
                                     ! (if spawning).
      integer mype                   ! My process number ( 0 < p < numprocs-1))
      integer tidlist(0:numprocs-1)  ! List of PVM task id's
      integer icode                  ! Return code

         ! If successful - number of processes running.

ccc-----------------------------------------------------------

ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
c                                                                     c
c  In PVM environments, we assume that the master process (rank 0)    c
c  passes 'numprocs' in as an argument.  On completion, all processes c
c  will have 'numprocs' as an (out) argument, storing the number of   c
c  processes running.                                                 c
c                                                                     c
ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc


     
ccc-----------------------------------------------------------
ccc   Local variables
      integer info                ! Return code for PVM calls 
      integer i
      integer mytid               ! My PVM TID

#ifndef _CRAYMPP 
ccc   declare variables for spawning environment
      character*255 binaryname    ! stores name of executable, retrieved
                                  ! from arg 0 of command line
      integer numspawned          ! number of tasks spawned
      integer myparent            ! PVM TID of parent
#endif

      integer MSG_NUMPROCS_BCAST
      integer MSG_ACK
      parameter(MSG_NUMPROCS_BCAST=888, MSG_ACK=333)
ccc-----------------------------------------------------------

ccc   Enroll in PVM
      call pvmfmytid(mytid)

ccc   CRAY MPP does not allow us to join global group (or use it in
ccc   pvmfgetinst(), so just get PE number
#ifdef _CRAYMPP
      call pvmfgetpe(mytid, mype)
#else
      call pvmfjoingroup(ALLGROUPNAME, mype)
#endif


#ifndef _CRAYMPP 
ccc   This code executes in a "spawning" environment
      if(numprocs .gt. 1) then
         if(mype .eq. 0) then           ! I'm the parent task
c           get name of executable (myself)
            call getarg(0, binaryname)
         
c           spawn children
            call pvmfspawn(binaryname, PVMDEFAULT, '*', 
     &          numprocs-1, tidlist(1), numspawned)
            if(numspawned .ne. numprocs-1) then
               write(0,*) 'pvmfspawn() was unsuccessful -- '
               write(0,*) '   ', numspawned, ' tasks were spawned'
               write(0,*) '   Task error codes:'
               do i=1,numprocs-1
                  write(0,*) '        ', tidlist(i) 
               enddo
               call pvmfhalt()
            endif
 
ccc         Now, wait for each child to acknowledge that they've made it
ccc         this far in the code - this will allow us to conclude that
ccc         everyone has joined group ALLGROUPNAME.
            do i=1,numprocs-1
               call pvmfrecv(-1, MSG_ACK, info)
            enddo
         else     ! I am one of the newly spawned tasks - ack parent
ccc         First, find out who spawned me
            call pvmfparent(myparent)

            call pvmfinitsend(PVMDATADEFAULT, info) 

ccc         Just pack any arbitrary integer
            call pvmfpack(INTEGER4, MSG_ACK, 1, 1, info)
            call pvmfsend(myparent, MSG_ACK, info)
         endif
      endif
ccc   End of code for spawning environments
#endif

ccc   At this point, it is assumed that all processes are running - Process 0
ccc   should now let all the other processes know how many of them exist
      if(mype .eq. 0) then
         call pvmfinitsend(PVMDATADEFAULT, info)
         call pvmfpack(INTEGER4, numprocs, 1, 1, info)
         call pvmfbcast(ALLGROUPNAME, MSG_NUMPROCS_BCAST, info)
      else
         call pvmfrecv(-1, MSG_NUMPROCS_BCAST, info)
         call pvmfunpack(INTEGER4, numprocs, 1, 1, info)
      endif

ccc   At this point, everyone should has a value for numprocs

ccc   Set up global data structures which will allow us to utilize
ccc   TID's.
      do i=0,numprocs-1
         call pvmfgettid(ALLGROUPNAME, i, tidlist(i)) 
      enddo

ccc   Everyone sync here 
      call pvmfbarrier(ALLGROUPNAME, numprocs, info)

ccc   Finally, give icode a value - if all goes well, should be
ccc   same value as numprocs
      icode = numprocs

      return
      end
=======================================================================


###################   Makefile  ############################

#########  CRAY T3D  at ARSC  ###############
#SYS_INC=/usr/include/mpp 
#FPP=/mpp/bin/gpp
#FPPFLAGS=-FP 
#FC=f90 
#FFLAGS=-dp  
#LD=f90
#LDFLAGS=
#DISPOSITION=
#########################################################


#########  LINUX with g77  ###########
SYS_INC=/usr/local/pvm3/include
FPP=/lib/cpp
FPPFLAGS=-P -traditional 
FC=g77 
FFLAGS=-m486 -Wall  
LD=g77
LDFLAGS=-L./ -L/usr/local/pvm3/lib/LINUX -lfpvm3 -lgpvm3 -lpvm3 
DISPOSITION=cp runit ${HOME}/pvm3/bin/LINUX/
############################################################
########  end of site-peculiar definitions  ################


INCLUDES=-I${SYS_INC}

runit : test.o startup.o
        ${LD} -o runit test.o startup.o ${LDFLAGS}
        ${DISPOSITION}

test.o : test.f
        ${FC} ${FFLAGS} -c test.f

startup.o : startup.f
        ${FC} ${FFLAGS} ${INCLUDES} -c startup.f
        rm -f startup.f

startup.f : startup.F
        ${FPP} ${FPPFLAGS} startup.F > startup.f


=================================================================


      program test

      integer np, mype, tids(0:127), ierr 

      np = 4
      call startup(np, mype, tids, ierr)

      print *, 'mype is ', mype, ' of ', np,
     &         ' processes.  My tid is ', tids(mype)

      end

=======================================================================

    Don Morton                              Email:  morton@arsc.edu  


    Visiting Scientist  (summer)            Voice: (907) 474-5507    


    Arctic Region Supercomputing Center     Fax  : (907) 450-8601    


    University of Alaska                                             


    Fairbanks, AK 99775                                              


    http://grizzly.cameron.edu/morton.html                           

=======================================================================

ARSC T3D User Group Met on August 1st

We had a nice turnout yesterday, despite the rain. Users and staff alike are both excited and apprehensive about the T3E. Excited? Of course. This is new technology that promises to solve bigger problems faster. Apprehensive? Of course. This is new technology that promises startup problems, incompatibilities, and a learning curve.

Some Concerns:

  • How easily will T3D codes port to the T3E?
  • Will there be a T3E version of AVS?
  • Will the T3E have a C compiler, or only C++? Users report difficulty compiling public domain and personal C code on various C++ compilers.
  • Will existing CRAFT codes port to the T3E? How small a subset is planned? What other implicit programming models will be available?
  • What will the file system look like? DMF? CRL?

Some Expectations:

  • Improved job scheduling and more immediate access to PEs may be possible, given the T3E's job swapping capability and its lifting of the power-of-two number of PEs per job restriction.
  • Turnaround on the Y-MP should improve once separated from the T3D.
  • Many users see the T3E as an major upgrade: they are already looking forward to running bigger simulations.
  • Everyone looks forward to faster processors, better interconnect bandwidth, and better I/O bandwidth.

Don Morton Migrates South

What's the mileage to Lawton, OK from Fairbanks, AK? (How 'bout route, bandwidth, and latency?)

At any rate, Don plans to start the long drive tomorrow morning. At ARSC, Don's departure is an early sign of approaching winter -- next thing you know, the sandhill cranes will start to fly. We thank Don for his many contributions to ARSC, the T3D User Group, and this Newsletter. We wish him a safe journey, a good year, and look forward to seeing him again next summer.


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top