ARSC HPC Users' Newsletter 397, November 7, 2008

See us at SC08!

ARSC will be at SC08, the 2008 Supercomputing conference, at the Austin Convention Center. Visit us November 17 - 20 at the University of Alaska Fairbanks booth 202, or at our kiosk in the DoD HPCMP booth 603. This year we spotlight the three competitively chosen HPCMP Capability Applications Project (CAP) users, who will be pioneer users of Pingo, as well as scientific activities in weather, climate change and our ongoing work in multi-core and accelerator evaluations.

Finding Optimization Options

[By: Don Bahls]

Both the PGI and PathScale compilers have "fast" compiler flags that enable a number of other compiler flags which generally provide good performance. Occasionally one of these flags might actually hinder code performance. Fortunately both PGI and PathScale have compiler flag combinations which will show the underlying flags that are enabled by the "fast" flags.

PGI

The "-fast" and "-fastsse" compiler flags can be used with the PGI compilers to enable compiler flags that generally provide good performance. By default, this doesn't show all of the underlying compiler flags that are used, however if the "-help" flag is added those flags will be displayed.

e.g.

mg56 % pgf90 -help -fast
Reading rcfile
/import/local/pkg/pgi/pgi-7.1.6/linux86-64/7.1-6/bin/.pgf90rc -fast        
Common optimizations; includes -O2 -Munroll=c:1 -Mnoframe -Mlre
-Mautoinline == -Mvect=sse -Mscalarsse -Mcache_align -Mflushz
...
...

For more information on available PGI compiler options see:

  • man pgcc
  • man pgCC
  • man pgf90

PathScale

The "-Ofast" compiler flag can be used with PathScale compilers to enable flags which generally aid performance. The "-show" flag will display the underlying compiler flags that are used by the PathScale compilers: e.g.

mg56 % pathf90 -show -Ofast
/import/local/pkg/pathscale/pathscale-3.1/lib/3.1/ipa_link \
...
-INTERNAL:lang=F90 -INTERNAL:return_val=on -INTERNAL:mldid_mstid=on \
-INTERNAL:return_info=on -OPT:ffast_math=ON -ipa \
...
  • man pathcc
  • man pathCC
  • man pathf90
  • man eko

Managing Multiple Programming Environment Modules

[By: Ed Kornkven]

(Or, One PrgEnv Module is Good, Two Can Be a Mess)

I may use programming environment modules differently than many of our users in that it is not uncommon for me to build a program using multiple compilers or versions of a compiler. As users of the ARSC systems know, ARSC uses the "modules" facility to bundle together all the path settings, environment variable settings and module dependencies that comprise the "programming environment" provided by a compiler version. So, all a user needs to do to compile with the PathScale 3.2 compiler for example, is


    module load PrgEnv.path-3.2

We could have said


    module load PrgEnv.pgi-7.2.2

to build using version 7.2.2 of the PGI compiler instead. If the PathScale environment is already loaded, we should first unload the PathScale module, then load the PGI module, or simply use the "switch" option to "module" to do the load and unload in one step:


    module switch PrgEnv.path-3.2 PrgEnv.pgi-7.2.2

It is a good idea to unload the previous module before loading the next one because unloading will make sure that environment variables and paths for the previous module aren't laying around with the potential for causing confusion between compilers. Having the correct module loaded means that when we compile using the "mpif90" wrapper, for example, we will use the intended compiler. Being a lazy typist, I use shell aliases to switch modules, for example:


    module_pgi

which is a Korn shell alias that is defined as


   alias module_pgi=\
   'module switch $(module -t list 2>&1 
 grep PrgEnv) PrgEnv.pgi-7.2.2'

This alias works like this: The $(module -t list) displays the modules we currently have loaded in a "terse" format, which means one module per line, just the name. The $(grep) will then pick out the module that contains the string "PrgEnv", which is the module that manages the version of the compiler we are using, by ARSC convention. If we have PrgEnv.path-3.2 loaded, this will return that name. That name is the first parameter to the "module switch" command, the second being PrgEnv.pgi-7.2.2, the module I want to switch to. The effect is to execute the "module switch" command shown earlier. I have aliases for switching to all the Midnight compilers using this basic scheme.

To this point, I've talked a bit about the ARSC PrgEnv modules and how to use them to easily change compilers. A bigger problem for me is remembering to change modules. My default PrgEnv, set in my .profile, is set to PrgEnv.path-3.2. If I am working on a project that needs the PGI compilers, when I logout and return to that work, I have to remember to switch modules before compiling. Note that this is especially an issue when building with the MPI-wrapped compilers (e.g., "mpif90"). If one compiles with mpif90, a Makefile designed for PGI might very well work without complaint when run with the PathScale programming environment. We could build with the "wrong" compiler and not even realize it. So I want my Makefile to check to make sure the right PrgEnv module is loaded. If the proper module is not loaded, the make should abort with an error message.

To do that, we use the "-show" option of mpif90 (or mpicc). That option displays the actual compiler commands that are executed "under the covers" of the mpif90 wrapper script. In the Makefile we scan the output of "mpif90 -show" for the compiler we are expecting. If we don't find it, the make fails. For example, suppose we want to build with PGI. Here is a simple Makefile that uses this method to do the module check:


    #############################################################
    #
    # Build a.out, making sure we are using the PGI compiler.
    #
    #############################################################
    
    SHELL = /bin/ksh
    
    all: confirm_compiler clean a.out 
    
    confirm_compiler:
            mpif90 -show 
 grep pgf90 > /dev/null 2>&1
    
    a.out: main.o
            mpif90 main.o
    
    main.o: main.f90
            mpif90 -c main.f90
    
    clean:
            rm -f *.o
    
    #############################################################

If we build with the PGI compiler loaded, the make works as usual. If a different module is installed, we get an error:


    Midnight> make
    mpif90 -show 
 grep pgf90 > /dev/null 2>&1
    make: *** [confirm_compiler] Error 1

We could set up a similar check for the correct version of the compiler using the "-V" compiler option. The Makefile above is a little extra paranoid, forcing a "make clean" for every build. The "clean" target removes all the object files, which will cause the recompilation of every file. That approach does negate one of the main benefits of make, rebuilding only as much as necessary. But that might be much less painful than getting object files from different compilers inadvertently linked together. If you don't prefer that behavior, just remove the target "clean" from the list of dependences of "all". Hopefully these examples will get your creative juices flowing. Please send in other ideas, or improvements to the ones given here.

Quick-Tip Q & A


A:[[ I've seen shell gurus moving around the command line with amazing
  [[ ease and grace.  What's your favorite shell short-cut?  What shell
  [[ do you use and do you use vi or emacs bindings?

#
# Thanks to Greg Newby for sharing his favorites!
#

I use ksh, bash, tcsh and zsh regularly.  When I can use any shell,
I often choose zsh, which combines many features of the others.

In zsh, if you want to look through all files & subdirectories,
no matter how many, you can do something like this (double asterisk
syntax):   "grep hello **/*c" This searches for "hello" in all files
ending in c, recursively.

The short-cut I use the most, which works in all the shells I use except
ksh, is a variable for the last argument to the previous command line.
The variable name is bang-dollar: !$

I use this very commonly when operating on a file.  For example,
this sequence of commands:

 ls -l testfile.txt    (get a long directory listing)
 wc -l !$              (how many lines in testfile.txt?)
 vi !$                 (edit it)
 scp !$ midnight.arsc.edu:  (copy it somewhere else)

Note that this feature requires command history to be enabled.

#
# Here a few short-cuts from co-editor Don Bahls
#

Using emacs bindings with tcsh or bash there are a few time saving
keystrokes I use a lot.  Type Ctr-p or Ctrl-n to go backward/forward
through the command history.  Then, for a command:

 Ctrl+a:  jump to the beginning of the line
 Ctrl+e:  jump to the end of a line
 Ctrl+d:  delete a character after the current cursor position
 Ctrl+r:  search backward through history for a pattern
 Esc+d: delete a complete word 


#
# Co-editor Ed Kornkven uses vi bindings with ksh
#

Type the ESC key to enter command history mode, then k or j to go
backward/forward through the command history.  Equivalent keystrokes to
Don's emacs bindings are:

 0:  jump to the beginning of the line
 $:  jump to the end of a line
 x:  delete a character after the current cursor position
 /:  search backward through history for a pattern
 dw: delete a complete word 



Q: Is there a command, like "seq," but which increments or decrements
   dates in the standard, YYYYMMDD format?  My wish would be for
   something like this:

     computer:~ % for dd in $(seq --yyyymmdd  20081029 20081102)
     > do
     >   echo /var/local/log/somelog.${dd}
     > done
     /var/local/log/somelog.20081029
     /var/local/log/somelog.20081030
     /var/local/log/somelog.20081031
     /var/local/log/somelog.20081101
     /var/local/log/somelog.20081102

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top