[gmx-users] Scaling on SGI Altix 3700 Bx2

[Haruna Cofer]

Hello, Dallas --

I will let the developers advise on how to best scale your particular
problem (perhaps more details about it are needed).  I only know about
the -sort, -shuffle, and -load options myself.  But I just wanted to
make sure you knew that Erik Lindahl has graciously implemented IA64
assembly for the upcoming 3.3 release, and I have found this to improve
the performance on our SGI Altix by about 1.6x.  So in your quest for
better performance, you might want to check out the CVS version.  It's
very easy to download and build, and attached are my porting notes for
it. 

-- Haruna  :)

=====

Porting Notes for GROMACS 3.3 Beta to SGI Altix
-----------------------------------------------

1. Download GROMACS 3.3 Beta source code using CVS (hit return when 
   prompted for a password):

   % cvs -z3 -d :pserver:anoncvs at cvs.gromacs.org:/home/gmx/cvs login
   % cvs -z3 -d :pserver:anoncvs at cvs.gromacs.org:/home/gmx/cvs co gmx
   
   Go to the gmx/ directory and build the configure scripts:
   
   % cd gmx
   % ./bootstrap

2. Download FFTW 2.1.5 (not 3.0.1) source code via anonymous ftp:

   % ftp ftp.fftw.org
   ftp> cd pub/fftw
   ftp> get fftw-2.1.5.tar.gz
   ftp> quit

  or via your web browser:

   ftp://ftp.fftw.org/pub/fftw/fftw-2.1.5.tar.gz

  Uncompress and unarchive the distribution, and it will be
  put in a directory called fftw-2.1.5.  For example:

   uncompress fftw-2.1.5.tar.gz
   tar xvf fftw-2.1.5.tar

  NOTE: We have to use the older FFTW 2.1.5 instead of the latest 
  FFTW 3.0.1 in order to build GROMACS, because FFTW 3.0.1 does
  not yet support MPI.

3. Download an updated "depcomp" script that contains support for
the Intel 7.x compilers and higher:

https://ccvs.cvshome.org/source/browse/ccvs/Attic/depcomp?logsort=date&search=&hideattic=1&sortby=log&hidecvsroot=1&diff_format=h&r1=1.1

Shift-click or Right-click the link called "download" to download the
latest revision of the depcomp script.

4. Make sure you have the Intel 7.x or 8.x compilers installed.  
For best performance, also make sure you have SGI's Message Passing
Toolkit (MPT) installed (any release should be fine).  

5. Set up your compiler environment to use an Intel compiler.  For
   example, if you are using an Intel 7.x compiler and the C shell
   (e.g., csh or tcsh):

    source /opt/intel/compiler70/ia64/bin/eccvars.csh
    source /opt/intel/compiler70/ia64/bin/efcvars.csh

   Or if you are using an Intel 7.x compiler and the Bourne shell 
   (e.g., bash):
 
    . /opt/intel/compiler70/ia64/bin/eccvars.sh
    . /opt/intel/compiler70/ia64/bin/efcvars.csh

6. Set up environment variables to build FFTW and GROMACS:

   a. If you are using an Intel 7.x compiler:

       setenv CC ecc
       setenv F77 efc
       setenv MPICC ecc

      Or if you are using an Intel 8.x compiler:

       setenv CC icc
       setenv F77 ifort
       setenv MPICC icc

   b. Determine your FFTW directory (after unarchiving the 
      fftw-2.1.5.tar.gz file) and your SGI MPT directory.  Then
      specify the following environment variables, substituting
      your FFTW directory for $FFTW_DIR and your SGI MPT directory
      for $SGI_MPT_DIR below:

       setenv CPPFLAGS "-I$FFTW_DIR/include -I$SGI_MPT_DIR/include
       setenv CFLAGS "-O3 -ip -tpp2 -ftz"
       setenv FFLAGS "-O3 -ip -tpp2 -ftz"
       setenv LDFLAGS "-O3 -ip -tpp2 -ftz -L$FFTW_DIR/lib
-L$SGI_MPT_DIR/lib"
       setenv LIBS "-lmpi"

7. Build FFTW 2.1.5:

   a. Copy the updated depcomp file to the fftw-2.1.5/ directory.  For
      example:
   
       cp depcomp fftw-2.1.5/depcomp

   b. Then build fftw-2.1.5, substituting your FFTW directory for
      $FFTW_DIR below:

       cd fftw-2.1.5
       ./configure --enable-float --enable-mpi --prefix=$FFTW_DIR
       make
       make install
       cd ../

8. Build GROMACS, substituting your GROMACS directory for
   $GROMACS_DIR below:
 
    cd gmx
    ./configure --enable-mpi --prefix=$GROMACS_DIR
    make
    make install

   The GROMACS binaries and libraries will be installed under the
   $GROMACS_DIR/ia64-unknown-linux-gnu/ directory.

=====