[gmx-users] intel compiler + amd64: help needed.

[Yang Ye]

Are you using the ICC with EM64T extension?

I would like you to look at the nature of the program: GROMACS code is 
unique as comparing to other computational software: its core code is 
written in assembly code which simply can not be optimized further by 
the compiler. What can be optimized is just the procedure calling time 
and the outer-loop code which accounts for little computational time (As 
there is no complete study, but a few benchmarking runs which fluctuate 
< 5% is considered as same for programs). CPMD and other software is 
another story which you have the free lunch of the compiler.

Yang Ye

Jones de Andrade wrote:
> Hi all.
>
> First of all, thanks everybody for the help. ;)
>
> Yang, I know, I know. I should keep to the GNU. But lets say that I 
> get addicted to the intel compiler on a cpmd winter school, when it 
> appeared in front of me as the only freely available for both amd and 
> intel cpus fortran90 compiler. Since it also came with performance for 
> free, that was a good deal. I also have just a handfull of tuning 
> flags I use for icc and ifort. But, like everybody in here, we choose 
> the machines for performance (which means, first corret result, and 
> later speed), same goes for me for the compilers. Moreover, as I said 
> before: I wana try QM/MM, and CPMD doesn't work with gcc.
>
> Back to the subject: David, thanks for the piece of gmx4 code. 
> Unfortunatelly, it don't work. I deleted all the objects from the 
> directory before trying to compile to assure that, and it yelded the 
> same error.
>
> On the other hand, the trick of compiling just mknb in gcc, and the 
> rest with icc got a bit farther, before crashing... but still gone 
> with about a thousand of weird warnings. I'm sending the window 
> desktop outputs attached in this message. Anyway, in order to make 
> that trick, I've used the following "script":
> ****************************************************************************************************
> export CPPFLAGS=-I/usr/local/lib64/fftw/gnu/include
> export LDFLAGS=-L/usr/local/lib64/fftw/gnu/lib
> export CC=/usr/local/bin/gcc
> export CFLAGS="-O3 -ffast-math -funroll-all-loops -fpeel-loops"
>
> make distclean
>
> ./configure --prefix=/usr/local/bin/chemistry/gromacs/ISI/ 
> --with-fft=fftw3
>
> cp ./src/gmxlib/nonbonded/nb_kernel/Makefile ./Makefile.mknb.gcc
> cp ./src/gmxlib/nonbonded/nb_kernel_ia32_3dnow/Makefile 
> ./Makefile.mknb_kernel_ia32_3dnow.gcc
> cp ./src/gmxlib/nonbonded/nb_kernel_ia32_sse/Makefile 
> ./Makefile.mknb_kernel_ia32_sse.gcc
> cp ./src/gmxlib/nonbonded/nb_kernel_ia32_sse2/Makefile 
> ./Makefile.mknb_kernel_ia32_sse2.gcc
> cp ./src/gmxlib/nonbonded/nb_kernel_x86_64_sse/Makefile 
> ./Makefile.mknb_kernel_x86_64_sse.gcc
> cp ./src/gmxlib/nonbonded/nb_kernel_x86_64_sse2/Makefile 
> ./Makefile.mknb_kernel_x86_64_sse2.gcc
>
> export PATH="$PATH /usr/local/intel/bin/ . /usr/local/intel/lib/"
> export LD_LIBRARY_PATH=/usr/local/intel/lib/
> export CPPFLAGS=-I/usr/local/lib64/fftw/intel/include
> export LDFLAGS=-L/usr/local/lib64/fftw/intel/lib
> export CC=/usr/local/intel/bin/icc
> export CXX=/usr/local/intel/bin/icpc
> export F77=/usr/local/intel/bin/ifort
> export CFLAGS=" "
> export CXXFLAGS=" "
> export FFLAGS=" "
> export CFLAGS="-O3 -axP -xW -ipo -static "
> export CXXFLAGS="-O3 -axP -xW -ipo -static "
> export FFLAGS="-O3 -axP -xW -ipo -static "
>
> make distclean
>
> ./configure --prefix=/usr/local/bin/chemistry/gromacs/ISI/ 
> --with-fft=fftw3 --disable-x86-64-sse
>
> cp ./src/gmxlib/nonbonded/nb_kernel/Makefile ./Makefile.mknb.icc
> cp ./src/gmxlib/nonbonded/nb_kernel_ia32_3dnow/Makefile 
> ./Makefile.mknb_kernel_ia32_3dnow.icc
> cp ./src/gmxlib/nonbonded/nb_kernel_ia32_sse/Makefile 
> ./Makefile.mknb_kernel_ia32_sse.icc
> cp ./src/gmxlib/nonbonded/nb_kernel_ia32_sse2/Makefile 
> ./Makefile.mknb_kernel_ia32_sse2.icc
> cp ./src/gmxlib/nonbonded/nb_kernel_x86_64_sse/Makefile 
> ./Makefile.mknb_kernel_x86_64_sse.icc
> cp ./src/gmxlib/nonbonded/nb_kernel_x86_64_sse2/Makefile 
> ./Makefile.mknb_kernel_x86_64_sse2.icc
>
> cp ./Makefile.mknb.gcc ./src/gmxlib/nonbonded/nb_kernel/Makefile
> cp ./Makefile.mknb_kernel_ia32_3dnow.gcc 
> ./src/gmxlib/nonbonded/nb_kernel_ia32_3dnow/Makefile
> cp ./Makefile.mknb_kernel_ia32_sse.gcc 
> ./src/gmxlib/nonbonded/nb_kernel_ia32_sse/Makefile
> cp ./Makefile.mknb_kernel_ia32_sse2.gcc 
> ./src/gmxlib/nonbonded/nb_kernel_ia32_sse2/Makefile
> cp ./Makefile.mknb_kernel_x86_64_sse.gcc 
> ./src/gmxlib/nonbonded/nb_kernel_x86_64_sse/Makefile
> cp ./Makefile.mknb_kernel_x86_64_sse2.gcc 
> ./src/gmxlib/nonbonded/nb_kernel_x86_64_sse2/Makefile
>
> make
> ****************************************************************************************************
>
> The output from it isn't attached. :( Sorry,  The gziped file is too 
> big for the list limits. Anyway, despite lots of warnings, the 
> compilations ends dieing in here:
> *********************************************************************************************
> /tmp/ipo_iccpmJSXr.c: undefined reference to `interaction_function'
> /tmp/ipo_iccpmJSXr.o(.text+0x2edde):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `interaction_function'
> /tmp/ipo_iccpmJSXr.o(.text+0x2ede6):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `interaction_function'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f1ca):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `_gmx_error'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f29b): In function `DS_Init':
> /tmp/ipo_iccpmJSXr.c: undefined reference to `save_calloc'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f64d):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `debug'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f682):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `debug'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f6f6):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `debug'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f76c): In function `init_block2':
> /tmp/ipo_iccpmJSXr.c: undefined reference to `save_calloc'
> /tmp/ipo_iccpmJSXr.o(.text+0x2f78d):/tmp/ipo_iccpmJSXr.c: undefined 
> reference to `save_calloc'
> make[3]: ** [grompp] Erro 1
> make[3]: Leaving directory 
> `/home/johannes/src/gromacs/gromacs-3.3/src/kernel'
> make[2]: ** [all-recursive] Erro 1
> make[2]: Leaving directory `/home/johannes/src/gromacs/gromacs-3.3/src'
> make[1]: ** [all] Erro 2
> make[1]: Leaving directory `/home/johannes/src/gromacs/gromacs-3.3/src'
> make: ** [all-recursive] Erro 1
> **********************************************************************************************
>
> Trying withou the "-ipo" flag now. It still yelds the same weird 
> warning (warning #147: declaration is incompatible with "double 
> invsqrt(double)" (declared at line 565 of 
> "/usr/local/intel/include/math.h")
>   static inline real invsqrt(real x)) and shows a lot of warnings like 
> these before stopping:
>
> **********************************************************************************************
> ld: skipping incompatible /usr/X11/lib/libXmu.so when searching for -lXmu
> ld: skipping incompatible /usr/X11/lib/libXmu.a when searching for -lXmu
> ld: skipping incompatible /usr/X11R6/lib/libXmu.so when searching for 
> -lXmu
> ld: skipping incompatible /usr/X11R6/lib/libXmu.a when searching for -lXmu
> ld: skipping incompatible /usr/X11/lib/libSM.so when searching for -lSM
> ld: skipping incompatible /usr/X11/lib/libSM.a when searching for -lSM
> ld: skipping incompatible /usr/X11R6/lib/libSM.so when searching for -lSM
> ld: skipping incompatible /usr/X11R6/lib/libSM.a when searching for -lSM
> ld: skipping incompatible /usr/X11/lib/libICE.so when searching for -lICE
> ld: skipping incompatible /usr/X11/lib/libICE.a when searching for -lICE
> ld: skipping incompatible /usr/X11R6/lib/libICE.so when searching for 
> -lICE
> ld: skipping incompatible /usr/X11R6/lib/libICE.a when searching for -lICE
> ld: skipping incompatible /usr/X11/lib/libXext.so when searching for 
> -lXext
> ld: skipping incompatible /usr/X11/lib/libXext.a when searching for -lXext
> ld: skipping incompatible /usr/X11R6/lib/libXext.so when searching for 
> -lXext
> ld: skipping incompatible /usr/X11R6/lib/libXext.a when searching for 
> -lXext
> ld: skipping incompatible /usr/X11/lib/libXp.so when searching for -lXp
> ld: skipping incompatible /usr/X11/lib/libXp.a when searching for -lXp
> ld: skipping incompatible /usr/X11R6/lib/libXp.so when searching for -lXp
> ld: skipping incompatible /usr/X11R6/lib/libXp.a when searching for -lXp
> ld: skipping incompatible /usr/X11/lib/libXt.so when searching for -lXt
> ld: skipping incompatible /usr/X11/lib/libXt.a when searching for -lXt
> ld: skipping incompatible /usr/X11R6/lib/libXt.so when searching for -lXt
> ld: skipping incompatible /usr/X11R6/lib/libXt.a when searching for -lXt
> ld: skipping incompatible /usr/X11/lib/libX11.so when searching for -lX11
> ld: skipping incompatible /usr/X11/lib/libX11.a when searching for -lX11
> ld: skipping incompatible /usr/X11R6/lib/libX11.so when searching for 
> -lX11
> ld: skipping incompatible /usr/X11R6/lib/libX11.a when searching for -lX11
> **********************************************************************************************
>
> I don't know why, but I got the feeling that this is the most 
> promissing stuff up to now! :) But what are those warnings and errors?
>
> Tom, just did as suggest with gdb. The output follows:
> *****************************************************************
> GNU gdb 6.3
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and 
> you are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for 
> details.
> This GDB was configured as "x86_64-suse-linux"...Using host 
> libthread_db library "/lib64/tls/libthread_db.so.1".
>
> (gdb) r -software_invsqrt
> Starting program: 
> /home/johannes/src/gromacs/gromacs-3.3/src/gmxlib/nonbonded/nb_kernel/mknb 
> -software_invsqrt
> >>> Gromacs nonbonded kernel generator (-h for help)
> >>> Generating single precision functions in C.
> >>> Using Gromacs software version of 1/sqrt(x).
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00002aaaab15a88c in __find_specmb () from /lib64/tls/libc.so.6
> (gdb) bt
> #0  0x00002aaaab15a88c in __find_specmb () from /lib64/tls/libc.so.6
> #1  0x00002aaaab140e6f in vfprintf () from /lib64/tls/libc.so.6
> #2  0x00002aaaab15e2a9 in vsprintf () from /lib64/tls/libc.so.6
> #3  0x00002aaaab149568 in sprintf () from /lib64/tls/libc.so.6
> #4  0x0000000000402a47 in mknb_code ()
> #5  0x0000000000402183 in mknb_declare_int ()
> #6  0x000000000040356d in mknb_declare_variables ()
> #7  0x0000000000401989 in main ()
> **********************************************************************
> Seems to yeld the same as before... :( Any bell ringing?
>
> People, really sorry for bothering the whole list with this subject. I 
> would not do so if I didn't want to give a try for the gromacs+cpmd 
> combo. Also, I do think that if this can be done, more people will be 
> able to get results from this (even if the results is "don't try, it 
> doesn't yeld you any performance improve on AMD machines", it will at 
> least let others without any doubt if the trial would be usefull)
>
> Thanks in advance for everything.
>
> Jones
> ------------------------------------------------------------------------
>
> _______________________________________________
> gmx-users mailing list    gmx-users at gromacs.org
> http://www.gromacs.org/mailman/listinfo/gmx-users
> Please don't post (un)subscribe requests to the list. Use the 
> www interface or send it to gmx-users-request at gromacs.org.
> Can't post? Read http://www.gromacs.org/mailing_lists/users.php

-- 
/Regards,/
Yang Ye
/Computational Biology Lab
School of Biological Sciences
Nanyang Technological University
Singapore
Tel: 6316-2884
/