[gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman

Thu Jul 16 20:29:00 CEST 2015

Hi,

Philip Fowler made a recent positive report on OpenCL+AMD+Mac and he was
using /usr/bin/cc Clang 6.1.0.6020053, so there may be an issue with Clang
6.0 (whatever real clang version that is...)

Mark

On Thu, Jul 16, 2015 at 7:20 PM Erik Lindahl <erik.lindahl at gmail.com> wrote:

> Hi,
>
> The one obvious difference I can see is that you have clang 6.0.0, while
> at least my Macbook pro says 6.1.0.
>
> Could you check if there is any update available for the developer tools,
> and if so does that help?
>
> It is actually much better to use gcc since you will then also use all CPU
> cores. However, that requires patching a header file in /usr/include; I’ll
> dig up what I did there - we need to put it in the documentation anyway.
>
> Cheers,
>
> Erik
>
> From: Carlo Camilloni <carlo.camilloni at gmail.com>
> <carlo.camilloni at gmail.com>
> Reply: Carlo Camilloni <carlo.camilloni at gmail.com>>
> <carlo.camilloni at gmail.com>
> Date: 16 Jul 2015 at 19:07:13
> To: Erik Lindahl <erik.lindahl at gmail.com>> <erik.lindahl at gmail.com>
> Cc: gromacs.org_gmx-developers at maillist.sys.kth.se
> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
> <gromacs.org_gmx-developers at maillist.sys.kth.se>,
> gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
> <gmx-developers at gromacs.org>
>
> Subject:  Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
> Parrinello-Rahman
>
>  Hi Erik,
>
> here it is:
>
>
>  GROMACS version:    VERSION 5.1-rc1
> Precision:          single
> Memory model:       64 bit
> MPI library:        none
> OpenMP support:     disabled
> GPU support:        enabled
> OpenCL support:     enabled
> invsqrt routine:    gmx_software_invsqrt(x)
> SIMD instructions:  AVX_256
> FFT library:        fftw-3.3.4-sse2-avx
> RDTSCP usage:       enabled
> C++11 compilation:  disabled
> TNG support:        enabled
> Tracing support:    disabled
> Built on:           Thu 16 Jul 2015 09:34:14 BST
> Built by:           carlo@ [CMAKE]
> Build OS/arch:      Darwin 13.4.0 x86_64
> Build CPU vendor:   GenuineIntel
> Build CPU brand:    Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
> Build CPU family:   6   Model: 62   Stepping: 4
> Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
> Clang 6.0.0.6000057
> C compiler flags:    -mavx    -Wall -Wno-unused -Wunused-value
> -Wunused-parameter -Wno-unknown-pragmas  -O3 -DNDEBUG
> C++ compiler:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
> Clang 6.0.0.6000057
> C++ compiler flags:  -mavx    -Wextra -Wno-missing-field-initializers
> -Wpointer-arith -Wall -Wno-unused-function -Wno-unknown-pragmas  -O3
> -DNDEBUG
> Boost version:      1.58.0 (external)
> OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
> OpenCL library:     /System/Library/Frameworks/OPENCL.framework
> OpenCL version:     1.2
>
>
> Running on 1 node with total 12 logical cores, 2 compatible GPUs
> Hardware detected:
>   CPU info:
>     Vendor: GenuineIntel
>     Brand:  Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
>     Family:  6  model: 62  stepping:  4
>     CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>     SIMD instructions most likely to fit this hardware: AVX_256
>     SIMD instructions selected at GROMACS compile time: AVX_256
>   GPU info:
>     Number of GPUs detected: 2
>     #0: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD,
> device version: OpenCL 1.2 , stat: compatible
>     #1: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD,
> device version: OpenCL 1.2 , stat: compatible
>
> Carlo
>
>
>
>  On 16 Jul 2015, at 18:59, Erik Lindahl <erik.lindahl at gmail.com> wrote:
>
>  Hi Carlo,
>
> IIRC, I managed to get it working on a Mac Pro with AMD Firepro D300 GPUs.
> Please post the contents of the top of your logfile (where it says
> everything about the compilers & config), and mention what hardware you
> tried it on - then we’ll see if we can reproduce it.
>
> Cheers,
>
> Erik
>
> From: Carlo Camilloni <carlo.camilloni at gmail.com>
> <carlo.camilloni at gmail.com>
> Reply: gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
> <gmx-developers at gromacs.org>
> Date: 16 Jul 2015 at 18:52:21
> To: gromacs.org_gmx-developers at maillist.sys.kth.se
> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
> <gromacs.org_gmx-developers at maillist.sys.kth.se>
> Subject:  Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
> Parrinello-Rahman
>
>   Hi,
>
> I tested the OpenCL kernel on my macbook (nvidia gpu) and here it produces
> the correct forces,
> so it could be a problem related to amd+osx, or maybe to some specific
> compiler/os version
>
> Carlo
>
>
> > On 15 Jul 2015, at 17:42, Carlo Camilloni <carlo.camilloni at gmail.com>
> wrote:
> >
> > Hi,
> >
> > these are the tests that fail:
> >
> > FAILED. Check checkpot.out (12 errors), checkforce.out (3516 errors)
> file(s) in dd121 for dd121
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
> file(s) in nbnxn-energy-groups for nbnxn-energy-groups
> > FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors)
> file(s) in nbnxn-free-energy for nbnxn-free-energy
> > FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors)
> file(s) in nbnxn-free-energy-vv for nbnxn-free-energy-vv
> > FAILED. Check checkpot.out (11 errors), checkforce.out (4039 errors)
> file(s) in nbnxn-ljpme-geometric for nbnxn-ljpme-geometric
> > FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors)
> file(s) in nbnxn-ljpme-LB for nbnxn-ljpme-LB
> > FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors)
> file(s) in nbnxn-ljpme-LB-geometric for nbnxn-ljpme-LB-geometric
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4029 errors)
> file(s) in nbnxn-vdw-force-switch for nbnxn-vdw-force-switch
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4032 errors)
> file(s) in nbnxn-vdw-potential-switch for nbnxn-vdw-potential-switch
> > FAILED. Check checkpot.out (4 errors), checkforce.out (250 errors)
> file(s) in nbnxn-vdw-potential-switch-argon for
> nbnxn-vdw-potential-switch-argon
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
> file(s) in nbnxn_pme for nbnxn_pme
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
> file(s) in nbnxn_pme_order5 for nbnxn_pme_order5
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
> file(s) in nbnxn_pme_order6 for nbnxn_pme_order6
> > FAILED. Check checkpot.out (9 errors), checkforce.out (4028 errors)
> file(s) in nbnxn_rf for nbnxn_rf
> > FAILED. Check checkpot.out (2 errors), checkforce.out (4 errors) file(s)
> in nbnxn_rzero for nbnxn_rzero
> > FAILED. Check mdrun.out, md.log file(s) in nbnxn_vsite for nbnxn_vsite
> > FAILED. Check checkpot.out (13 errors), checkforce.out (15512 errors)
> file(s) in octahedron for octahedron
> > FAILED. Check mdrun.out, md.log file(s) in position-restraints for
> position-restraints
> > FAILED. Check mdrun.out, md.log file(s) in pull_constraint for
> pull_constraint
> > FAILED. Check checkpot.out (10 errors), checkforce.out (4021 errors)
> file(s) in pull_cylinder for pull_cylinder
> > FAILED. Check checkpot.out (11 errors), checkforce.out (39054 errors)
> file(s) in swap_x for swap_x
> > FAILED. Check checkpot.out (11 errors), checkforce.out (39053 errors)
> file(s) in swap_y for swap_y
> > FAILED. Check checkpot.out (12 errors), checkforce.out (39054 errors)
> file(s) in swap_z for swap_z
> > 23 out of 60 complex tests FAILED
> > FAILED. Check mdrun.out, md.log file(s) in expanded for expanded
> > FAILED. Check mdrun.out, md.log file(s) in transformAtoB for
> transformAtoB
> > 2 out of 10 freeenergy tests FAILED
> >
> >
> > Carlo
> >
> >
> >>
> >>
> >> Message: 4
> >> Date: Wed, 15 Jul 2015 15:35:13 +0000
> >> From: Mark Abraham <mark.j.abraham at gmail.com>
> >> To: gmx-developers at gromacs.org,
> >> gromacs.org_gmx-developers at maillist.sys.kth.se
> >> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
> >> Parrinello-Rahman
> >> Message-ID:
> >> <CAMNuMATveVRRyBBwn312xrY+w3M7deC2Hs3A7PZnaeugkw+VVA at mail.gmail.com>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Hi,
> >>
> >> Thanks. If a difference of that magnitude can be seen, then it should
> also
> >> show up when running the regressiontests (e.g. cmake
> >> -DREGRESSIONTEST_DOWNLOAD=on and then make check) as a failure
> >> of complex/nbnxn-ljpme-LB (which is the only P-R test that can run on
> the
> >> GPU). If other tests fail, then the problem is actually more widespread.
> >>
> >> It may be that there is some issue with some part of the
> Mac+clang+OpenCL
> >> stack - we didn't target it during development, and at the last minute
> when
> >> Erik was unexpectedly able to get it to compile. I don't know if he got
> >> tests to pass. Erik?
> >>
> >> Mark
> >>
> >> On Wed, Jul 15, 2015 at 5:22 PM Carlo Camilloni <
> carlo.camilloni at gmail.com>
> >> wrote:
> >>
> >>>
> >>> Dear Mark and Szilard,
> >>>
> >>> thanks for your answer. I filed a bug in redmine but in the meantime I
> was
> >>> running more tests and I am a bit scared by what I found:
> >>>
> >>> what I have done is the following I have performed a single step run
> with
> >>> gmx51-rc1 compiled with cuda, again clang and so on
> >>> and compared the forces on the first step with -nb cpu or not (I am
> using
> >>> -pforce 1), the forces are identical:
> >>>
> >>> ie.:
> >>>
> >>> cuda-gpu
> >>>
> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
> >>>
> >>>
> >>> cpu:
> >>>
> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
> >>>
> >>> if I do the same test on the version compiled with OpenCL
> >>>
> >>> cpu:
> >>>
> >>> (the former are done on my macbook pro avx2_256 this latter on a MacPro
> >>> avx_256, this should
> >>> explain the small differences in the forces)
> >>>
> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90838e+03
> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24848e+02
> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89470e+02
> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42043e+02
> >>>
> >>> opencl-gpu:
> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.48597e+03
> >>> step 0 atom 2 x 3.852 5.659 2.211 force 6.26942e+02
> >>> step 0 atom 3 x 3.979 5.665 2.303 force 8.44032e+02
> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.92786e+02
> >>>
> >>> I am afraid there is something wrong in OpenCL kernels.
> >>>
> >>> I am using the topol-nvt-nogen.tpr I have uploaded on redmine.
> >>>
> >>> Best,
> >>> Carlo
> >>>
> >>>
> >>>
> >>> --
> >>> Gromacs Developers mailing list
> >>>
> >>> * Please search the archive at
> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List
> before
> >>> posting!
> >>>
> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>>
> >>> * For (un)subscribe requests visit
> >>>
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> >>> or send a mail to gmx-developers-request at gromacs.org.
> >>>
> >> -------------- next part --------------
> >> An HTML attachment was scrubbed...
> >> URL: <
> http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150715/fb1d4126/attachment.html
> >
> >>
> >> ------------------------------
> >>
> >> --
> >> Gromacs Developers mailing list
> >>
> >> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
> >>
> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
> >>
> >> * For (un)subscribe requests visit
> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
> >>
> >> End of gromacs.org_gmx-developers Digest, Vol 135, Issue 17
> >> ***********************************************************
> >
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
>
>  --
> Erik Lindahl <erik.lindahl at gmail.com>
> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
> University
> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
> Inst. Technology
> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>
>
>  --
> Erik Lindahl <erik.lindahl at gmail.com>
> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
> University
> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
> Inst. Technology
> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150716/49bee86f/attachment-0003.html>