[gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman
Szilárd Páll
pall.szilard at gmail.com
Mon Jul 20 20:07:23 CEST 2015
Hi,
At this point my guess is that the source of the issue is the particular
version of the Apple OpenCL runtime or compiler you have. Unfortunately,
our version info does not contain any OpenCL runtime/compiler version
information, but you should be able to query that and compare it against
someone else's version:
system_profiler | grep -A 10 OpenCL
Cheers,
--
Szilárd
On Mon, Jul 20, 2015 at 7:50 PM, Carlo Camilloni <carlo.camilloni at gmail.com>
wrote:
> Hi again,
>
> I don’t know if you had the opportunity to reproduce this bug, in the
> meantime I have compiled it with gcc 4.8, openMP and sse4.1,
> again cpu results are consistently right while OpenCL results are
> consistently wrong, so it looks like it is NOT a problem of Clang 6
> but it could be either a problem of the gpu driver or maybe OpenCL itself,
> I don’t know..
>
> this is the new log
>
> GROMACS version: VERSION 5.1-rc1
> Precision: single
> Memory model: 64 bit
> MPI library: thread_mpi
> OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
> GPU support: enabled
> OpenCL support: enabled
> invsqrt routine: gmx_software_invsqrt(x)
> SIMD instructions: SSE4.1
> FFT library: fftw-3.3.4-sse2-avx
> RDTSCP usage: enabled
> C++11 compilation: disabled
> TNG support: enabled
> Tracing support: disabled
> Built on: Mon 20 Jul 2015 18:21:43 BST
> Built by: carlo [CMAKE]
> Build OS/arch: Darwin 13.4.0 x86_64
> Build CPU vendor: GenuineIntel
> Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
> Build CPU family: 6 Model: 62 Stepping: 4
> Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler: /opt/local/bin/gcc GNU 4.8.5
> C compiler flags: -msse4.1 -Wextra -Wno-missing-field-initializers
> -Wno-sign-compare -Wpointer-arith -Wall -Wno-unused -Wunused-value
> -Wunused-parameter -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast -Wno-array-bounds
> C++ compiler: /opt/local/bin/g++ GNU 4.8.5
> C++ compiler flags: -msse4.1 -Wextra -Wno-missing-field-initializers
> -Wpointer-arith -Wall -Wno-unused-function -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast -Wno-array-bounds
> Boost version: 1.58.0 (external)
> OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
> OpenCL library: /System/Library/Frameworks/OPENCL.framework
> OpenCL version: 1.2
>
>
> Best,
> Carlo
>
>
>
> On 17 Jul 2015, at 13:54, Carlo Camilloni <carlo.camilloni at gmail.com>
> wrote:
>
> Hi,
>
> Unfortunately on the Mac Pro I am running os x 10.9.5 and the newer clang
> is not yet available so I cannot test it
> with a different version. Erik if you have time to tell me what to change
> to get gromacs compiled with gcc I can give it a try
>
> Carlo
>
>
> On 16 Jul 2015, at 20:28, Mark Abraham <mark.j.abraham at gmail.com> wrote:
>
> Hi,
>
> Philip Fowler made a recent positive report on OpenCL+AMD+Mac and he was
> using /usr/bin/cc Clang 6.1.0.6020053, so there may be an issue with Clang
> 6.0 (whatever real clang version that is...)
>
> Mark
>
> On Thu, Jul 16, 2015 at 7:20 PM Erik Lindahl <erik.lindahl at gmail.com>
> wrote:
>
>> Hi,
>>
>> The one obvious difference I can see is that you have clang 6.0.0, while
>> at least my Macbook pro says 6.1.0.
>>
>> Could you check if there is any update available for the developer tools,
>> and if so does that help?
>>
>> It is actually much better to use gcc since you will then also use all
>> CPU cores. However, that requires patching a header file in /usr/include;
>> I’ll dig up what I did there - we need to put it in the documentation
>> anyway.
>>
>> Cheers,
>>
>> Erik
>>
>> From: Carlo Camilloni <carlo.camilloni at gmail.com>
>> <carlo.camilloni at gmail.com>
>> Reply: Carlo Camilloni <carlo.camilloni at gmail.com>>
>> <carlo.camilloni at gmail.com>
>> Date: 16 Jul 2015 at 19:07:13
>> To: Erik Lindahl <erik.lindahl at gmail.com>> <erik.lindahl at gmail.com>
>> Cc: gromacs.org_gmx-developers at maillist.sys.kth.se
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>,
>> gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
>> <gmx-developers at gromacs.org>
>>
>> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> Parrinello-Rahman
>>
>> Hi Erik,
>>
>> here it is:
>>
>>
>> GROMACS version: VERSION 5.1-rc1
>> Precision: single
>> Memory model: 64 bit
>> MPI library: none
>> OpenMP support: disabled
>> GPU support: enabled
>> OpenCL support: enabled
>> invsqrt routine: gmx_software_invsqrt(x)
>> SIMD instructions: AVX_256
>> FFT library: fftw-3.3.4-sse2-avx
>> RDTSCP usage: enabled
>> C++11 compilation: disabled
>> TNG support: enabled
>> Tracing support: disabled
>> Built on: Thu 16 Jul 2015 09:34:14 BST
>> Built by: carlo@ [CMAKE]
>> Build OS/arch: Darwin 13.4.0 x86_64
>> Build CPU vendor: GenuineIntel
>> Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
>> Build CPU family: 6 Model: 62 Stepping: 4
>> Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
>> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
>> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>> C compiler:
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
>> Clang 6.0.0.6000057
>> C compiler flags: -mavx -Wall -Wno-unused -Wunused-value
>> -Wunused-parameter -Wno-unknown-pragmas -O3 -DNDEBUG
>> C++ compiler:
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
>> Clang 6.0.0.6000057
>> C++ compiler flags: -mavx -Wextra -Wno-missing-field-initializers
>> -Wpointer-arith -Wall -Wno-unused-function -Wno-unknown-pragmas -O3
>> -DNDEBUG
>> Boost version: 1.58.0 (external)
>> OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
>> OpenCL library: /System/Library/Frameworks/OPENCL.framework
>> OpenCL version: 1.2
>>
>>
>> Running on 1 node with total 12 logical cores, 2 compatible GPUs
>> Hardware detected:
>> CPU info:
>> Vendor: GenuineIntel
>> Brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
>> Family: 6 model: 62 stepping: 4
>> CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
>> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
>> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>> SIMD instructions most likely to fit this hardware: AVX_256
>> SIMD instructions selected at GROMACS compile time: AVX_256
>> GPU info:
>> Number of GPUs detected: 2
>> #0: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD,
>> device version: OpenCL 1.2 , stat: compatible
>> #1: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD,
>> device version: OpenCL 1.2 , stat: compatible
>>
>> Carlo
>>
>>
>>
>> On 16 Jul 2015, at 18:59, Erik Lindahl <erik.lindahl at gmail.com> wrote:
>>
>> Hi Carlo,
>>
>> IIRC, I managed to get it working on a Mac Pro with AMD Firepro D300
>> GPUs. Please post the contents of the top of your logfile (where it says
>> everything about the compilers & config), and mention what hardware you
>> tried it on - then we’ll see if we can reproduce it.
>>
>> Cheers,
>>
>> Erik
>>
>> From: Carlo Camilloni <carlo.camilloni at gmail.com>
>> <carlo.camilloni at gmail.com>
>> Reply: gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
>> <gmx-developers at gromacs.org>
>> Date: 16 Jul 2015 at 18:52:21
>> To: gromacs.org_gmx-developers at maillist.sys.kth.se
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>
>> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> Parrinello-Rahman
>>
>> Hi,
>>
>> I tested the OpenCL kernel on my macbook (nvidia gpu) and here it
>> produces the correct forces,
>> so it could be a problem related to amd+osx, or maybe to some specific
>> compiler/os version
>>
>> Carlo
>>
>>
>> > On 15 Jul 2015, at 17:42, Carlo Camilloni <carlo.camilloni at gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > these are the tests that fail:
>> >
>> > FAILED. Check checkpot.out (12 errors), checkforce.out (3516 errors)
>> file(s) in dd121 for dd121
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn-energy-groups for nbnxn-energy-groups
>> > FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors)
>> file(s) in nbnxn-free-energy for nbnxn-free-energy
>> > FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors)
>> file(s) in nbnxn-free-energy-vv for nbnxn-free-energy-vv
>> > FAILED. Check checkpot.out (11 errors), checkforce.out (4039 errors)
>> file(s) in nbnxn-ljpme-geometric for nbnxn-ljpme-geometric
>> > FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors)
>> file(s) in nbnxn-ljpme-LB for nbnxn-ljpme-LB
>> > FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors)
>> file(s) in nbnxn-ljpme-LB-geometric for nbnxn-ljpme-LB-geometric
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4029 errors)
>> file(s) in nbnxn-vdw-force-switch for nbnxn-vdw-force-switch
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4032 errors)
>> file(s) in nbnxn-vdw-potential-switch for nbnxn-vdw-potential-switch
>> > FAILED. Check checkpot.out (4 errors), checkforce.out (250 errors)
>> file(s) in nbnxn-vdw-potential-switch-argon for
>> nbnxn-vdw-potential-switch-argon
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn_pme for nbnxn_pme
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn_pme_order5 for nbnxn_pme_order5
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn_pme_order6 for nbnxn_pme_order6
>> > FAILED. Check checkpot.out (9 errors), checkforce.out (4028 errors)
>> file(s) in nbnxn_rf for nbnxn_rf
>> > FAILED. Check checkpot.out (2 errors), checkforce.out (4 errors)
>> file(s) in nbnxn_rzero for nbnxn_rzero
>> > FAILED. Check mdrun.out, md.log file(s) in nbnxn_vsite for nbnxn_vsite
>> > FAILED. Check checkpot.out (13 errors), checkforce.out (15512 errors)
>> file(s) in octahedron for octahedron
>> > FAILED. Check mdrun.out, md.log file(s) in position-restraints for
>> position-restraints
>> > FAILED. Check mdrun.out, md.log file(s) in pull_constraint for
>> pull_constraint
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4021 errors)
>> file(s) in pull_cylinder for pull_cylinder
>> > FAILED. Check checkpot.out (11 errors), checkforce.out (39054 errors)
>> file(s) in swap_x for swap_x
>> > FAILED. Check checkpot.out (11 errors), checkforce.out (39053 errors)
>> file(s) in swap_y for swap_y
>> > FAILED. Check checkpot.out (12 errors), checkforce.out (39054 errors)
>> file(s) in swap_z for swap_z
>> > 23 out of 60 complex tests FAILED
>> > FAILED. Check mdrun.out, md.log file(s) in expanded for expanded
>> > FAILED. Check mdrun.out, md.log file(s) in transformAtoB for
>> transformAtoB
>> > 2 out of 10 freeenergy tests FAILED
>> >
>> >
>> > Carlo
>> >
>> >
>> >>
>> >>
>> >> Message: 4
>> >> Date: Wed, 15 Jul 2015 15:35:13 +0000
>> >> From: Mark Abraham <mark.j.abraham at gmail.com>
>> >> To: gmx-developers at gromacs.org,
>> >> gromacs.org_gmx-developers at maillist.sys.kth.se
>> >> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> >> Parrinello-Rahman
>> >> Message-ID:
>> >> <CAMNuMATveVRRyBBwn312xrY+w3M7deC2Hs3A7PZnaeugkw+VVA at mail.gmail.com>
>> >> Content-Type: text/plain; charset="utf-8"
>> >>
>> >> Hi,
>> >>
>> >> Thanks. If a difference of that magnitude can be seen, then it should
>> also
>> >> show up when running the regressiontests (e.g. cmake
>> >> -DREGRESSIONTEST_DOWNLOAD=on and then make check) as a failure
>> >> of complex/nbnxn-ljpme-LB (which is the only P-R test that can run on
>> the
>> >> GPU). If other tests fail, then the problem is actually more
>> widespread.
>> >>
>> >> It may be that there is some issue with some part of the
>> Mac+clang+OpenCL
>> >> stack - we didn't target it during development, and at the last minute
>> when
>> >> Erik was unexpectedly able to get it to compile. I don't know if he got
>> >> tests to pass. Erik?
>> >>
>> >> Mark
>> >>
>> >> On Wed, Jul 15, 2015 at 5:22 PM Carlo Camilloni <
>> carlo.camilloni at gmail.com>
>> >> wrote:
>> >>
>> >>>
>> >>> Dear Mark and Szilard,
>> >>>
>> >>> thanks for your answer. I filed a bug in redmine but in the meantime
>> I was
>> >>> running more tests and I am a bit scared by what I found:
>> >>>
>> >>> what I have done is the following I have performed a single step run
>> with
>> >>> gmx51-rc1 compiled with cuda, again clang and so on
>> >>> and compared the forces on the first step with -nb cpu or not (I am
>> using
>> >>> -pforce 1), the forces are identical:
>> >>>
>> >>> ie.:
>> >>>
>> >>> cuda-gpu
>> >>>
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
>> >>>
>> >>>
>> >>> cpu:
>> >>>
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
>> >>>
>> >>> if I do the same test on the version compiled with OpenCL
>> >>>
>> >>> cpu:
>> >>>
>> >>> (the former are done on my macbook pro avx2_256 this latter on a
>> MacPro
>> >>> avx_256, this should
>> >>> explain the small differences in the forces)
>> >>>
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90838e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24848e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89470e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42043e+02
>> >>>
>> >>> opencl-gpu:
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.48597e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 6.26942e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 8.44032e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.92786e+02
>> >>>
>> >>> I am afraid there is something wrong in OpenCL kernels.
>> >>>
>> >>> I am using the topol-nvt-nogen.tpr I have uploaded on redmine.
>> >>>
>> >>> Best,
>> >>> Carlo
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Gromacs Developers mailing list
>> >>>
>> >>> * Please search the archive at
>> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List
>> before
>> >>> posting!
>> >>>
>> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>
>> >>> * For (un)subscribe requests visit
>> >>>
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> >>> or send a mail to gmx-developers-request at gromacs.org.
>> >>>
>> >> -------------- next part --------------
>> >> An HTML attachment was scrubbed...
>> >> URL: <
>> http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150715/fb1d4126/attachment.html
>> >
>> >>
>> >> ------------------------------
>> >>
>> >> --
>> >> Gromacs Developers mailing list
>> >>
>> >> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>> >>
>> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>
>> >> * For (un)subscribe requests visit
>> >>
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>> >>
>> >> End of gromacs.org_gmx-developers Digest, Vol 135, Issue 17
>> >> ***********************************************************
>> >
>>
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>> --
>> Erik Lindahl <erik.lindahl at gmail.com>
>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
>> University
>> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
>> Inst. Technology
>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>>
>> --
>> Erik Lindahl <erik.lindahl at gmail.com>
>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
>> University
>> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
>> Inst. Technology
>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>
>
>
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150720/a686157f/attachment-0003.html>
More information about the gromacs.org_gmx-developers
mailing list