[gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman
Mark Abraham
mark.j.abraham at gmail.com
Fri Aug 14 15:25:58 CEST 2015
Hi,
Just to follow up here - we have built smarts into mdrun in 5.1 to refuse
to run OpenCL on Mac OS X 10.10.3.
BTW, we have also enabled support for multiple GPUs per node with OpenCL,
but only with thread-MPI (for now).
Mark
On Tue, Jul 21, 2015 at 12:16 AM Erik Lindahl <erik.lindahl at gmail.com>
wrote:
> Hi Carlo,
>
> I finally had some time to double-check, and my installation indeed passes
> all regression tests on a Mac Pro with dual AMD Firepro D300 using OS X
> 10.10.4. Based on the other comments today, this appears to be caused by OS
> X 10.9 on your machine. Unfortunately OpenCL code is compiled at runtime,
> and if the error occurs regardless of gcc or clang, it is probably either
> in the system header files or driver, which means it won’t even help to
> compile statically on a machine with OS X 10.10.
>
> Unless anybody has any alternative idea, it sounds as if we’ll have to
> consider a runtime check to disable OpenCL for AMD on versions prior to
> 10.10.4.
>
>
> Cheers,
>
> Erik
>
>
>
>
>
> From: Carlo Camilloni <carlo.camilloni at gmail.com>
> <carlo.camilloni at gmail.com>
> Reply: Carlo Camilloni <carlo.camilloni at gmail.com>>
> <carlo.camilloni at gmail.com>
> Date: 20 Jul 2015 at 19:50:26
> To: Mark Abraham <mark.j.abraham at gmail.com>> <mark.j.abraham at gmail.com>
> Cc: gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
> <gmx-developers at gromacs.org>,
> gromacs.org_gmx-developers at maillist.sys.kth.se
> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
> <gromacs.org_gmx-developers at maillist.sys.kth.se>, Erik Lindahl
> <erik.lindahl at gmail.com>> <erik.lindahl at gmail.com>
>
> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
> Parrinello-Rahman
>
> Hi again,
>
> I don’t know if you had the opportunity to reproduce this bug, in the
> meantime I have compiled it with gcc 4.8, openMP and sse4.1,
> again cpu results are consistently right while OpenCL results are
> consistently wrong, so it looks like it is NOT a problem of Clang 6
> but it could be either a problem of the gpu driver or maybe OpenCL itself,
> I don’t know..
>
> this is the new log
>
> GROMACS version: VERSION 5.1-rc1
> Precision: single
> Memory model: 64 bit
> MPI library: thread_mpi
> OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
> GPU support: enabled
> OpenCL support: enabled
> invsqrt routine: gmx_software_invsqrt(x)
> SIMD instructions: SSE4.1
> FFT library: fftw-3.3.4-sse2-avx
> RDTSCP usage: enabled
> C++11 compilation: disabled
> TNG support: enabled
> Tracing support: disabled
> Built on: Mon 20 Jul 2015 18:21:43 BST
> Built by: carlo [CMAKE]
> Build OS/arch: Darwin 13.4.0 x86_64
> Build CPU vendor: GenuineIntel
> Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
> Build CPU family: 6 Model: 62 Stepping: 4
> Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler: /opt/local/bin/gcc GNU 4.8.5
> C compiler flags: -msse4.1 -Wextra -Wno-missing-field-initializers
> -Wno-sign-compare -Wpointer-arith -Wall -Wno-unused -Wunused-value
> -Wunused-parameter -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast -Wno-array-bounds
> C++ compiler: /opt/local/bin/g++ GNU 4.8.5
> C++ compiler flags: -msse4.1 -Wextra -Wno-missing-field-initializers
> -Wpointer-arith -Wall -Wno-unused-function -O3 -DNDEBUG -funroll-all-loops
> -fexcess-precision=fast -Wno-array-bounds
> Boost version: 1.58.0 (external)
> OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
> OpenCL library: /System/Library/Frameworks/OPENCL.framework
> OpenCL version: 1.2
>
>
> Best,
> Carlo
>
>
>
> On 17 Jul 2015, at 13:54, Carlo Camilloni <carlo.camilloni at gmail.com>
> wrote:
>
> Hi,
>
> Unfortunately on the Mac Pro I am running os x 10.9.5 and the newer clang
> is not yet available so I cannot test it
> with a different version. Erik if you have time to tell me what to change
> to get gromacs compiled with gcc I can give it a try
>
> Carlo
>
>
> On 16 Jul 2015, at 20:28, Mark Abraham <mark.j.abraham at gmail.com> wrote:
>
> Hi,
>
> Philip Fowler made a recent positive report on OpenCL+AMD+Mac and he was
> using /usr/bin/cc Clang 6.1.0.6020053, so there may be an issue with Clang
> 6.0 (whatever real clang version that is...)
>
> Mark
>
> On Thu, Jul 16, 2015 at 7:20 PM Erik Lindahl <erik.lindahl at gmail.com>
> wrote:
>
>> Hi,
>>
>> The one obvious difference I can see is that you have clang 6.0.0, while
>> at least my Macbook pro says 6.1.0.
>>
>> Could you check if there is any update available for the developer tools,
>> and if so does that help?
>>
>> It is actually much better to use gcc since you will then also use all
>> CPU cores. However, that requires patching a header file in /usr/include;
>> I’ll dig up what I did there - we need to put it in the documentation
>> anyway.
>>
>> Cheers,
>>
>> Erik
>>
>> From: Carlo Camilloni <carlo.camilloni at gmail.com>
>> <carlo.camilloni at gmail.com>
>> Reply: Carlo Camilloni <carlo.camilloni at gmail.com>>
>> <carlo.camilloni at gmail.com>
>> Date: 16 Jul 2015 at 19:07:13
>> To: Erik Lindahl <erik.lindahl at gmail.com>> <erik.lindahl at gmail.com>
>> Cc: gromacs.org_gmx-developers at maillist.sys.kth.se
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>,
>> gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
>> <gmx-developers at gromacs.org>
>>
>> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> Parrinello-Rahman
>>
>> Hi Erik,
>>
>> here it is:
>>
>>
>> GROMACS version: VERSION 5.1-rc1
>> Precision: single
>> Memory model: 64 bit
>> MPI library: none
>> OpenMP support: disabled
>> GPU support: enabled
>> OpenCL support: enabled
>> invsqrt routine: gmx_software_invsqrt(x)
>> SIMD instructions: AVX_256
>> FFT library: fftw-3.3.4-sse2-avx
>> RDTSCP usage: enabled
>> C++11 compilation: disabled
>> TNG support: enabled
>> Tracing support: disabled
>> Built on: Thu 16 Jul 2015 09:34:14 BST
>> Built by: carlo@ [CMAKE]
>> Build OS/arch: Darwin 13.4.0 x86_64
>> Build CPU vendor: GenuineIntel
>> Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
>> Build CPU family: 6 Model: 62 Stepping: 4
>> Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
>> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
>> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>> C compiler:
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
>> Clang 6.0.0.6000057
>> C compiler flags: -mavx -Wall -Wno-unused -Wunused-value
>> -Wunused-parameter -Wno-unknown-pragmas -O3 -DNDEBUG
>> C++ compiler:
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
>> Clang 6.0.0.6000057
>> C++ compiler flags: -mavx -Wextra -Wno-missing-field-initializers
>> -Wpointer-arith -Wall -Wno-unused-function -Wno-unknown-pragmas -O3
>> -DNDEBUG
>> Boost version: 1.58.0 (external)
>> OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
>> OpenCL library: /System/Library/Frameworks/OPENCL.framework
>> OpenCL version: 1.2
>>
>>
>> Running on 1 node with total 12 logical cores, 2 compatible GPUs
>> Hardware detected:
>> CPU info:
>> Vendor: GenuineIntel
>> Brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
>> Family: 6 model: 62 stepping: 4
>> CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx
>> msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2
>> sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>> SIMD instructions most likely to fit this hardware: AVX_256
>> SIMD instructions selected at GROMACS compile time: AVX_256
>> GPU info:
>> Number of GPUs detected: 2
>> #0: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD,
>> device version: OpenCL 1.2 , stat: compatible
>> #1: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD,
>> device version: OpenCL 1.2 , stat: compatible
>>
>> Carlo
>>
>>
>>
>> On 16 Jul 2015, at 18:59, Erik Lindahl <erik.lindahl at gmail.com> wrote:
>>
>> Hi Carlo,
>>
>> IIRC, I managed to get it working on a Mac Pro with AMD Firepro D300
>> GPUs. Please post the contents of the top of your logfile (where it says
>> everything about the compilers & config), and mention what hardware you
>> tried it on - then we’ll see if we can reproduce it.
>>
>> Cheers,
>>
>> Erik
>>
>> From: Carlo Camilloni <carlo.camilloni at gmail.com>
>> <carlo.camilloni at gmail.com>
>> Reply: gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
>> <gmx-developers at gromacs.org>
>> Date: 16 Jul 2015 at 18:52:21
>> To: gromacs.org_gmx-developers at maillist.sys.kth.se
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>>
>> <gromacs.org_gmx-developers at maillist.sys.kth.se>
>> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> Parrinello-Rahman
>>
>> Hi,
>>
>> I tested the OpenCL kernel on my macbook (nvidia gpu) and here it
>> produces the correct forces,
>> so it could be a problem related to amd+osx, or maybe to some specific
>> compiler/os version
>>
>> Carlo
>>
>>
>> > On 15 Jul 2015, at 17:42, Carlo Camilloni <carlo.camilloni at gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > these are the tests that fail:
>> >
>> > FAILED. Check checkpot.out (12 errors), checkforce.out (3516 errors)
>> file(s) in dd121 for dd121
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn-energy-groups for nbnxn-energy-groups
>> > FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors)
>> file(s) in nbnxn-free-energy for nbnxn-free-energy
>> > FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors)
>> file(s) in nbnxn-free-energy-vv for nbnxn-free-energy-vv
>> > FAILED. Check checkpot.out (11 errors), checkforce.out (4039 errors)
>> file(s) in nbnxn-ljpme-geometric for nbnxn-ljpme-geometric
>> > FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors)
>> file(s) in nbnxn-ljpme-LB for nbnxn-ljpme-LB
>> > FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors)
>> file(s) in nbnxn-ljpme-LB-geometric for nbnxn-ljpme-LB-geometric
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4029 errors)
>> file(s) in nbnxn-vdw-force-switch for nbnxn-vdw-force-switch
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4032 errors)
>> file(s) in nbnxn-vdw-potential-switch for nbnxn-vdw-potential-switch
>> > FAILED. Check checkpot.out (4 errors), checkforce.out (250 errors)
>> file(s) in nbnxn-vdw-potential-switch-argon for
>> nbnxn-vdw-potential-switch-argon
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn_pme for nbnxn_pme
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn_pme_order5 for nbnxn_pme_order5
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors)
>> file(s) in nbnxn_pme_order6 for nbnxn_pme_order6
>> > FAILED. Check checkpot.out (9 errors), checkforce.out (4028 errors)
>> file(s) in nbnxn_rf for nbnxn_rf
>> > FAILED. Check checkpot.out (2 errors), checkforce.out (4 errors)
>> file(s) in nbnxn_rzero for nbnxn_rzero
>> > FAILED. Check mdrun.out, md.log file(s) in nbnxn_vsite for nbnxn_vsite
>> > FAILED. Check checkpot.out (13 errors), checkforce.out (15512 errors)
>> file(s) in octahedron for octahedron
>> > FAILED. Check mdrun.out, md.log file(s) in position-restraints for
>> position-restraints
>> > FAILED. Check mdrun.out, md.log file(s) in pull_constraint for
>> pull_constraint
>> > FAILED. Check checkpot.out (10 errors), checkforce.out (4021 errors)
>> file(s) in pull_cylinder for pull_cylinder
>> > FAILED. Check checkpot.out (11 errors), checkforce.out (39054 errors)
>> file(s) in swap_x for swap_x
>> > FAILED. Check checkpot.out (11 errors), checkforce.out (39053 errors)
>> file(s) in swap_y for swap_y
>> > FAILED. Check checkpot.out (12 errors), checkforce.out (39054 errors)
>> file(s) in swap_z for swap_z
>> > 23 out of 60 complex tests FAILED
>> > FAILED. Check mdrun.out, md.log file(s) in expanded for expanded
>> > FAILED. Check mdrun.out, md.log file(s) in transformAtoB for
>> transformAtoB
>> > 2 out of 10 freeenergy tests FAILED
>> >
>> >
>> > Carlo
>> >
>> >
>> >>
>> >>
>> >> Message: 4
>> >> Date: Wed, 15 Jul 2015 15:35:13 +0000
>> >> From: Mark Abraham <mark.j.abraham at gmail.com>
>> >> To: gmx-developers at gromacs.org,
>> >> gromacs.org_gmx-developers at maillist.sys.kth.se
>> >> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> >> Parrinello-Rahman
>> >> Message-ID:
>> >> <CAMNuMATveVRRyBBwn312xrY+w3M7deC2Hs3A7PZnaeugkw+VVA at mail.gmail.com>
>> >> Content-Type: text/plain; charset="utf-8"
>> >>
>> >> Hi,
>> >>
>> >> Thanks. If a difference of that magnitude can be seen, then it should
>> also
>> >> show up when running the regressiontests (e.g. cmake
>> >> -DREGRESSIONTEST_DOWNLOAD=on and then make check) as a failure
>> >> of complex/nbnxn-ljpme-LB (which is the only P-R test that can run on
>> the
>> >> GPU). If other tests fail, then the problem is actually more
>> widespread.
>> >>
>> >> It may be that there is some issue with some part of the
>> Mac+clang+OpenCL
>> >> stack - we didn't target it during development, and at the last minute
>> when
>> >> Erik was unexpectedly able to get it to compile. I don't know if he got
>> >> tests to pass. Erik?
>> >>
>> >> Mark
>> >>
>> >> On Wed, Jul 15, 2015 at 5:22 PM Carlo Camilloni <
>> carlo.camilloni at gmail.com>
>> >> wrote:
>> >>
>> >>>
>> >>> Dear Mark and Szilard,
>> >>>
>> >>> thanks for your answer. I filed a bug in redmine but in the meantime
>> I was
>> >>> running more tests and I am a bit scared by what I found:
>> >>>
>> >>> what I have done is the following I have performed a single step run
>> with
>> >>> gmx51-rc1 compiled with cuda, again clang and so on
>> >>> and compared the forces on the first step with -nb cpu or not (I am
>> using
>> >>> -pforce 1), the forces are identical:
>> >>>
>> >>> ie.:
>> >>>
>> >>> cuda-gpu
>> >>>
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
>> >>>
>> >>>
>> >>> cpu:
>> >>>
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
>> >>>
>> >>> if I do the same test on the version compiled with OpenCL
>> >>>
>> >>> cpu:
>> >>>
>> >>> (the former are done on my macbook pro avx2_256 this latter on a
>> MacPro
>> >>> avx_256, this should
>> >>> explain the small differences in the forces)
>> >>>
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90838e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24848e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89470e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42043e+02
>> >>>
>> >>> opencl-gpu:
>> >>> step 0 atom 1 x 3.940 5.612 2.226 force 1.48597e+03
>> >>> step 0 atom 2 x 3.852 5.659 2.211 force 6.26942e+02
>> >>> step 0 atom 3 x 3.979 5.665 2.303 force 8.44032e+02
>> >>> step 0 atom 4 x 3.992 5.610 2.139 force 7.92786e+02
>> >>>
>> >>> I am afraid there is something wrong in OpenCL kernels.
>> >>>
>> >>> I am using the topol-nvt-nogen.tpr I have uploaded on redmine.
>> >>>
>> >>> Best,
>> >>> Carlo
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Gromacs Developers mailing list
>> >>>
>> >>> * Please search the archive at
>> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List
>> before
>> >>> posting!
>> >>>
>> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>>
>> >>> * For (un)subscribe requests visit
>> >>>
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> >>> or send a mail to gmx-developers-request at gromacs.org.
>> >>>
>> >> -------------- next part --------------
>> >> An HTML attachment was scrubbed...
>> >> URL: <
>> http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150715/fb1d4126/attachment.html
>> >
>> >>
>> >> ------------------------------
>> >>
>> >> --
>> >> Gromacs Developers mailing list
>> >>
>> >> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>> >>
>> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>> >>
>> >> * For (un)subscribe requests visit
>> >>
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>> >>
>> >> End of gromacs.org_gmx-developers Digest, Vol 135, Issue 17
>> >> ***********************************************************
>> >
>>
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>> --
>> Erik Lindahl <erik.lindahl at gmail.com>
>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
>> University
>> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
>> Inst. Technology
>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>>
>> --
>> Erik Lindahl <erik.lindahl at gmail.com>
>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
>> University
>> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
>> Inst. Technology
>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>
>
>
> --
> Erik Lindahl <erik.lindahl at gmail.com>
> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm
> University
> Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal
> Inst. Technology
> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150814/e6b5f5d3/attachment-0003.html>
More information about the gromacs.org_gmx-developers
mailing list