[gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman
Erik Lindahl
erik.lindahl at gmail.com
Tue Jul 21 00:16:09 CEST 2015
Hi Carlo,
I finally had some time to double-check, and my installation indeed passes all regression tests on a Mac Pro with dual AMD Firepro D300 using OS X 10.10.4. Based on the other comments today, this appears to be caused by OS X 10.9 on your machine. Unfortunately OpenCL code is compiled at runtime, and if the error occurs regardless of gcc or clang, it is probably either in the system header files or driver, which means it won’t even help to compile statically on a machine with OS X 10.10.
Unless anybody has any alternative idea, it sounds as if we’ll have to consider a runtime check to disable OpenCL for AMD on versions prior to 10.10.4.
Cheers,
Erik
From: Carlo Camilloni <carlo.camilloni at gmail.com>
Reply: Carlo Camilloni <carlo.camilloni at gmail.com>>
Date: 20 Jul 2015 at 19:50:26
To: Mark Abraham <mark.j.abraham at gmail.com>>
Cc: gmx-developers at gromacs.org <gmx-developers at gromacs.org>>, gromacs.org_gmx-developers at maillist.sys.kth.se <gromacs.org_gmx-developers at maillist.sys.kth.se>>, Erik Lindahl <erik.lindahl at gmail.com>>
Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman
Hi again,
I don’t know if you had the opportunity to reproduce this bug, in the meantime I have compiled it with gcc 4.8, openMP and sse4.1,
again cpu results are consistently right while OpenCL results are consistently wrong, so it looks like it is NOT a problem of Clang 6
but it could be either a problem of the gpu driver or maybe OpenCL itself, I don’t know..
this is the new log
GROMACS version: VERSION 5.1-rc1
Precision: single
Memory model: 64 bit
MPI library: thread_mpi
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 32)
GPU support: enabled
OpenCL support: enabled
invsqrt routine: gmx_software_invsqrt(x)
SIMD instructions: SSE4.1
FFT library: fftw-3.3.4-sse2-avx
RDTSCP usage: enabled
C++11 compilation: disabled
TNG support: enabled
Tracing support: disabled
Built on: Mon 20 Jul 2015 18:21:43 BST
Built by: carlo [CMAKE]
Build OS/arch: Darwin 13.4.0 x86_64
Build CPU vendor: GenuineIntel
Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
Build CPU family: 6 Model: 62 Stepping: 4
Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /opt/local/bin/gcc GNU 4.8.5
C compiler flags: -msse4.1 -Wextra -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall -Wno-unused -Wunused-value -Wunused-parameter -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast -Wno-array-bounds
C++ compiler: /opt/local/bin/g++ GNU 4.8.5
C++ compiler flags: -msse4.1 -Wextra -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast -Wno-array-bounds
Boost version: 1.58.0 (external)
OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
OpenCL library: /System/Library/Frameworks/OPENCL.framework
OpenCL version: 1.2
Best,
Carlo
On 17 Jul 2015, at 13:54, Carlo Camilloni <carlo.camilloni at gmail.com> wrote:
Hi,
Unfortunately on the Mac Pro I am running os x 10.9.5 and the newer clang is not yet available so I cannot test it
with a different version. Erik if you have time to tell me what to change to get gromacs compiled with gcc I can give it a try
Carlo
On 16 Jul 2015, at 20:28, Mark Abraham <mark.j.abraham at gmail.com> wrote:
Hi,
Philip Fowler made a recent positive report on OpenCL+AMD+Mac and he was using /usr/bin/cc Clang 6.1.0.6020053, so there may be an issue with Clang 6.0 (whatever real clang version that is...)
Mark
On Thu, Jul 16, 2015 at 7:20 PM Erik Lindahl <erik.lindahl at gmail.com> wrote:
Hi,
The one obvious difference I can see is that you have clang 6.0.0, while at least my Macbook pro says 6.1.0.
Could you check if there is any update available for the developer tools, and if so does that help?
It is actually much better to use gcc since you will then also use all CPU cores. However, that requires patching a header file in /usr/include; I’ll dig up what I did there - we need to put it in the documentation anyway.
Cheers,
Erik
From: Carlo Camilloni <carlo.camilloni at gmail.com>
Reply: Carlo Camilloni <carlo.camilloni at gmail.com>>
Date: 16 Jul 2015 at 19:07:13
To: Erik Lindahl <erik.lindahl at gmail.com>>
Cc: gromacs.org_gmx-developers at maillist.sys.kth.se <gromacs.org_gmx-developers at maillist.sys.kth.se>>, gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman
Hi Erik,
here it is:
GROMACS version: VERSION 5.1-rc1
Precision: single
Memory model: 64 bit
MPI library: none
OpenMP support: disabled
GPU support: enabled
OpenCL support: enabled
invsqrt routine: gmx_software_invsqrt(x)
SIMD instructions: AVX_256
FFT library: fftw-3.3.4-sse2-avx
RDTSCP usage: enabled
C++11 compilation: disabled
TNG support: enabled
Tracing support: disabled
Built on: Thu 16 Jul 2015 09:34:14 BST
Built by: carlo@ [CMAKE]
Build OS/arch: Darwin 13.4.0 x86_64
Build CPU vendor: GenuineIntel
Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
Build CPU family: 6 Model: 62 Stepping: 4
Build CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc Clang 6.0.0.6000057
C compiler flags: -mavx -Wall -Wno-unused -Wunused-value -Wunused-parameter -Wno-unknown-pragmas -O3 -DNDEBUG
C++ compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ Clang 6.0.0.6000057
C++ compiler flags: -mavx -Wextra -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function -Wno-unknown-pragmas -O3 -DNDEBUG
Boost version: 1.58.0 (external)
OpenCL include dir: /System/Library/Frameworks/OpenCL.framework
OpenCL library: /System/Library/Frameworks/OPENCL.framework
OpenCL version: 1.2
Running on 1 node with total 12 logical cores, 2 compatible GPUs
Hardware detected:
CPU info:
Vendor: GenuineIntel
Brand: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
Family: 6 model: 62 stepping: 4
CPU features: aes apic avx clfsh cmov cx8 cx16 f16c htt lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
SIMD instructions most likely to fit this hardware: AVX_256
SIMD instructions selected at GROMACS compile time: AVX_256
GPU info:
Number of GPUs detected: 2
#0: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD, device version: OpenCL 1.2 , stat: compatible
#1: name: AMD Radeon HD - FirePro D500 Compute Engine, vendor: AMD, device version: OpenCL 1.2 , stat: compatible
Carlo
On 16 Jul 2015, at 18:59, Erik Lindahl <erik.lindahl at gmail.com> wrote:
Hi Carlo,
IIRC, I managed to get it working on a Mac Pro with AMD Firepro D300 GPUs. Please post the contents of the top of your logfile (where it says everything about the compilers & config), and mention what hardware you tried it on - then we’ll see if we can reproduce it.
Cheers,
Erik
From: Carlo Camilloni <carlo.camilloni at gmail.com>
Reply: gmx-developers at gromacs.org <gmx-developers at gromacs.org>>
Date: 16 Jul 2015 at 18:52:21
To: gromacs.org_gmx-developers at maillist.sys.kth.se <gromacs.org_gmx-developers at maillist.sys.kth.se>>
Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with Parrinello-Rahman
Hi,
I tested the OpenCL kernel on my macbook (nvidia gpu) and here it produces the correct forces,
so it could be a problem related to amd+osx, or maybe to some specific compiler/os version
Carlo
> On 15 Jul 2015, at 17:42, Carlo Camilloni <carlo.camilloni at gmail.com> wrote:
>
> Hi,
>
> these are the tests that fail:
>
> FAILED. Check checkpot.out (12 errors), checkforce.out (3516 errors) file(s) in dd121 for dd121
> FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors) file(s) in nbnxn-energy-groups for nbnxn-energy-groups
> FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors) file(s) in nbnxn-free-energy for nbnxn-free-energy
> FAILED. Check checkpot.out (26 errors), checkforce.out (2998 errors) file(s) in nbnxn-free-energy-vv for nbnxn-free-energy-vv
> FAILED. Check checkpot.out (11 errors), checkforce.out (4039 errors) file(s) in nbnxn-ljpme-geometric for nbnxn-ljpme-geometric
> FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors) file(s) in nbnxn-ljpme-LB for nbnxn-ljpme-LB
> FAILED. Check checkpot.out (14 errors), checkforce.out (52 errors) file(s) in nbnxn-ljpme-LB-geometric for nbnxn-ljpme-LB-geometric
> FAILED. Check checkpot.out (10 errors), checkforce.out (4029 errors) file(s) in nbnxn-vdw-force-switch for nbnxn-vdw-force-switch
> FAILED. Check checkpot.out (10 errors), checkforce.out (4032 errors) file(s) in nbnxn-vdw-potential-switch for nbnxn-vdw-potential-switch
> FAILED. Check checkpot.out (4 errors), checkforce.out (250 errors) file(s) in nbnxn-vdw-potential-switch-argon for nbnxn-vdw-potential-switch-argon
> FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors) file(s) in nbnxn_pme for nbnxn_pme
> FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors) file(s) in nbnxn_pme_order5 for nbnxn_pme_order5
> FAILED. Check checkpot.out (10 errors), checkforce.out (4027 errors) file(s) in nbnxn_pme_order6 for nbnxn_pme_order6
> FAILED. Check checkpot.out (9 errors), checkforce.out (4028 errors) file(s) in nbnxn_rf for nbnxn_rf
> FAILED. Check checkpot.out (2 errors), checkforce.out (4 errors) file(s) in nbnxn_rzero for nbnxn_rzero
> FAILED. Check mdrun.out, md.log file(s) in nbnxn_vsite for nbnxn_vsite
> FAILED. Check checkpot.out (13 errors), checkforce.out (15512 errors) file(s) in octahedron for octahedron
> FAILED. Check mdrun.out, md.log file(s) in position-restraints for position-restraints
> FAILED. Check mdrun.out, md.log file(s) in pull_constraint for pull_constraint
> FAILED. Check checkpot.out (10 errors), checkforce.out (4021 errors) file(s) in pull_cylinder for pull_cylinder
> FAILED. Check checkpot.out (11 errors), checkforce.out (39054 errors) file(s) in swap_x for swap_x
> FAILED. Check checkpot.out (11 errors), checkforce.out (39053 errors) file(s) in swap_y for swap_y
> FAILED. Check checkpot.out (12 errors), checkforce.out (39054 errors) file(s) in swap_z for swap_z
> 23 out of 60 complex tests FAILED
> FAILED. Check mdrun.out, md.log file(s) in expanded for expanded
> FAILED. Check mdrun.out, md.log file(s) in transformAtoB for transformAtoB
> 2 out of 10 freeenergy tests FAILED
>
>
> Carlo
>
>
>>
>>
>> Message: 4
>> Date: Wed, 15 Jul 2015 15:35:13 +0000
>> From: Mark Abraham <mark.j.abraham at gmail.com>
>> To: gmx-developers at gromacs.org,
>> gromacs.org_gmx-developers at maillist.sys.kth.se
>> Subject: Re: [gmx-developers] gromacs 5.1rc1 OpenCL problem with
>> Parrinello-Rahman
>> Message-ID:
>> <CAMNuMATveVRRyBBwn312xrY+w3M7deC2Hs3A7PZnaeugkw+VVA at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi,
>>
>> Thanks. If a difference of that magnitude can be seen, then it should also
>> show up when running the regressiontests (e.g. cmake
>> -DREGRESSIONTEST_DOWNLOAD=on and then make check) as a failure
>> of complex/nbnxn-ljpme-LB (which is the only P-R test that can run on the
>> GPU). If other tests fail, then the problem is actually more widespread.
>>
>> It may be that there is some issue with some part of the Mac+clang+OpenCL
>> stack - we didn't target it during development, and at the last minute when
>> Erik was unexpectedly able to get it to compile. I don't know if he got
>> tests to pass. Erik?
>>
>> Mark
>>
>> On Wed, Jul 15, 2015 at 5:22 PM Carlo Camilloni <carlo.camilloni at gmail.com>
>> wrote:
>>
>>>
>>> Dear Mark and Szilard,
>>>
>>> thanks for your answer. I filed a bug in redmine but in the meantime I was
>>> running more tests and I am a bit scared by what I found:
>>>
>>> what I have done is the following I have performed a single step run with
>>> gmx51-rc1 compiled with cuda, again clang and so on
>>> and compared the forces on the first step with -nb cpu or not (I am using
>>> -pforce 1), the forces are identical:
>>>
>>> ie.:
>>>
>>> cuda-gpu
>>>
>>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
>>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
>>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
>>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
>>>
>>>
>>> cpu:
>>>
>>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90839e+03
>>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24845e+02
>>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89472e+02
>>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42053e+02
>>>
>>> if I do the same test on the version compiled with OpenCL
>>>
>>> cpu:
>>>
>>> (the former are done on my macbook pro avx2_256 this latter on a MacPro
>>> avx_256, this should
>>> explain the small differences in the forces)
>>>
>>> step 0 atom 1 x 3.940 5.612 2.226 force 1.90838e+03
>>> step 0 atom 2 x 3.852 5.659 2.211 force 4.24848e+02
>>> step 0 atom 3 x 3.979 5.665 2.303 force 6.89470e+02
>>> step 0 atom 4 x 3.992 5.610 2.139 force 7.42043e+02
>>>
>>> opencl-gpu:
>>> step 0 atom 1 x 3.940 5.612 2.226 force 1.48597e+03
>>> step 0 atom 2 x 3.852 5.659 2.211 force 6.26942e+02
>>> step 0 atom 3 x 3.979 5.665 2.303 force 8.44032e+02
>>> step 0 atom 4 x 3.992 5.610 2.139 force 7.92786e+02
>>>
>>> I am afraid there is something wrong in OpenCL kernels.
>>>
>>> I am using the topol-nvt-nogen.tpr I have uploaded on redmine.
>>>
>>> Best,
>>> Carlo
>>>
>>>
>>>
>>> --
>>> Gromacs Developers mailing list
>>>
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>>> posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>>> or send a mail to gmx-developers-request at gromacs.org.
>>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150715/fb1d4126/attachment.html>
>>
>> ------------------------------
>>
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or send a mail to gmx-developers-request at gromacs.org.
>>
>> End of gromacs.org_gmx-developers Digest, Vol 135, Issue 17
>> ***********************************************************
>
--
Gromacs Developers mailing list
* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before posting!
* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or send a mail to gmx-developers-request at gromacs.org.
--
Erik Lindahl <erik.lindahl at gmail.com>
Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University
Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal Inst. Technology
Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
--
Erik Lindahl <erik.lindahl at gmail.com>
Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University
Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal Inst. Technology
Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
--
Gromacs Developers mailing list
* Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before posting!
* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers or send a mail to gmx-developers-request at gromacs.org.
--
Erik Lindahl <erik.lindahl at gmail.com>
Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University
Professor of Theoretical biophysics, Dept. Theoretical Physics, Royal Inst. Technology
Science for Life Laboratory, Box 1031, 17121 Solna, Sweden
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20150720/71745e29/attachment-0001.html>
More information about the gromacs.org_gmx-developers
mailing list