[gmx-developers] cudaStreamSynchronize failed in cu_blockwait_nb

[Berk Hess]

Just curious, why are you running plain cut-off?
(I didn't even make CPU kernels for that, the RF kernels is then used)

Cheers,

Berk

On 10/22/2012 05:23 PM, Carsten Kutzner wrote:
> Hi Szilárd,
>
> thanks a lot for fixing it!
>
> Carsten
>
>
> On Oct 22, 2012, at 5:20 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
>
>> Hi,
>>
>> The CUDA plain cut-off kernel's pointer was incorrectly assigned (stupid copy-paste bug). Just pushed a bugfix: https://gerrit.gromacs.org/#/c/1553/
>>
>> Cheers,
>> --
>> Szilárd
>>
>>
>> On Fri, Oct 19, 2012 at 3:20 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
>> Hi,
>>
>> That sounds like a nasty bug that I have not seen for quite a while. This happens generally when some serious memory corruption puts the GPU in a "bad state". For the future, you could try to reset the GPU by reloading the driver, but if that does not help you will have to reboot.
>>
>> I was able to reproduce the bug and in fact on our development machine the NVIDIA driver seems to get into a messed up state in which mdrun will hang, no matter whether I launch in on the GTX 580 or 680. Reloading the driver seems to fix this issue.
>>
>> Thanks for the report, I'll looking into this bug and will give you an update!
>>
>> Cheers,
>> --
>> Szilárd
>>
>>
>>
>> On Fri, Oct 19, 2012 at 12:01 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>> Hi,
>>
>> we updated to the newest driver, but later I found that this crash is caused by
>> a .tpr file with Coulomb-type=cutoff instead of PME:
>>
>> - I start with a PME .tpr file that runs with the recent 4.6 on both a GTX580 and 680,
>>    and even using both
>> - I change to cutoff setting (no other changes!); this tpr still runs on the 580,
>>    but on the 680 produces the fatal error:
>>    "cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch failure"
>>    Moreover, after that any other mdrun using any GPU on that node will read in the
>>    previously working, PME) .tpr file and then hang. After rebooting, I can again
>>    run the PME .tpr file.
>>
>> Carsten
>>
>>
>>
>> On Oct 17, 2012, at 3:10 PM, Szilárd Páll <szilard.pall at cbr.su.se> wrote:
>>
>>> HI,
>>>
>>> Your driver might be simply too old for a GTX680. You'll need at least a very late 295.xx driver and preferably the 304.54 (or later).
>>>
>>> Cheers,
>>> --
>>> Szilárd
>>>
>>>
>>> On Wed, Oct 17, 2012 at 2:10 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>>> BTW this executable works on a GTX580, but shows the fatal error
>>> on a GTX680 - both mounted in the same workstation.
>>>
>>> Carsten
>>>
>>>
>>> On Oct 17, 2012, at 12:05 PM, Carsten Kutzner <ckutzne at gwdg.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> what am I doing wrong if I get this error code:
>>>>
>>>> -------------------------------------------------------
>>>> Program mdrun_threads, VERSION 4.6-dev-20121016-4af4561
>>>> Source code file: /home/ckutzne/installations/git-gromacs-4-6-department/src/mdlib/nbnxn_cuda/nbnxn_cuda.cu, line: 558
>>>>
>>>> Fatal error:
>>>> cudaStreamSynchronize failed in cu_blockwait_nb: unspecified launch failure
>>>>
>>>> For more information and tips for troubleshooting, please check the GROMACS
>>>> website at http://www.gromacs.org/Documentation/Errors
>>>> -------------------------------------------------------
>>>>
>>>> Thanks,
>>>> Carsten
>>>> --
>>>> gmx-developers mailing list
>>>> gmx-developers at gromacs.org
>>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>>> Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-developers-request at gromacs.org.
>>>
>>> --
>>> Dr. Carsten Kutzner
>>> Max Planck Institute for Biophysical Chemistry
>>> Theoretical and Computational Biophysics
>>> Am Fassberg 11, 37077 Goettingen, Germany
>>> Tel. +49-551-2012313, Fax: +49-551-2012302
>>> http://www.mpibpc.mpg.de/grubmueller/kutzner
>>>
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>>>
>>> --
>>> gmx-developers mailing list
>>> gmx-developers at gromacs.org
>>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>>> Please don't post (un)subscribe requests to the list. Use the
>>> www interface or send it to gmx-developers-request at gromacs.org.
>>
>> --
>> Dr. Carsten Kutzner
>> Max Planck Institute for Biophysical Chemistry
>> Theoretical and Computational Biophysics
>> Am Fassberg 11, 37077 Goettingen, Germany
>> Tel. +49-551-2012313, Fax: +49-551-2012302
>> http://www.mpibpc.mpg.de/grubmueller/kutzner
>>
>> --
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-developers-request at gromacs.org.
>>
>>
>> -- 
>> gmx-developers mailing list
>> gmx-developers at gromacs.org
>> http://lists.gromacs.org/mailman/listinfo/gmx-developers
>> Please don't post (un)subscribe requests to the list. Use the
>> www interface or send it to gmx-developers-request at gromacs.org.
>
> --
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics
> Am Fassberg 11, 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> http://www.mpibpc.mpg.de/grubmueller/kutzner
>