[gmx-users] GMX 2018 regression tests: cufftPlanMany R2C plan failure (error code 5)

[Peter Kroon]

Jup, start with rebooting before trying anything else. There's probably
still old drivers loaded in the kernel.


Peter


On 08-02-18 14:14, Alex wrote:
> Mark and Peter,
>
> Thanks for commenting. I was told that all CUDA tests passed, but I
> will double check on how many of those were actually run. Also, we
> never rebooted the box after CUDA install, and finally we had a bunch
> of gromacs (2016.4) jobs running, because we didn't want to interrupt
> postdoc's work... All of those were with -nb cpu though. Could those
> factors have affected our regression tests?
>
> It will really suck, if these are hardware-related...
>
> Thanks,
>
> Alex
>
>
> On 2/8/2018 3:03 AM, Mark Abraham wrote:
>> Hi,
>>
>> Or leftovers of the drivers that are now mismatching. That has caused
>> timeouts for us.
>>
>> Mark
>>
>> On Thu, Feb 8, 2018 at 10:55 AM Peter Kroon <p.c.kroon at rug.nl> wrote:
>>
>>> Hi,
>>>
>>>
>>> with changing failures like this I would start to suspect the hardware
>>> as well. Mark's suggestion of looking at simpler test programs than GMX
>>> is a good one :)
>>>
>>>
>>> Peter
>>>
>>>
>>> On 08-02-18 09 <08-02%2018%2009>:10, Mark Abraham wrote:
>>>> Hi,
>>>>
>>>> That suggests that your new CUDA installation is differently
>>>> incomplete.
>>> Do
>>>> its samples or test programs run?
>>>>
>>>> Mark
>>>>
>>>> On Thu, Feb 8, 2018 at 1:20 AM Alex <nedomacho at gmail.com> wrote:
>>>>
>>>>> Update: we seem to have had a hiccup with an orphan CUDA install and
>>> that
>>>>> was causing issues. After wiping everything off and rebuilding the
>>> errors
>>>>> from the initial post disappeared. However, two tests failed during
>>>>> regression:
>>>>>
>>>>> 95% tests passed, 2 tests failed out of 39
>>>>>
>>>>> Label Time Summary:
>>>>> GTest              = 170.83 sec (33 tests)
>>>>> IntegrationTest    = 125.00 sec (3 tests)
>>>>> MpiTest            =   4.90 sec (3 tests)
>>>>> UnitTest           =  45.83 sec (30 tests)
>>>>>
>>>>> Total Test time (real) = 1225.65 sec
>>>>>
>>>>> The following tests FAILED:
>>>>>    9 - GpuUtilsUnitTests (Timeout)
>>>>> 32 - MdrunTests (Timeout)
>>>>> Errors while running CTest
>>>>> CMakeFiles/run-ctest-nophys.dir/build.make:57: recipe for target
>>>>> 'CMakeFiles/run-ctest-nophys' failed
>>>>> make[3]: *** [CMakeFiles/run-ctest-nophys] Error 8
>>>>> CMakeFiles/Makefile2:1160: recipe for target
>>>>> 'CMakeFiles/run-ctest-nophys.dir/all' failed
>>>>> make[2]: *** [CMakeFiles/run-ctest-nophys.dir/all] Error 2
>>>>> CMakeFiles/Makefile2:971: recipe for target
>>>>> 'CMakeFiles/check.dir/rule'
>>>>> failed
>>>>> make[1]: *** [CMakeFiles/check.dir/rule] Error 2
>>>>> Makefile:546: recipe for target 'check' failed
>>>>> make: *** [check] Error 2
>>>>>
>>>>> Any ideas? I can post the complete log, if needed.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Alex
>>>>> -- 
>>>>> Gromacs Users mailing list
>>>>>
>>>>> * Please search the archive at
>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>>>> posting!
>>>>>
>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>>>
>>>>> * For (un)subscribe requests visit
>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>>>> send a mail to gmx-users-request at gromacs.org.
>>>>>
>>>
>>> -- 
>>> Gromacs Users mailing list
>>>
>>> * Please search the archive at
>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>>> posting!
>>>
>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>>
>>> * For (un)subscribe requests visit
>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>>> send a mail to gmx-users-request at gromacs.org.
>