[gmx-users] GMX 2018 regression tests: cufftPlanMany R2C plan failure (error code 5)

[Mark Abraham]

On Thu, Feb 8, 2018 at 6:54 PM Szilárd Páll <pall.szilard at gmail.com> wrote:

> BTW, do you have persistence mode (PM) set (see in the nvidia-smi output)?
> If you do not have PM it set nor is there an X server that keeps the driver
> loaded, the driver gets loaded every time a CUDA application is started.
> This could be causing the lag which shows up as long execution time for our
> rather simple unit tests that should take milliseconds rather than seconds
> when PM is on (or X is running).
>

The earlier report of bin/gpu_utils-test showed that many tests were slow,
not just at startup. See immediately below

Mark
 > > > > > > > Here you are:

> >> > > > > > >
> >> > > > > > > [==========] Running 35 tests from 7 test cases.
> >> > > > > > > [----------] Global test environment set-up.
> >> > > > > > > [----------] 7 tests from HostAllocatorTest/0, where
> >> TypeParam =
> >> > > int
> >> > > > > > > [ RUN      ] HostAllocatorTest/0.EmptyMemoryAlwaysWorks
> >> > > > > > > [       OK ] HostAllocatorTest/0.EmptyMemoryAlwaysWorks
> (5457
> >> > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/0.VectorsWit
> >> hDefaultHostAllocato
> >> > > > > > rAlwaysWorks
> >> > > > > > > [       OK ] HostAllocatorTest/0.VectorsWit
> >> hDefaultHostAllocato
> >> > > > > > rAlwaysWorks
> >> > > > > > > (2861 ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/0.TransfersWithoutPinningWork
> >> > > > > > > [       OK ] HostAllocatorTest/0.TransfersWithoutPinningWork
> >> > (3254
> >> > > > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/0.FillInputA
> >> lsoWorksAfterCalling
> >> > > > Reserve
> >> > > > > > > [       OK ] HostAllocatorTest/0.FillInputA
> >> lsoWorksAfterCalling
> >> > > > Reserve
> >> > > > > > > (2221 ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/0.TransfersW
> >> ithPinningWorkWithCu
> >> > da
> >> > > > > > > [       OK ] HostAllocatorTest/0.TransfersW
> >> ithPinningWorkWithCu
> >> > da
> >> > > > (3801
> >> > > > > > ms)
> >> > > > > > > [ RUN      ]
> >> > > HostAllocatorTest/0.ManualPinningOperationsWorkWithCuda
> >> > > > > > > [       OK ]
> >> > > HostAllocatorTest/0.ManualPinningOperationsWorkWithCuda
> >> > > > > > (2157
> >> > > > > > > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/0.StatefulAllocatorUsesMemory
> >> > > > > > > [       OK ] HostAllocatorTest/0.StatefulAllocatorUsesMemory
> >> > (2179
> >> > > > ms)
> >> > > > > > > [----------] 7 tests from HostAllocatorTest/0 (21930 ms
> total)
> >> > > > > > >
> >> > > > > > > [----------] 7 tests from HostAllocatorTest/1, where
> >> TypeParam =
> >> > > > float
> >> > > > > > > [ RUN      ] HostAllocatorTest/1.EmptyMemoryAlwaysWorks
> >> > > > > > > [       OK ] HostAllocatorTest/1.EmptyMemoryAlwaysWorks
> (2739
> >> > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/1.VectorsWit
> >> hDefaultHostAllocato
> >> > > > > > rAlwaysWorks
> >> > > > > > > [       OK ] HostAllocatorTest/1.VectorsWit
> >> hDefaultHostAllocato
> >> > > > > > rAlwaysWorks
> >> > > > > > > (2731 ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/1.TransfersWithoutPinningWork
> >> > > > > > > [       OK ] HostAllocatorTest/1.TransfersWithoutPinningWork
> >> > (3281
> >> > > > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/1.FillInputA
> >> lsoWorksAfterCalling
> >> > > > Reserve
> >> > > > > > > [       OK ] HostAllocatorTest/1.FillInputA
> >> lsoWorksAfterCalling
> >> > > > Reserve
> >> > > > > > > (2164 ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/1.TransfersW
> >> ithPinningWorkWithCu
> >> > da
> >> > > > > > > [       OK ] HostAllocatorTest/1.TransfersW
> >> ithPinningWorkWithCu
> >> > da
> >> > > > (3908
> >> > > > > > ms)
> >> > > > > > > [ RUN      ]
> >> > > HostAllocatorTest/1.ManualPinningOperationsWorkWithCuda
> >> > > > > > > [       OK ]
> >> > > HostAllocatorTest/1.ManualPinningOperationsWorkWithCuda
> >> > > > > > (2202
> >> > > > > > > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/1.StatefulAllocatorUsesMemory
> >> > > > > > > [       OK ] HostAllocatorTest/1.StatefulAllocatorUsesMemory
> >> > (2261
> >> > > > ms)
> >> > > > > > > [----------] 7 tests from HostAllocatorTest/1 (19287 ms
> total)
> >> > > > > > >
> >> > > > > > > [----------] 7 tests from HostAllocatorTest/2, where
> >> TypeParam =
> >> > > > > > > gmx::BasicVector<float>
> >> > > > > > > [ RUN      ] HostAllocatorTest/2.EmptyMemoryAlwaysWorks
> >> > > > > > > [       OK ] HostAllocatorTest/2.EmptyMemoryAlwaysWorks
> (2771
> >> > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/2.VectorsWit
> >> hDefaultHostAllocato
> >> > > > > > rAlwaysWorks
> >> > > > > > > [       OK ] HostAllocatorTest/2.VectorsWit
> >> hDefaultHostAllocato
> >> > > > > > rAlwaysWorks
> >> > > > > > > (2846 ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/2.TransfersWithoutPinningWork
> >> > > > > > > [       OK ] HostAllocatorTest/2.TransfersWithoutPinningWork
> >> > (3283
> >> > > > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/2.FillInputA
> >> lsoWorksAfterCalling
> >> > > > Reserve
> >> > > > > > > [       OK ] HostAllocatorTest/2.FillInputA
> >> lsoWorksAfterCalling
> >> > > > Reserve
> >> > > > > > > (2131 ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/2.TransfersW
> >> ithPinningWorkWithCu
> >> > da
> >> > > > > > > [       OK ] HostAllocatorTest/2.TransfersW
> >> ithPinningWorkWithCu
> >> > da
> >> > > > (3833
> >> > > > > > ms)
> >> > > > > > > [ RUN      ]
> >> > > HostAllocatorTest/2.ManualPinningOperationsWorkWithCuda
> >> > > > > > > [       OK ]
> >> > > HostAllocatorTest/2.ManualPinningOperationsWorkWithCuda
> >> > > > > > (2232
> >> > > > > > > ms)
> >> > > > > > > [ RUN      ] HostAllocatorTest/2.StatefulAllocatorUsesMemory
> >> > > > > > > [       OK ] HostAllocatorTest/2.StatefulAllocatorUsesMemory
> >> > (2164
> >> > > > ms)
> >> > > > > > > [----------] 7 tests from HostAllocatorTest/2 (19261 ms
> total)
> >> > > > > > >
> >> > > > > > > [----------] 3 tests from AllocatorTest/0, where TypeParam =
> >> > > > > > > gmx::Allocator<float, gmx::HostAllocationPolicy>
> >> > > > > > > [ RUN      ] AllocatorTest/0.AllocatorAlignAllocatesWithAli
> >> > gnment
> >> > > > > > > [       OK ] AllocatorTest/0.AllocatorAlignAllocatesWithAli
> >> > gnment
> >> > > (0
> >> > > > > ms)
> >> > > > > > > [ RUN      ] AllocatorTest/0.VectorAllocatesAndResizesWithA
> >> > lignment
> >> > > > > > > [       OK ] AllocatorTest/0.VectorAllocatesAndResizesWithA
> >> > lignment
> >> > > > (0
> >> > > > > > ms)
> >> > > > > > > [ RUN      ]
> >> > > AllocatorTest/0.VectorAllocatesAndReservesWithAlignment
> >> > > > > > > [       OK ]
> >> > > AllocatorTest/0.VectorAllocatesAndReservesWithAlignment
> >> > > > (0
> >> > > > > > ms)
> >> > > > > > > [----------] 3 tests from AllocatorTest/0 (0 ms total)
> >> > > > > > >
> >> > > > > > > [----------] 3 tests from AllocatorTest/1, where TypeParam =
> >> > > > > > > gmx::Allocator<int, gmx::HostAllocationPolicy>
> >> > > > > > > [ RUN      ] AllocatorTest/1.AllocatorAlignAllocatesWithAli
> >> > gnment
> >> > > > > > > [       OK ] AllocatorTest/1.AllocatorAlignAllocatesWithAli
> >> > gnment
> >> > > (0
> >> > > > > ms)
> >> > > > > > > [ RUN      ] AllocatorTest/1.VectorAllocatesAndResizesWithA
> >> > lignment
> >> > > > > > > [       OK ] AllocatorTest/1.VectorAllocatesAndResizesWithA
> >> > lignment
> >> > > > (0
> >> > > > > > ms)
> >> > > > > > > [ RUN      ]
> >> > > AllocatorTest/1.VectorAllocatesAndReservesWithAlignment
> >> > > > > > > [       OK ]
> >> > > AllocatorTest/1.VectorAllocatesAndReservesWithAlignment
> >> > > > (0
> >> > > > > > ms)
> >> > > > > > > [----------] 3 tests from AllocatorTest/1 (0 ms total)
> >> > > > > > >
> >> > > > > > > [----------] 3 tests from AllocatorTest/2, where TypeParam =
> >> > > > > > > gmx::Allocator<gmx::BasicVector<float>,
> >> > gmx::HostAllocationPolicy>
> >> > > > > > > [ RUN      ] AllocatorTest/2.AllocatorAlignAllocatesWithAli
> >> > gnment
> >> > > > > > > [       OK ] AllocatorTest/2.AllocatorAlignAllocatesWithAli
> >> > gnment
> >> > > (0
> >> > > > > ms)
> >> > > > > > > [ RUN      ] AllocatorTest/2.VectorAllocatesAndResizesWithA
> >> > lignment
> >> > > > > > > [       OK ] AllocatorTest/2.VectorAllocatesAndResizesWithA
> >> > lignment
> >> > > > (0
> >> > > > > > ms)
> >> > > > > > > [ RUN      ]
> >> > > AllocatorTest/2.VectorAllocatesAndReservesWithAlignment
> >> > > > > > > [       OK ]
> >> > > AllocatorTest/2.VectorAllocatesAndReservesWithAlignment
> >> > > > (0
> >> > > > > > ms)
> >> > > > > > > [----------] 3 tests from AllocatorTest/2 (0 ms total)
> >> > > > > > >
> >> > > > > > > [----------] 5 tests from PinnedMemoryCheckerTest
> >> > > > > > > [ RUN      ] PinnedMemoryCheckerTest.
> >> > DefaultContainerIsRecognized
> >> > > > > > > [       OK ] PinnedMemoryCheckerTest.
> >> > DefaultContainerIsRecognized
> >> > > > (3248
> >> > > > > > ms)
> >> > > > > > > [ RUN      ] PinnedMemoryCheckerTest.
> >> > NonpinnedContainerIsRecognized
> >> > > > > > > [       OK ] PinnedMemoryCheckerTest.
> >> > NonpinnedContainerIsRecognized
> >> > > > > > (2752
> >> > > > > > > ms)
> >> > > > > > > [ RUN      ] PinnedMemoryCheckerTest.Pinned
> >> ContainerIsRecognized
> >> > > > > > > [       OK ] PinnedMemoryCheckerTest.Pinned
> >> ContainerIsRecognized
> >> > > > (2709
> >> > > > > > ms)
> >> > > > > > > [ RUN      ] PinnedMemoryCheckerTest.Defaul
> >> tCBufferIsRecognized
> >> > > > > > > [       OK ] PinnedMemoryCheckerTest.Defaul
> >> tCBufferIsRecognized
> >> > > > (2675
> >> > > > > > ms)
> >> > > > > > > [ RUN      ] PinnedMemoryCheckerTest.Pinned
> >> CBufferIsRecognized
> >> > > > > > > [       OK ] PinnedMemoryCheckerTest.Pinned
> >> CBufferIsRecognized
> >> > > (2686
> >> > > > > ms)
> >> > > > > > > [----------] 5 tests from PinnedMemoryCheckerTest (14071 ms
> >> > total)
> >> > > > > > >
> >> > > > > > > [----------] Global test environment tear-down
> >> > > > > > > [==========] 35 tests from 7 test cases ran. (74549 ms
> total)
> >> > > > > > > [  PASSED  ] 35 tests.
>