[gmx-developers] False positives in regression tests for GROMACS 2020.3 and Intel MPI 2019.8

[Christoph Pospiech]

Hi,

I saw two cases recently where regression tests for GROMACS 2020.3 and Intel 
MPI 2019.8 failed. I guess these are "false failures". In both cases, the 
failures are only happening in the following sections.
* complex
* freeenergy
* rotation
In all three sections many if not all sub test are reported as failure.
All other tests (53 out of 56) tests pass.

In one of the two cases I could work around the problem by calling
"unset  I_MPI_ROOT_modshare" just before starting the build process with 
cmake. Having deleted this environment variable, all 56 regression tests 
passed. The difference in the output for one of the regression tests looks like 
this.

diff -u regressiontests-release-2020-67a5ed7*/complex/acetonitrilRF/grompp.out
--- regressiontests-release-2020-67a5ed7/complex/acetonitrilRF/grompp.out      
2020-12-07 06:34:16.987193129 -0600
+++ regressiontests-release-2020-67a5ed7_fail/complex/acetonitrilRF/grompp.out 
2020-12-07 06:12:13.522192066 -0600
@@ -1,3 +1,5 @@
+[0] MPI startup(): I_MPI_ROOT_modshare environment variable is not supported.
+[0] MPI startup(): To check the list of supported variables, use the mpi_info 
utility or refer to https://software.intel.com/en-us/mpi-library/
documentation/get-started.
 turning all bonds into constraints...
 Analysing residue names:
 There are:   526      Water residues

I took more diffs in other sub directories of  regressiontests-
release-2020-67a5ed7, and all had the same two extra lines. However, I didn't 
check *all* sub directories, though. It appears that the failures are related 
to warning messages issued by Intel MPI 2019.8.  These offending lines seem 
always to start with "[0] MPI startup():". Is it possible to tell the 
regression test utility to ignore all output lines starting with "[0] MPI 
startup():"?

I would be volunteering to test whether this helps, if I am told where to put 
this extra if statement into the code of the unit test utility.
-- 
Dr. Christoph Pospiech
Senior HPC & AI Performance Engineer

T +49 (351) 86269826
M +49 (171) 7655871
E cpospiech at lenovo.com

Lenovo Global Technology (Deutschland) GmbH
Meitnerstr. 9
70563 Stuttgart

Geschäftsführung: Christophe Philippe Marie Laurent und Colm Brendan Gleeson
(jeweils einzelvertretungsberechtigt)
Prokura: Dieter Stehle & Henrik Bächle (Einzelprokura)                                        
Sitz der Gesellschaft: Stuttgart                                                             
HRB-Nr.: 758298, AG Stuttgart
WEEE-Reg.-Nr.: DE79679404