[gmx-developers] What is the best choice to profile

[Mark Abraham]

On Thu, Aug 7, 2014 at 9:50 PM, Theodore Si <sjyzhxw at gmail.com> wrote:

>  Hi Mark,
>
> Thanks for your attention. I have some questions for you.
>

That's nice, but you haven't answered my question, or responded to my
earlier suggestions for simplifying how you're trying to use VT to find
where the problem is. We're not black boxes with infinite time, here ;-)

1. Profilers that just need to compile with standard profiling flags are
> fine - just configure with -DCMAKE_BUILD_TYPE=Profile and go for it.
> It seems that this is convenient. What profilers can be used in this way? What
> tools do you guys use to profile?
>

I don't know. We use none routinely, for the aforementioned reasons.
They're hard to get to build, run without perturbation, and analyze the
results. Some of that is intrinsic to the problem space. There's a project
making slow progress to integrate some support for more routine use of
Extrae.

2. If you need to pass things to the wrapper compiler, make yourself a
> script to wrap the wrapper compiler and give that to CMake.
> I think it's the case when I compile GMX with Vampir Trace. We need to use
> the command vtcc -vt:cc mpicc sourcefile.c -o a.out to instrument MPI
> code. Should "-vt:cc mpicc" be passed to vtcc, which is set to
> CMAKE_C_COMPILER, as argument? I don't know how to "make yourself a script
> to wrap the wrapper compiler and give that to CMake". How to give a script
> to CMake exactly? I have not used CMake before, please give me some
> instructions.
>

A compiler is a command you run that expects a bunch of flags and input
files to follow. So you just make a shell script that conforms to that
interface, so that CMake can be given a script and it will follow the rules
it expects compilers to follow. So something like

#!/bin/sh
vtcc -vt:cc mpicc $@

in myvtcc.sh could be passed to cmake with

cmake .. -DCMAKE_C_COMPILER=/path/to/myvtcc.sh

You'd have to do exactly the same kind of thing to hack a Makefile or a
command line - you need to replace all calls to the actual compiler with
"vtcc -vt:cc mpicc" which is what this shell script is doing in a way that
makes it easy for CMake not to have to be robust against spaces and quoting.


> When I use Vampir Trace, some weird errors occur:
>
> 1. In some source files, some multiple-line comments are processed by the
> compiler. If I change them to one-line comments, it works fine. I have no
> idea why that happens.
>

No idea, but someone's not conforming to a language spec. Given that a wide
range of compilers are happy with those input files, I'm guessing VT's
doing some parsing that isn't implemented quite well enough. You can avoid
those particular problems by configuring gromacs without TNG support, e.g.
cmake -DGMX_USE_TNG=off


> 2. If I use the following configuration:
> cmake .. -DCMAKE_INSTALL_PREFIX=$MYPRG/gmx -DCU
> DA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DCUDA_CUDA_LIBRARY=/usr/lib64/libcuda.so
> -DCMAKE_C_COMPILER=vtcc -DCMAKE_CXX_COMPILER=vtcxx -DCUDA_NVCC_EXECUTABLE=vtnvcc
> -DGMX_MPI=ON -DGMX_BUILD_OWN_FFTW=ON
> The compilation can be done. And when I run grompp to generate tpr, it's
> OK and Vampir Trace will generate trace files. When I run mdrun without
> parameters, it also generate trace files. However, when I actually start
> mdrun with a tpr file, it will give a segmentation fault.
>

I made suggestions here earlier.

Mark


>
>
> On 8/7/2014 10:56 PM, Mark Abraham wrote:
>
> Hi,
>
>  First, what are you hoping to learn? Many questions have a known answer,
> or are known not to have a clear answer ;-)
>
>  It's a compound problem. Profilers that just need to compile with
> standard profiling flags are fine - just configure with
> -DCMAKE_BUILD_TYPE=Profile and go for it. Those that need to influence the
> compilation or linker line are more problematic. Just passing the (full
> path to) the wrapper compiler should work fine. If you need to pass things
> to the wrapper compiler, make yourself a script to wrap the wrapper
> compiler and give that to CMake. Whether the things the tool's wrapper
> compiler does clashes with things GROMACS is doing varies a lot, and you
> will need to get involved in the details to see what the origin of any
> problems are. That's not anybody's fault, per se, but the writer of a
> wrapper compiler typically hopes the end user is not managing details
> themselves, but to get high performance you often have to manage details,
> and some of those details show up on the compiler command lines generated
> by the GROMACS build system. And naturally you'll be interested in
> MPI+OpenMP+CUDA, each of which compounds the problem with further wrapper
> compilers or command-line stuff.
>
>  Once it's working, you have the problem of whether you can get useful
> data. Instrumenting every function call, or compromising function inlining
> is guaranteed to be useless because that overhead kills things. The main
> interesting case to profile is where the MD step iteration time is a few
> milliseconds, and you can't introduce thousands of increments of tens of
> nanoseconds and get sensible profiles. So either you have to restrict the
> instrumentation to high-level functions (which is painful; the output at
> the end of the GROMACS log file is a coarse version of this averaged over
> many steps and execution contexts), or use a sampling-based approach.
>
>  Then you need to start collecting data after the run-time performance
> tuning that mdrun does has already stabilized - at least a few hundred MD
> steps. Longer if the MD load is imbalanced, which is also the main
> interesting case to consider for code modifications.
>
>  Mark
>
>
> On Wed, Aug 6, 2014 at 10:58 AM, Theodore Si <sjyzhxw at gmail.com> wrote:
>
>> what kind of tools work best with gromacs 5.0?
>>
>> --
>> Gromacs Developers mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
>> or send a mail to gmx-developers-request at gromacs.org.
>>
>
>
>
>
>
> --
> Gromacs Developers mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers
> or send a mail to gmx-developers-request at gromacs.org.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20140808/57aeb6e9/attachment-0001.html>