[gmx-users] Why does the -append option exist?

[Mark Abraham]

On 5/06/2011 12:31 PM, Dimitar Pachov wrote:
>
>     This script is not using mdrun -append.
>
>
> -append is the default, it doesn't need to be explicitly listed.

Ah yes, very true.
>
>     Your original post suggested the use of -append was a problem. Why
>     aren't we seeing a script with mdrun -append? Also, please provide
>     the full script - it looks like there might be a loop around your
>     tpbconv-then-mdrun fragment.
>
>
> There is no loop; this is a job script with PBS directives. The header 
> of it looks like:
> ===========================
> #!/bin/bash
> #$ -S /bin/bash
> #$ -pe mpich 8
> #$ -ckpt reloc
> #$ -l mem_total=6G
> ===========================
>
> as usual submitted by:
>
> qsub -N aaaa myjob.q
>
>
>     Note that a useful trouble-shooting technique can be to construct
>     your command line in a shell variable, echo it to stdout
>     (redirected as suitable) and then execute the contents of the
>     variable. Now, nobody has to parse a shell script to know what
>     command line generated what output, and it can be co-located with
>     the command's stdout.
>
>
> I somewhat understand your point, but could give an example if you 
> think it is really necessary?

It's just generally helpful if your stdout has "mpirun -np 8 
/path/to/mdrun_mpi -deffnm run_4 -cpi run_4" at the top of it so that 
you have a definitive record of what you did under the environment that 
existed at the time of execution.

> As I said, the queue is like this: you submit the job, it finds an 
> empty node, it goes there, however seconds later another user with 
> higher privileges on that particular node submits a job, his job kicks 
> out my job, mine goes on the queue again, it finds another empty node, 
> goes there, then another user with high privileges on that node 
> submits a job, which consequently kicks out my job again, and the 
> cycle repeats itself ... theoretically, it could continue forever, 
> depending on how many and where the empty nodes are, if any.

You've said that *now* - but previously you've said nothing about why 
you were getting lots of restarts. In my experience, PBS queues suspend 
jobs rather than deleting them, in order that resources are not wasted. 
Apparently other places do things this way. I think that this 
information is highly relevant to explaining your observations.

> These many restarts suggest that the queue was full with relatively 
> short jobs ran by users with high privileges. Technically, I cannot 
> see why the same processes should be running simultaneously because at 
> any instant my job runs only on one node, or it stays in 
> the queuing list.

I/O can be buffered such that the termination of the process and the 
completion of its I/O are asynchronous. Perhaps it *shouldn't* be that 
way, but this is a problem for the administrators of your cluster to 
address. They know how the file system works. If the next job executes 
before the old one has finished output, then I think the symptoms you 
observe might be possible.

Note that there is nothing GROMACS can do about that, unless somehow 
GROMACS can apply a lock in the first mdrun that is respected by your 
file system such that a subsequent mdrun cannot open the same file until 
all pending I/O has completed. I'd expect proper HPC file systems do 
that automatically, but I don't really know.

>>
>>     From md-1-2360.out:
>>     =====================================
>>     :::::::
>>     Getting Loaded...
>>     Reading file run1.tpr, VERSION 4.5.4 (single precision)
>>
>>     Reading checkpoint file run1.cpt generated: Tue May 31 10:45:22 2011
>>
>>
>>     Loaded with Money
>>
>>     Making 2D domain decomposition 4 x 2 x 1
>>
>>     WARNING: This run will generate roughly 4915 Mb of data
>>
>>     starting mdrun 'run1'
>>     100000000 steps, 200000.0 ps (continuing from step 51879590,
>>     103759.2 ps).
>>     =====================================
>
>     These aren't showing anything other than that the restart is
>     coming from the same point each time.
>
>
>>     And from the last generated output md-1-2437.out (I think I
>>     killed the job at that point because of the above observed behavior):
>>     =====================================
>>     :::::::
>>     Getting Loaded...
>>     Reading file run1.tpr, VERSION 4.5.4 (single precision)
>>     =====================================
>>
>>     I have at least 5-6 additional examples like this one. In some of
>>     them the *xtc file does have size greater than zero yet still
>>     very small, but it starts from some random frame (for example, in
>>     one of the cases it contains frames from ~91000ps to ~104000ps,
>>     but all frames before 91000ps are missing).
>
>     I think that demonstrating a problem requires that the set of
>     output files were fine before one particular restart, and weird
>     afterwards. I don't think we've seen that yet.
>
>
> I don't understand your point here. I am providing you with all info I 
> have. I am showing the output files of 3 restarts, and they are 
> different in a sense that the last two did not progress further enough 
> before another job restart occurred. The first was fine before the 
> restart, and the others were not exactly fine after the restart. At 
> this point I realize that what I call "restart" and what you call 
> "restart" might be two different things. And here is where the problem 
> might be lying.
>
>
>>     I realize there might be another problem, but the bottom line is
>>     that there is no mechanism that can prevent this from happening
>>     if many restarts are required, and particularly if the timing
>>     between these restarts is prone to be small (distributed
>>     computing could easily satisfy this condition).
>>
>>     Any suggestions, particularly related to the minimum resistance
>>     path to regenerate the missing data? :)
>>
>>
>>>         Using the checkpoint capability & appending make sense when
>>>         many restarts are expected, but unfortunately it is exactly
>>>         then when these options completely fail! As a new user of
>>>         Gromacs, I must say I am disappointed, and would like to
>>>         obtain an explanation of why the usage of these options is
>>>         clearly stated to be safe when it is not, and why the append
>>>         option is the default, and why at least a single warning has
>>>         not been posted anywhere in the docs & manuals?
>>
>>         I can understand and sympathize with your frustration if
>>         you've experienced the loss of a simulation. Do be careful
>>         when suggesting that others' actions are blame-worthy, however.
>>
>>
>>     I have never suggested this. As a user, I am entitled to ask.
>
>     Sure. However, talking about something that can "completely fail"
>
>
> This is a fact, backed up by my evidences => I don't see anything bad 
> directed to anybody.
>
>     which makes you "disappointed"
>
>
> This is me being honest => again not related to anybody else.
>
>     and wanting to "obtain an explanation"
>
>
> Well, this even is funny :) - many people want this, especially in 
> science. Is that bad?
>
>     about why something doesn't work as stated and lacks "a single
>     warning"
>
>
> Again a fact => again nothing bad here.
>
>     suggests that someone has done something less than appropriate
>
>
> This is a completely personal interpretation, and I am personally not 
> responsible of how people perceive information. For unknown to me 
> reason you moved into a very defensive mode. What could I do?
>
>     , and so blame-worthy. It also assumes that the actions of a new
>     user were correct, and the actions of a developer with long
>     experience were not.
>
>
> Sorry, this is too much. Where was this suggested? It seems to me you 
> took it too personally.
>
>     This may or may not prove to be true. Starting such a discussion
>     from a conciliatory (rather than antagonistic) stance is usually
>     more productive. The shared objective should be to fix the
>     problem, not prove that someone did something wrong.
>
>
> Agree, and I did it. Again, your perception does not seem to be 
> correlated with my intended approach.

Words are open to interpretation. Communicating well requires that you 
consider the impact of your words on your reader. You want people who 
can address the problem to want to help. You don't want them to feel 
defensive about the situation - whether you think that would be an 
over-reaction or not.

>
>     An alternative way of wording your paragraph could have been:
>     "Using the checkpoint capability & appending make sense when many
>     restarts are expected, however I observe that under such
>     circumstances this capability can fail. I am a new user of
>     GROMACS, might I have been using them incorrectly? Are the
>     developers aware of any situations under which the capability is
>     unreliable? If so, should the default behaviour be different, and
>     should this issue be documented somewhere?"
>
>
> This is helpful, but again a bit too much. I don't tell you how to 
> write, please do the same.

OK, but how the tone of how you write determines whether people will 
respond, no matter how important your message.

> Moreover, how could I ask questions the answers to which were mostly 
> known to me before sending my post?

These are the same ideas about which you asked in your original post.

>>     And since my questions were not clearly answered, I will repeat
>>     them in a structured way:
>>
>>     1. Why is the usage of these options (-cpi and -append) clearly
>>     stated to be safe when in fact it is not?
>
>     Because they are believed to be safe. Jussi's suggestion about
>     file locking may have merit.
>
>
>>     2. Why have you made the -append option the default in the most
>>     current GMX versions?
>
>     Because it's the most convenient mode of operation.
>
>
>
>>     3. Why has not a single warning been posted anywhere in the docs
>>     & manuals? (this question is somewhat clear - because you did not
>>     know about such a problem, but people say "ignorance of the law
>>     excuses no one", which means ignoring to put a warning for
>>     something that you were not 100% certain it would be error-free
>>     could not be an excuse)
>
>     Because no-one is aware of a problem to warn about.
>
>
> No, people are aware, they just do not think it is a problem, because 
> there is an easy work-around (-noappend), although not 
> as convenient and clean. Ask users of the Condor distributed grid 
> using Gromacs.

You asked why there was no warning in the documentation - that's because 
no-one who can fix the documentation is aware of a problem. If 
Condor-using people want to keep using a work-around and not 
communicate, that's their prerogative. But if the issue isn't 
communicated, then it isn't going to be documented, whether it's a real 
issue or not.

>>     I am blame-worthy - for blindly believing what was written in the
>>     manual without taking the necessary precautions. Lesson learned.
>>
>>         However, developers' time rarely permits addressing "feature
>>         X doesn't work, why not?" in a productive way. Solving bugs
>>         can be hard, but will be easier (and solved faster!) if the
>>         user who thinks a problem exists follows good procedure. See
>>         http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>>         <http://www.chiark.greenend.org.uk/%7Esgtatham/bugs.html>
>>
>>
>>     Implying that I did not follow a certain procedure related to a
>>     certain problem without you knowing what my initial intention was
>>     is just a speculation.
>
>     I don't follow your point. If your intent is to get the problem
>     being fixed, the advice on that web page is useful.
>
>
> My intend was clearly stated before, but for the sake of 
> clarification, let's repeat it again:
>
> 1. To let you know about the existence of such a problem.

Great. So far, I think it's an artefact of the combination of your PBS 
and file system configuration.

> 2. To find out why I encountered the problem, although I have read and 
> followed all of the Gromacs documentation related to the used by me 
> features.

As above - it's not really the fault of GROMACS. I don't know if a 
better solution exists.

> 3. To somewhat improve the way the documentation is written.

OK, I will add a short note to mdrun -h noting that there exist 
execution environments where timing of file access across separate 
GROMACS processes might be a problem.

Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-users/attachments/20110605/b64dfd7c/attachment.html>