[gmx-developers] DD support in ASPC

[Mark Abraham]

On 6/03/2012 8:57 PM, Tomáš Trnka wrote:
> Hello,
>
> the ASPC code as currently submitted for 4.6 review doesn't work with domain
> decomposition. This stems from the fact that I have neither any significant
> experience in MPI programming nor deep understanding of the DD core.
>
> However, supporting DD shouldn't be too hard. For each shell, ASPC keeps a
> history of its relative displacement from the carrying nucleus and uses it to
> predict the new position in each integration step. There are no cross-shell
> dependencies (a shell's history is only used to predict the position for that
> shell). So the only thing that's necessary for working DD is that when a shell
> crosses from one node to another, it needs to carry its history over.
>
> Could anyone please provide me with some pointers how this can be best
> achieved? I've tried several approaches but every time the thing got too
> complicated too soon:
>
> 1) extend t_state with the ASPC history arrays - main problems with this were
> that t_state is touched in hundreds of places throughout the code; also the
> placement of shells onto individual DD nodes would need to be recorded
> somewhere (another global index?)

What's a nucleus? If it's an atom, then things should be reasonably 
straightforward, piggy-backing the existing accounting for which atoms 
belong to which processor. If it's not an atom, then expressing a 
nucleus as being attached to some appropriate master atom and preceding 
as before might be the way forward.

Mark

> 2) keep the history within gmx_shellfc_t and collect/distribute in
> make_local_shells() - old shell indices would need to be kept somewhere there
> too to be able to do the collection or possibly the indices could be collected
> from the nodes, too; I have a bad feeling about doing collection in here as it
> is called from dd_partition_system, so it's rather counter-intuitive and could
> break something
>
> 3) ???
>
> Thanks for any suggestions!
> Best regards
>
> Tomáš Trnka