[gmx-developers] DD support in ASPC

[Tomáš Trnka]

Hello,

the ASPC code as currently submitted for 4.6 review doesn't work with domain 
decomposition. This stems from the fact that I have neither any significant 
experience in MPI programming nor deep understanding of the DD core.

However, supporting DD shouldn't be too hard. For each shell, ASPC keeps a 
history of its relative displacement from the carrying nucleus and uses it to 
predict the new position in each integration step. There are no cross-shell 
dependencies (a shell's history is only used to predict the position for that 
shell). So the only thing that's necessary for working DD is that when a shell 
crosses from one node to another, it needs to carry its history over.

Could anyone please provide me with some pointers how this can be best 
achieved? I've tried several approaches but every time the thing got too 
complicated too soon:

1) extend t_state with the ASPC history arrays - main problems with this were 
that t_state is touched in hundreds of places throughout the code; also the 
placement of shells onto individual DD nodes would need to be recorded 
somewhere (another global index?)

2) keep the history within gmx_shellfc_t and collect/distribute in 
make_local_shells() - old shell indices would need to be kept somewhere there 
too to be able to do the collection or possibly the indices could be collected 
from the nodes, too; I have a bad feeling about doing collection in here as it 
is called from dd_partition_system, so it's rather counter-intuitive and could 
break something

3) ???

Thanks for any suggestions!
Best regards

Tomáš Trnka