I did an AIMD run in VASP using 64 cores (32 cores/node) and 250 GB of memory (125 GB/node) for 5 ps (1 fs time step). The calculation took 12.6 hours to complete. CPU efficiency was 99.7% and memory efficiency was 42%.
So I estimated for a 50 ps run with the exact same parameters I would need approximately 126 hours. So I requested 6 days to be safe. Interestingly after 6 days the run timed out and completed only ~22 ps when I checked the output files. CPU efficiency was 98.9% and memory efficiency was 158%!
I don't understand why the memory is getting overloaded and then I guess "slowing" down the calculation once it gets to about 20 ps? Perhaps I'm missing something. Maybe I need more memory and cores? Or can I get around this by simply splitting the 50 ps into individual 10 ps runs?
This won't directly answer your question, but a note of caution about focusing too much on optimization.
It's definitely worth doing some experimentation to find an efficient combination of memory, # of cpus, etc when you're getting ready to run big simulations. This is especially true for challenging problems where you will be at the limit of your computational resources. But, let not the perfect be the enemy of the good for a few reasons:
It's basically impossible to do controlled benchmark experiments. Your code will be running on a cluster with other jobs, interacting with them in nontrivial ways that you can't anticipate.
Small changes in the exact type of CPU or network traffic can affect performance in unexpected nonlinear ways.
Changes in the parameters of your simulation, or even different runs of the same simulation, can have different performance.
CPU time is cheap, your time is valuable. Once you've established a protocol that is not monstrously inefficient, it's probably good enough to proceed. You can always monitor performance to make sure nothing is going wrong.