10
$\begingroup$

I did an AIMD run in VASP using 64 cores (32 cores/node) and 250 GB of memory (125 GB/node) for 5 ps (1 fs time step). The calculation took 12.6 hours to complete. CPU efficiency was 99.7% and memory efficiency was 42%.

So I estimated for a 50 ps run with the exact same parameters I would need approximately 126 hours. So I requested 6 days to be safe. Interestingly after 6 days the run timed out and completed only ~22 ps when I checked the output files. CPU efficiency was 98.9% and memory efficiency was 158%!

I don't understand why the memory is getting overloaded and then I guess "slowing" down the calculation once it gets to about 20 ps? Perhaps I'm missing something. Maybe I need more memory and cores? Or can I get around this by simply splitting the 50 ps into individual 10 ps runs?

Improve this question
$\endgroup$
8
  • $\begingroup$ +1. But it might help to say exactly what you mean by "CPU efficiency" and "memory efficiency". Also: the cost of things can change over the course of a simulation. As the MD simulation runs, the geometry of the atoms and molecules changes, which may result in more or less numerically intense calculations to do. Luckily you save checkpoint files so that you can start from near where you left off? $\endgroup$
    – Nike Dattani
    Sep 17, 2020 at 16:57
  • $\begingroup$ I'm still relatively new to all this but I believe "CPU efficiency" refers to "core-walltime" (ratio of the actual core time from all cores divided by the number of cores requested divided by the run time). "memory efficiency" refers to ratio of the high-water mark of memory used by all tasks divided by the memory requested for the job. Hopefully that's more clear now. And yes VASP outputs the most recent "state" if it crashes so I can continue from there fortunately. $\endgroup$
    – DoubleKx
    Sep 17, 2020 at 17:14
  • $\begingroup$ Ok this "CPU efficiency" has nothing to do with the CPU, but more to do with "parallelization efficiency". And thanks for clearing that up that your job used 158% of the total RAM you requested. Consider my other comment about how the cost of an MD iteration might change as the geometries of the atoms and molecules get closer together or farther apart. The fact that the RAM usage is higher in the second case than the first case suggests to me that the program wanted to use more RAM later in the calculation (for whatever reason). This alone would explain the need for more CPU time. $\endgroup$
    – Nike Dattani
    Sep 17, 2020 at 17:22
  • $\begingroup$ If the phase space is homogeneous in all directions, you should have an scalable prediction. But, how do you ensure that? Maybe at the first time steps the route is more soft than at the final steps... $\endgroup$
    – Camps ♦
    Sep 17, 2020 at 17:29
  • $\begingroup$ @NikeDattani perhaps you're right and that is the cause. From inspection some atoms have deviated a little bit but nothing significant. I'll restart from there for more 10 ps and see how much RAM it uses. If it's the geometry then RAM usage should be high. But should the cost really be that large? Don't people normally do MD at high-temperatures to study materials in a melted state (i.e. geometry significantly deviates)? $\endgroup$
    – DoubleKx
    Sep 17, 2020 at 22:48

2 Answers 2

Reset to default
3
$\begingroup$

This won't directly answer your question, but a note of caution about focusing too much on optimization.

It's definitely worth doing some experimentation to find an efficient combination of memory, # of cpus, etc when you're getting ready to run big simulations. This is especially true for challenging problems where you will be at the limit of your computational resources. But, let not the perfect be the enemy of the good for a few reasons:

  • It's basically impossible to do controlled benchmark experiments. Your code will be running on a cluster with other jobs, interacting with them in nontrivial ways that you can't anticipate.
  • Small changes in the exact type of CPU or network traffic can affect performance in unexpected nonlinear ways.
  • Changes in the parameters of your simulation, or even different runs of the same simulation, can have different performance.
  • CPU time is cheap, your time is valuable. Once you've established a protocol that is not monstrously inefficient, it's probably good enough to proceed. You can always monitor performance to make sure nothing is going wrong.
Improve this answer
$\endgroup$
2
$\begingroup$

For MD I recommend The Gamma Cantered KPOINTS with Grid mesh 1 1 1.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Please elaborate your answer with more details $\endgroup$
    – Thomas ♦
    Oct 7, 2020 at 9:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .