11
$\begingroup$

I have a system that has 1 k-point, 54 atoms, NBANDS=152, and NGZ=80. I'm running an AIMD simulation on VASP version 5.4.4

The supercomputer has 32 cores/node. It's a LINUX cluster linked by Infiniband, modern multicore machine. I was wondering what would be the best way to select parameters and request resources to get an efficient and fast calculation. Here are some scenarios I've considered. I'm curious what would be the "best" method (in terms of efficiency and speed).

  1. Based on what I read here
  • I should request something like 19 bands (152/8). Maybe set NBANDS=160 to request an even 20 cores.
  • Would I then set NCORE=20 in my INCAR file and request 20 cores on 1 node?
  • Or would it be better to set NCORE = 10. And then request something like 2 nodes and 10 cores per node? or would that create too much communication overhead and slow down the simulation?
  1. If I decide to use # of cores = # of atoms = 54
  • Would I set NCORE=32, and then request 54 cores over 2 nodes (32 on 1 node + 22 on the second)? Or 9 cores/node and 6 nodes?
  1. Can't I just request the whole node? 1 node, 32 cores
  • so then set NCORE=32 and I just request the whole node. However the VASP manual suggests against this because that would make NPAR=1
  • this confuses me a bit because I would assume 32 cores on one node would be more efficient than having cores across several nodes communicating with each other
Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
13
$\begingroup$

As someone that has done a lot of benchmarking for VASP, I would suggest you try the experimental approach. I do believe VASP will add additional bands for you if need be for parallelization, so I would not worry about that personally. The layout of the node physically (32 cores on 1 processor vs 16 cores on 2 processors vs special AMD processor layouts on a single CPU) can differ significantly from cluster to cluster you cannot know what is optimal without trying.

Since you are running MD simulations it seems, I think it is well worth it to benchmark each system before you run a long simulation. Minor changes don't require you to rebenchmark but if you go from 50 to 150 to 300 atoms, the ideal may change. Run a series of quick calculations with the entire range of NCORE that seems reasonable. Use the best result. I tend to check every factor of the largest node.

For 32 cores, I would check NCORE=(1, 2, 4, 8, 16, 32). I would time it against 10 or so geometric steps. This may seem like a waste of time, but it may end up saving large amounts of time in the future.

I would almost always suggest requesting entire nodes unless you have a good reason to not do so. You may eventually see a KPAR option as well while looking around, I have heard mixed opinions. I have personally never got a better result with kpoint parallelization than without it. It may make a memory difference though.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you! This is a really great suggestion - I'll try this. For ~10 geometric steps it shouldn't take too long. I didn't bother considering KPAR since I only have 1 k-point. Do you think there would be a significant advantage of trying 64 cores (2 nodes)? Or is that pointless because I only have 54 atoms? $\endgroup$
    – DoubleKx
    Aug 20, 2020 at 4:17
  • 1
    $\begingroup$ You might not have much of an advantage, but keep in mind scaling is not based on atoms it is based on bands / valence electrons. 54 hydrogen will be significantly easier than 54 Uranium atoms. Also be sure to use vasp-gam for a significant speed increase if you are only doing the gamma point. $\endgroup$
    – Tristan Maxson
    Aug 20, 2020 at 4:35
  • 1
    $\begingroup$ Also, for the sake of completeness, make sure that a single node isn't better. Scaling is not linear and you can get slowdowns. Especially since 2 nodes will require communication between nodes. $\endgroup$
    – Tristan Maxson
    Aug 20, 2020 at 4:36
  • 1
    $\begingroup$ The 2 nodes was actually a bit faster on my cluster. I posted my results as an answer below $\endgroup$
    – DoubleKx
    Aug 21, 2020 at 3:52
7
$\begingroup$

I thought I would post the results of the benchmarking as per Tristan's suggestion. Maybe it will be useful for someone in the future. I only did 10 steps in the MD (1 fs step size, 10 fs total).

Percent differences are all relative to the NCORE=1 run with 32 cores and 1 node. There's a 33% speedup by going to the 2 nodes with 64 cores (NCORE=32).

Obviously this may differ depending on your cluster. This is the Graham cluster on Sharcnet in Canada by the way - so hopefully it's useful to fellow Canadians out there :)

enter image description here

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Interestingly, looking at the 32 cores, 1 node: setting NCORE=32 was the fastest. According to the VASP manual it should have been slower? Wondering if anyone has thoughts on this? $\endgroup$
    – DoubleKx
    Aug 21, 2020 at 4:09
  • 1
    $\begingroup$ Interesting sidenote, bad NCORE settings for 2 nodes can make it worse than a single node. If I was running this personally, I would probably consider using a single node just because of the minor performance difference to run an additional calculation alongside it. Shows how much of a difference going from one node to another makes in timing as well. $\endgroup$
    – Tristan Maxson
    Aug 21, 2020 at 4:56
  • 1
    $\begingroup$ Also, not sure if you did but just to remind you. Vasp-gam or vasp-gamma depending on version is a massive speed increase for gamma only calculations. This was a really good answer to add to really show how benchmarking cannot be replaced. $\endgroup$
    – Tristan Maxson
    Aug 21, 2020 at 4:58
  • 1
    $\begingroup$ @TristanMaxson Yup, I used vasp-gam since I only had the one k-point. I need to do ~100 ps runs for several temperatures so the 2 nodes will save me a lot of time going forward. Thanks for all your suggestions! $\endgroup$
    – DoubleKx
    Aug 21, 2020 at 5:08
  • $\begingroup$ For benchmarking, I recommend also normalizing by the number of cores / nodes when you care about total throughput rather than fastest single calculations. Even though two nodes is a 33% increase, two jobs concurrently on one node each is a 38% increase. $\endgroup$
    – Tristan Maxson
    Aug 21, 2020 at 5:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .