I am currently running a fairly large system (NELECT>7400) on the OpenACC GPU port of VASP 6.4.1 on 16 Nvidia A100 GPUs: 4 GPUs x 4 nodes (NCSA Delta 4-Way NVIDIA A100 GPU Compute Nodes). Each of these nodes has an AMD Milan processor with 64 cores and 256G RAM.

I am trying to optimize my parallelization parameters to leverage the CPUs and GPUs fully. Currently I see a CPU usage of only about 10-15% during the electronic steps. SSH'ing into a running node and executing top I see:

Tasks: 985 total,   5 running, 980 sleeping,   0 stopped,   0 zombie
%Cpu(s):  9.9 us,  2.0 sy,  0.0 ni, 88.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 257657.6 total, 178810.2 free,  62478.3 used,  16369.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 180957.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                               
 666852 user      20   0   71.2g  14.8g 903220 R 199.0   5.9  68:33.14 vasp_gam                                                                                                                              
 666851 user      20   0   71.2g  14.6g 865724 R 191.4   5.8  69:11.01 vasp_gam                                                                                                                              
 666849 user      20   0   80.5g  15.1g 979208 R 183.1   6.0  68:41.78 vasp_gam                                                                                                                              
 666850 user      20   0   71.2g  14.6g 866220 R 182.5   5.8  68:09.62 vasp_gam  
  57391 root      20   0  436628  41512   9348 S   1.3   0.0 169:23.16 nv-hostengine  

My calculation is Gamma-point only so my only choices for NCORE and KPAR are NCORE=KPAR=1. I know the efficiency of the GPU implementation of VASP relies on NSIM being set higher than the default, however when I set NSIM>4 I run out of memory on the GPUs.

Here is a condensed version of my job script:

#SBATCH --job-name="vasp"
#SBATCH --partition=gpuA100x4
#SBATCH --nodes=4
#SBATCH --mem=0
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=16   # spread out to use 1 core per numa, set to 64 if tasks is 1
#SBATCH --gpus-per-node=4
#SBATCH --gpu-bind=closest   # select a cpu close to gpu on pci bus topology
#SBATCH --exclusive  # dedicated node for this job
#SBATCH -t 8:00:00

srun -N 4 -n 16 vasp_gam > output

As requested by in the comments, here is my full INCAR:




#Restart and I/O



Are there any changes to how I call vasp_gam that would increase CPU utilization and presumably improve runtime?

  • 1
    $\begingroup$ Can you post your full INCAR? This paper may be helpful for you: drive.google.com/file/d/1kPFNc-y0ezn_ANatYDpE04x603U-gxlL/… $\endgroup$
    – kpoint
    Oct 23, 2023 at 5:38
  • $\begingroup$ Thanks, I think this is what I needed. I'm running a job right now to copy their configuration and see if I notice any speedup. $\endgroup$
    – user8097
    Oct 23, 2023 at 22:11
  • $\begingroup$ One thing you could try is turning off WAVECAR if you don't need it. I think that the memory usage increases at the end of a run if WAVECAR is on. Otherwise, I found the most speedup in my own calculations by setting KPAR = number of GPUs/node, but since you're using vasp_gam I'm not sure if that is possible/will help. $\endgroup$
    – kpoint
    Oct 29, 2023 at 0:42
  • $\begingroup$ Thanks, the memory use is a problem during the electronic steps. The paper you linked was helpful; in the end I am running about as optimally as I can at the moment. If you'd like to post it as an answer, I'm happy to accept it. $\endgroup$
    – user8097
    Oct 30, 2023 at 17:39
  • $\begingroup$ VASP uses NVIDIA's NCCL library for its GPU MPI, which has the unfortunate side-effect that you can only use 1 MPI process per GPU -- ie with the usual MPI parallelism you can only possibly use 4 of the 64 cores. To use the other 60, you will have to use OpenMP threading, which is not generally very efficient for such simulations. Your CPU usage per process is only ~200%, suggesting 2 threads per process. What is your setting for OMP_NUM_THREADS? $\endgroup$ Dec 6, 2023 at 2:36

1 Answer 1


There is a conference paper from NERSC that has benchmarks for different VASP settings on Perlmutter using NVIDIA A100 GPUs and AMD Milan CPUs.

In the paper, they mention that running VASP on a GPU node leaves many CPUs idle. Here is their conclusion:

One can see that enabling OpenMP threads significantly slows down the HSE workloads, especially when using fewer threads per task (see blue bars for Si256 hse and B.hR105 hse). However, OpenMP threads benefit other workloads, although not significantly, especially for one-node runs (up to 6%). In practice, enabling eight or 16 threads per task to utilize otherwise idling CPU resources for non-HSE workloads can get a small additional speedup. The significant slowdown from using OpenMP threads for HSE workloads can be fixed when setting NBLOCK_FOCK= NBANDS/Total_number_of_MPI_processes in the INCAR file. NBLOCK_FOCK, the blocking factor in the Fock-exchange operator (an undocumented INCAR tag), is set to 2*OMP_NUM_THREADS internally in VASP. Apparently, it benefits from a larger NBLOCK_FOCK than 2*OMP_NUM_THREADS, especially for smaller OMP_NUM_THREADS values, to achieve optimal performance. When a proper NBLOCK_FOCK is used for HSE, the OpenMP threads also benefit the VASP performance similar to the rest of the workloads.

So it seems that increasing CPU usage does not significantly affect VASP performance on GPUs. There are other benchmarks in the paper as well. The best things to do are to set KPAR = # of GPUs, which I think will not work for a gamma point calculation, and to use all the GPU memory which you are already doing.

Here is a direct link to the paper, it may also be available on the conference website at some point as well.

  • 1
    $\begingroup$ This is a "link-only" answer, please put the relevant contents from the paper into the answer. $\endgroup$ Nov 1, 2023 at 12:16

You must log in to answer this question.