5
$\begingroup$

I am a newbie using VASP, currently I am trying to calculate the formation energy per atom and pre bandgap using DFT with VASP on a GPU machine.

I wrote the following script for calculating the bandgap:

# -*- coding: utf-8 -*-
from ase.io import read
from ase.calculators.vasp import Vasp
from ase.optimize import BFGS
import numpy as np
import os

os.environ['VASP_PP_PATH'] = '/scratch'

# Load the structure from a CIF file
structure = read('mp-1218989.cif')

# Set up VASP calculator for relaxation
calc_relax = Vasp(xc='PBE',  # Exchange-correlation functional
                  encut=400,  # Plane-wave cutoff
                  kpts=(3, 3, 3),  # k-point grid for relaxation
                  ibrion=2,  # Ion relaxation: CG algorithm
                  nsw=50,  # Number of steps for ionic relaxation
                  ismear=0,  # Gaussian smearing
                  sigma=0.05,  # Width of smearing in eV
                  lreal='Auto',  # Use real-space projection for GPU acceleration
                  command='mpirun -np 1 vasp_gpu')  # VASP run command

# Attach the calculator to the structure for relaxation
structure.set_calculator(calc_relax)

# Optimize the structure
opt = BFGS(structure)
opt.run(fmax=0.01)

# Now setting up the NSCF calculation for band structure
calc_nscf = Vasp(xc='PBE',  # Exchange-correlation functional
                 encut=400,  # Plane-wave cutoff
                 kpts=(11, 11, 11),  # denser k-point mesh for band structure
                 ismear=0,  # Gaussian smearing (must use ISMEAR < -1 or 0)
                 sigma=0.05,
                 ibrion=-1,  # No ionic relaxation
                 nsw=0,  # No ionic steps
                 icharg=11,  # Non-self-consistent field run
                 command='mpirun -np 1 vasp_gpu')

# Attach the non-self-consistent calculator
structure.set_calculator(calc_nscf)

# Calculate the electronic structure (no need to run optimize)
structure.calc.calculate(structure)

# Extract eigenvalues
eigenvalues = structure.calc.get_eigenvalues(kpt=0, spin=0)  # For the first k-point and spin

# Calculate bandgap
occupied = np.max(eigenvalues[eigenvalues < 0])
unoccupied = np.min(eigenvalues[eigenvalues > 0])
bandgap = unoccupied - occupied

print('Bandgap (PBE):', bandgap, 'eV')

I still have not figured out how to calculate the formation energy. This is because the above code is not utilising the GPU properly. I have 2 gpus and 20 cpus in my system.

I have tried to increase the value here mpirun -np 1 vasp_gpu to more than 1 but it is just showing that my system is not capable. How do I make this calculation take advantage of the GPU machine?

Error

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 2
slots that were requested by the application:

  vasp_gpu

Either request fewer slots for your application, or make more slots
available for use.

A "slot" is the Open MPI term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which Open MPI processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, Open MPI defaults to the number of processor cores

In all the above cases, if you want Open MPI to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --oversubscribe option to ignore the
number of available slots when deciding the number of processes to
launch.

it doesn’t matter how many cpu or gpu’s i am taking if i assign more than 1 here mpirun -np 2 vasp_gpu i am getting the above error

$\endgroup$
6
  • $\begingroup$ Good question. It would help if you shared (a) does your VASP utilize 1 or more GPUs if called from the command line or bash script on the GPU machine, but without ase and python? (b) what your ase + VASP does and shows and outputs when you say "it's just showing that the system is not capable". $\endgroup$ Commented Apr 24 at 18:45
  • $\begingroup$ I don't use VASP, but I think it uses NCCL for the GPU communications which means you cannot use more MPI processes than you have GPUs. You should be able to use threads instead, if you want to use more of the CPU resource. What GPUs do you have? They will need to be NVIDIA Tesla cards (e.g. V100, A100, H100). $\endgroup$ Commented Apr 25 at 1:32
  • $\begingroup$ @AndreyPoletayev I have updated the question with the error I am getting $\endgroup$
    – harsh
    Commented Apr 25 at 5:00
  • 1
    $\begingroup$ @harsh so from your edit I take it that VASP runs fine if using mpirun -np 1 for one GPU? Because that does change the original question. Reminding you also of the second part of my question - does the error persist if python and ase are not used, and VASP is just called from command line or bash script? $\endgroup$ Commented Apr 25 at 9:22
  • 1
    $\begingroup$ @harsh firstly, there's no point using an RTX 2080, they are horrendously slow for 64-bit calculations. Secondly, your error is saying that your OpenMPI configuration file has only 1 core, you should increase the slots to 2 or more. Thirdly, you haven't posted the error you get when you try more than one thread, but that's how to use more cores than 1 per GPU with a NCCL application like VASP. $\endgroup$ Commented Apr 25 at 21:30

0

You must log in to answer this question.