Deciding the number of CPUs for a DFT calculation for band structure in Quantum ESPRESSO?

Question

If I am using Quantum ESPRESSO for bandstructure calculations, how can I know the number of CPUs necessary for such a calculation? I deal mostly with ternary transition metal oxides which have at least 10 atoms per unit cell. An example would be TiVO4. In the supercluster where I run these calculations, there are 40 CPUs per node. The CPU time available for a calculation is one week.

Answer 1

I am not an expert on Quantum ESPRESSO -QE- (just played with it some time ago), but as with any code, the choose of the number of CPUs to run the code is intrinsically related to how each task is coded and which of the parallel interface is used. It is not because you have access to thousand of CPUs that your job will run faster.

Specifically for QE, chapter 3 of the QE User's Guide is dedicated to parallelism. From this chapter, we have the several levels of parallelization implemented in QE:

• world: is the group of all processors (MPI_COMM_WORLD). images: Processors can then be divided into different "images", each corresponding to a different self-consistent or linear-response calculation, loosely coupled to others.
• pools: each image can be subpartitioned into "pools", each taking care of a group of k-points.
• bands: each pool is subpartitioned into "band groups", each taking care of a group of Kohn-Sham orbitals (also called bands, or wavefunctions). Especially useful for calculations with hybrid functionals.
• PW: orbitals in the PW basis set, as well as charges and density in either reciprocal or real space, are distributed across processors. This is usually referred to as "PW parallelization". All linear-algebra operations on array of PW / real-space grids are automatically and effectively parallelized. 3D FFT is used to transform electronic wave functions from reciprocal to real space and vice versa. The 3D FFT is parallelized by distributing planes of the 3D grid in real space to processors (in reciprocal space, it is columns of G-vectors that are distributed to processors).
• tasks: In order to allow good parallelization of the 3D FFT when the number of processors exceeds the number of FFT planes, FFTs on Kohn-Sham states are redistributed to ``task'' groups so that each group can process several wavefunctions at the same time. Alternatively, when this is not possible, a further subdivision of FFT planes is performed.
• linear-algebra group: A further level of parallelization, independent on PW or k-point parallelization, is the parallelization of subspace diagonalization / iterative orthonormalization. Both operations required the diagonalization of arrays whose dimension is the number of Kohn-Sham states (or a small multiple of it). All such arrays are distributed block-like across the ``linear-algebra group'', a subgroup of the pool of processors, organized in a square 2D grid. As a consequence the number of processors in the linear-algebra group is given by n2, where n is an integer; n2 must be smaller than the number of processors in the PW group. The diagonalization is then performed in parallel using standard linear algebra operations. (This diagonalization is used by, but should not be confused with, the iterative Davidson algorithm). The preferred option is to use ELPA and ScaLAPACK; alternative built-in algorithms are anyway available.

Here, you also will find the needed command line switches in order to setup a parallel calculation (-nimage, -npools, -nband, -ntg, -ndiag and -northo).

Answer 2

As mentioned in the previous answer and comments, choosing the correct number of processors depends on several factors. Quantum ESPRESSO has several levels of parallelization, that can be used accordingly to the system being simulated.

Matching the k-points sampling with the number of pools, for example, tends to decrease the calculation times. The convergence thresholds can also play an important role in the calculation times, and the number of processors can speed up the simulation or even get it worst.

Since band structure and Dos calculation rely on pw.x package, from the pw.x user guide we have:

pw.x can run in principle on any number of processors. The effectiveness of parallelization is ultimately judged by the ”scaling”, i.e. how the time needed to perform a job scales with the number of processors, and depends upon:

• the size and type of the system under study;
• the judicious choice of the various levels of parallelization (detailed in Sec.4.4);
• the availability of fast interprocess communications (or lack of it).

Ideally one would like to have linear scaling, i.e. T ∼ T0/Np for Np processors, where T0 is the estimated time for serial execution. In addition, one would like to have linear scaling of the RAM per processor: ON ∼ O0/Np, so that large-memory systems fit into the RAM of each processor. Parallelization on k-points:

• guarantees (almost) linear scaling if the number of k-points is a multiple of the number of pools;
• requires little communications (suitable for ethernet communications);
• reduces the required memory per processor by distributing wavefunctions (but not other quantities like the charge density), unless you setdiskio=’high’.

Parallelization on PWs:

• yields good to very good scaling, especially if the number of processors in a pool is a divisor of N3 and Nr3(the dimensions along the z-axis of the FFT grids,nr3 and nr3s, which coincide for NCPPs);
• requires heavy communications (suitable for Gigabit ethernet up to 4, 8 CPUs at most,specialized communication hardware needed for 8 or more processors);
• yields almost linear reduction of memory per processor with the number of processors in the pool.

A note on scaling: optimal serial performances are achieved when the data are as much as possible kept into the cache. As a side effect, PW parallelization may yield superlinear (better than linear) scaling, thanks to the increase in serial speed coming from the reduction of datasize (making it easier for the machine to keep data in the cache).

VERY IMPORTANT: For each system there is an optimal range of number of processors on which to run the job. A too large number of processors will yield performance degradation. If the size of pools is especially delicate: Np should not exceed N3 and Nr3, and should ideally be no larger than 1/2÷1/4 N3 and/or Nr3. In order to increase scalability, it is often convenient to further subdivide a pool of processors into ”task groups”. When the number of processors exceeds the number of FFT planes, data can be redistributed to ”task groups” so that each group can process several wavefunctions at the same time. The optimal number of processors for ”linear-algebra” parallelization, taking care of multiplication and diagonalization of M×M matrices, should be determined by observing the performances of cdiagh/rdiagh(pw.x) orortho(cp.x) for different numbers of processors inthe linear-algebra group (must be a square integer).

Those are general guidelines for the choice of the parallelization levels. Therefore some effort has to be applied to optimize the number of processors against the structure to be evaluated. With the time performing the simulation some feeling is acquired, easing the process.