Case study: KPOINT and band parallelization in Quantum ESPRESSO

Question

I am testing Quantum ESPRESSO with different values for Npool and bands in order to calculate run time.

pools: each image can be subpartitioned into "pools", each taking care of a group of k-points.
bands: each pool is subpartitioned into "band groups", each taking care of a group of Kohn-Sham orbitals (also called bands, or wavefunctions). Especially useful for calculations with hybrid functionals.

I took a simple 32-atom supercell of aluminum. Here is the input:

&control
    calculation='scf'
    restart_mode='from_scratch',
    pseudo_dir = '.'
    outdir=./tmp/,
    prefix='al'
    tprnfor = .true.
    tstress = .true.
    disk_io='none'
 /
 &system
    ibrav=  0, A=4.05, nat=  32, ntyp= 1, ecutwfc =60.0,
    occupations='smearing', smearing='marzari-vanderbilt', degauss=0.05
 /
 &electrons
    diagonalization='david'
    mixing_beta = 0.7
 /
CELL_PARAMETERS alat
2 0 0
0 2 0
0 0 2

ATOMIC_SPECIES
 Al  26.98 Al.pbe-n-kjpaw_psl.1.0.0.UPF 
ATOMIC_POSITIONS crystal
      Al 0.00000000        0.00000000        0.00000000
      Al 0.25000000        0.25000000        0.00000000
      Al 0.25000000        0.00000000        0.25000000
      Al 0.00000000        0.25000000        0.25000000
      Al 0.50000000        0.00000000        0.00000000
      Al 0.75000000        0.25000000        0.00000000
      Al 0.75000000        0.00000000        0.25000000
      Al 0.50000000        0.25000000        0.25000000
      Al 0.00000000        0.50000000        0.00000000
      Al 0.25000000        0.75000000        0.00000000
      Al 0.25000000        0.50000000        0.25000000
      Al 0.00000000        0.75000000        0.25000000
      Al 0.50000000        0.50000000        0.00000000
      Al 0.75000000        0.75000000        0.00000000
      Al 0.75000000        0.50000000        0.25000000
      Al 0.50000000        0.75000000        0.25000000
      Al 0.00000000        0.00000000        0.50000000
      Al 0.25000000        0.25000000        0.50000000
      Al 0.25000000        0.00000000        0.75000000
      Al 0.00000000        0.25000000        0.75000000
      Al 0.50000000        0.00000000        0.50000000
      Al 0.75000000        0.25000000        0.50000000
      Al 0.75000000        0.00000000        0.75000000
      Al 0.50000000        0.25000000        0.75000000
      Al 0.00000000        0.50000000        0.50000000
      Al 0.25000000        0.75000000        0.50000000
      Al 0.25000000        0.50000000        0.75000000
      Al 0.00000000        0.75000000        0.75000000
      Al 0.50000000        0.50000000        0.50000000
      Al 0.75000000        0.75000000        0.50000000
      Al 0.75000000        0.50000000        0.75000000
      Al 0.50000000        0.75000000        0.75000000
K_POINTS automatic
11 11 11  0 0 0

Here, I am running on single node with 48 processor with different number of nkand nb

For example :
mpiexec.hydra -np 48 /USERS/pranav/compilation/q-e-qe-6.8/bin/pw.x -in al.scf.david.in -nk 3 -nb 4

In First study, I varied nk while fixed nb=1. Here is the result

nk nb time(sec) 
1 1 206
2 1 163
3 1 144
4 1 138
6 1 138
8 1 131
12 1 141

In second run, I fixed nk at 3 and varied nb and Here is the result

nk nb time(sec) 
3  1 144
3  2 181
3  4 255
3  8 426

Why is run-time increasing with increase in nb in second run? I was hoping for decreasing in trend.

Note: QE v6. 8 is compiled with intel-oneapi and internal FFT library

Answer 1

I don't know for certain, but a fairly general model of the time taken $T(N)$ for a parallel calculation on $N$ processes is,

$T(N) = S + \frac{P}{N} + C(N-1)$

where $S\geq 0$ is the serial part of the work (the part which can't be parallelised), $P\geq 0$ is the parallelisable work, and $C\geq 0$ represents the cost of doing parallel communications.

If $P$ is large and $S$ and $C$ are small, then

$T(N)\approx \frac{P}{N}$

and your calculation will scale perfectly. If, on the other hand, $S$ is large compared to $P$, then

$T(N)\approx S$

and your calculation will take the same amount of time, regardless of how many processes you use! Even worse, if $C$ is not negligible then

$T(N)\approx S + C(N-1)$

and the time will actually increase with more processes.

The final ingredient is to note that the communication cost actually comes as two parts, a latency cost and a bandwidth cost. The latency cost is the time it takes to establish a communication -- even a zero-length message takes this amount of time. The bandwidth cost is the time it takes to send data, once the communication has been established. The time it takes to send long messages depends mostly on the bandwidth, but short messages are dominated by the latency. As you increase $N$, most algorithms will end up sending more messages, and each message will be shorter. Shorter messages take less time to send when you're limited by bandwidth, but at some point they are so short that latency becomes more important -- after this point, the message time becomes approximately constant.

In your case, we have two different parallelisation schemes. The k-point parallelism is scaling well, which is generally found to be the case since only a few one-to-all communications are required per SCF cycle and almost all the work distributes well over k-point.

When we look at band-parallelism, your times are increasing. Band-parallelism requires more communications inside the SCF cycle, for example when performing orthonormalisation or subspace rotations; these actually require all-to-all communications. This communication cost will be small if there is a lot of work to do, but your calculation is not actually very large.

Assuming that your pseudopotential has 3 valence electrons per aluminium atom, your 32-atom cell has 96 electrons, meaning 48 valence bands. I don't know how many conduction bands you're using, but let's suppose you have 16 to give us 64 bands in total. It looks like you've instructed QE to use a block-Davidson optimisation method, in which case these bands will be split into subgroups; I don't know how large these subgroups will be, but perhaps 4 groups of 16 bands each.

16 bands is not very many to distribute, so it's likely that there won't be much work in between communications, and the communications will be fairly short, i.e. dominated by latency. These two factors mean that the cost of parallelising the calculation is quite high, and it is unlikely to scale well.

This situation could be improved by increasing the number of bands in the subgroups, which would mean fewer subgroups. I don't know whether Quantum Espresso allows you to control that, but if it does then I suggest you try it. It would increase the work in between communications and also lead to fewer, longer messages; both of these should improve the scaling, though not necessarily the calculation time.