Why is Q-Chem producing a parallel efficiency greater than 1?

Question

The efficiency $E$ of a parallel calculation is defined as

\begin{align} E &= \frac{S}{N}, & S &= \frac{T_1}{T_N}, \end{align}

where $S$ is the speedup and $N$ is the number of workers, and $T_{1/N}$ is the time for one core, $N$ cores, respectively.

By definition the efficiency of a parallel calculation should never be able to exceed 1.

Considering this, I came upon an interesting result in Q-Chem when I was benchmarking the two parallel options: the OpenMP threaded options (-nt) and the message passing interface option (-np)

As can be seen, I am getting an efficiency greater than 1 for -nt processes!

All calculations were done with qchem/feb2019/env_intel_2018_parallel version and were run on the same compute node.

The number of SCF cycles are identical between the two calculations, but the energy and DIIS error is not exactly the same:

The 1 process calculations:

Cycle       Energy         DIIS Error
 ---------------------------------------
14    -690.0167217745      3.47E-07 

the 4,8, etc. calculation process:

Cycle       Energy         DIIS Error
---------------------------------------
14    -690.0167223190      3.49E-07

Is there an explanation?

Answer 1

Yes, this is correct explanation. There are three SCF codes and two integral libraries and which ones are executed depends on the job parameters (single core, multicore, multi-node). If accurate scaling benchmarks are desired, I would use execution with 2 threads as the base. One should also pay attention to the current defaults re: integral and scf engines - these can change with updates.

The energy difference is smaller than the specified convergence threshold, so there is nothing to worry about. The differences smaller than convergence threshold are perfectly normal for different modes of execution.

Answer 2

While I am not completely familiar with all the details in Q-Chem's SCFs, I am aware that there are in fact multiple SCF implementations and multiple implementations for computing two-electron integrals in Q-Chem.

I could therefore imagine that -nt and -np triggers different code branches (for all cases but 1 thread, where the same is used). This explains why there is the sudden jump from 1 to 2 for the-nt plot, but otherwise similar slopes as you increase $N$.

Another point to note is that efficiencies beyond $1$ are actually possible. This can happen because a couple of reasons. One is that increasing the number of CPUs makes in total also more CPU cache available (which is faster to access than RAM) and overall may change the balance between how much is computed and how much is stored. Since for modern systems an access to RAM well equals a few thousand FLOPs, this can lead to a gain in performance. Therefore for small problem sizes it can happen that this so-called superlinear speedup makes the parallel efficiency go beyond $1$ (see also this computer science thread).

While this could be what happens here, the distinct jump from $N = 1$ to $N = 2$ makes me suspect that the observed behaviour is just an artifact of the code used at $N = 1$ being a lot worse than the code for the -nt branch.