10
$\begingroup$


How to maximize the efficiency/time-to-solution of the calculations in Q-Chem on HPC clusters?

MPI parallelization is supposed to improve time-to-solution as compared to OpenMP. The system I tested numerous times has 80 alpha and 80 beta electrons (requested basis set is 6-311+G(d,p), there are 158 shells and 462 basis functions). The job (geometry optimization followed by frequency calculation) was running on 16 cpus either on a single node:
qchem -np 1 -nt 16 INPUT.inp OUTPUT.out

or on 4 nodes with 4 cpus in MPI fashion:
qchem -np 4 -nt 4 INPUT.inp OUTPUT.out

Still, OpenMP works much faster. Am I wrong here? Should it be exactly like this?

I also tried hybrid MPI+OpenMP way. For that I added 'export OMP_NUM_THREADS=4' line to my SLURM script, but it still lost in time to OpenMP, but at least was after than MPI.

Is it possible to accelerate such calculations?

$\endgroup$
  • 1
    $\begingroup$ Which method are you using? (Different methods are parallelised differently in Q-Chem) $\endgroup$ – Michael F. Herbst May 18 at 12:21
  • $\begingroup$ I am working with DFT functionals: PBE0, B3LYP and M06-2X $\endgroup$ – Dmitry Eremin May 18 at 17:22
4
$\begingroup$

I am not at all surprised that using 16 cores on one node with OpenMP is faster than using 16 cores spread across 4 different computers (nodes). The 16 cores on a single node are usually part of the same "chip" and are definitely attached to the same motherboard. The 16 cores spread across 4 computers will not be able to communicate with each other nearly as quickly as if they were in the same chip or on the same motherboard.

Is it possible to accelerate such calculations?

Since for this calculation, OpenMP on a single node gave the fastest results out of the three experiments you did, to acceleration your results, I suggest you try to use more OpenMP cores on a single node (I would be surprised if your HPC cluster has nodes with only 16 cores, because for the last several years, most HPC nodes have 24, 32, or 40 cores.

If you still want faster results, you can add a second node. This will allow you to double your total number of cores, but please do not expect the speed to be twice as fast as when you used 1 node, because there can be overhead due to the fact that you are now using two separate computers with two separate motherboards (connected by Omni-Path, or InfiniBand, or even an Ethernet cable in some rare cases!). If you have two calculations like this to do, I recommend that you run each of them on a separate node with maximum OpenMP parallelization, rather than to run one calculation on two nodes, and then the other calculation on the same two nodes.

So when should you use MPI? Two examples of when I'd recommend MPI are:

  • When a single node does not have enough RAM for the calculation, you can use "distributed memory" meaning that data needed for the calculation can be spread across multiple nodes to give you sufficient RAM to store all the required data. This is probably not your case, because while 462 spatial orbitals is a fairly large amount, the integrals will indeed be manageable on a single node on most HPC cluster built any time in the last several years.

  • When the algorithm you are using is "embarrassingly parallel", meaning that there is essentially no communication needed between the nodes, so doubling the number of nodes will in fact nearly double the overall speed of the calculation. This is the case for FCIQMC, but not so much for the SCF in DFT (which is what you're doing, and is probably why your use of MPI is slowing you down in comparison to using OpenMP with the same number of total cores).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thank you, @NikeDattani! This is now very clear to me! $\endgroup$ – Dmitry Eremin May 31 at 4:28
  • $\begingroup$ No problem. I'm glad it helped! $\endgroup$ – Nike Dattani May 31 at 4:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.