9
$\begingroup$

I am performing an SCF Quantum ESPRESSO calculation on an HPC system, but it is taking significantly longer than usual. When I inspect the output file, I notice that it is stuck at a specific line, yet the calculation appears to be still running in the background.

Interestingly, when I ran the exact same calculation on my laptop using just 2 cores, it completed within just 1 minute. However, when I tested it on the HPC using both 10 cores then 2 cores, the problem persisted regardless of the number of cores utilized.

Could someone please explain to me the possible reasons for this difference in calculation time between the HPC system and my laptop?

PS: My input contains nosym=.true. and noinv=.true. in the &SYSTEM block. However, when I remove these two lines from the input, the calculation ran smoothly without any issues on the HPC.

The last few lines in case of HPC:

    Estimated static dynamical RAM per process >      30.73 MB

     Estimated max dynamical RAM per process >      36.84 MB

     Estimated total dynamical RAM >     147.34 MB

     Initial potential from superposition of free atoms

     starting charge   35.99975, renormalised to   36.00000
     Starting wfcs are   19 randomized atomic wfcs +    1 random wfcs

     total cpu time spent up to now is      110.7 secs

     Self-consistent Calculation

     iteration #  1     ecut=    70.00 Ry     beta= 0.70
     Davidson diagonalization with overlap

---- Real-time Memory Report at c_bands before calling an iterative solver
           252 MiB given to the printing process from OS
             0 MiB allocation reported by mallinfo(arena+hblkhd)
         22597 MiB available memory on the node where the printing process lives

The lines in case of laptop:

     Estimated static dynamical RAM per process >      57.66 MB

     Estimated max dynamical RAM per process >      69.90 MB

     Estimated total dynamical RAM >     139.80 MB

     Initial potential from superposition of free atoms

     starting charge      35.9998, renormalised to      36.0000
     Starting wfcs are   19 randomized atomic wfcs +    1 random wfcs

     total cpu time spent up to now is        2.4 secs

     Self-consistent Calculation

     iteration #  1     ecut=    70.00 Ry     beta= 0.70
     Davidson diagonalization with overlap

---- Real-time Memory Report at c_bands before calling an iterative solver
            79 MiB given to the printing process from OS
             0 MiB allocation reported by mallinfo(arena+hblkhd)
           832 MiB available memory on the node where the printing process lives
------------------
     ethr =  1.00E-02,  avg # of iterations =  3.7

     Threshold (ethr) on eigenvalues was too large:
     Diagonalizing with lowered threshold

     Davidson diagonalization with overlap

---- Real-time Memory Report at c_bands before calling an iterative solver
            84 MiB given to the printing process from OS
             0 MiB allocation reported by mallinfo(arena+hblkhd)
           825 MiB available memory on the node where the printing process lives
------------------
     ethr =  7.58E-04,  avg # of iterations =  1.0

     total cpu time spent up to now is       12.1 secs
$\endgroup$
9
  • $\begingroup$ +1 one, but can you show us the last few lines of output in the case on the HPC system, and from the laptop the first few lines after those? $\endgroup$ Jul 22 at 13:48
  • $\begingroup$ @NikeDattani I added that to the post $\endgroup$
    – Camilla
    Jul 24 at 9:03
  • $\begingroup$ There could be some issue with the way you are submitting jobs, for example, you may have to request memory explicitly. Or there might be something wrong with the software installation on the HPC, or the parallelisation. $\endgroup$
    – S R Maiti
    Jul 24 at 9:06
  • $\begingroup$ @SRMaiti I added something to the post, can you check it please? $\endgroup$
    – Camilla
    Jul 24 at 9:19
  • 1
    $\begingroup$ @ShernRenTee My system administrators (SA) suggested the same thing, I just wanted to understand what's the problem exactly and if someone have an idea about how it can be solved in case my SA ask me questions. $\endgroup$
    – Camilla
    Jul 25 at 10:00

2 Answers 2

11
$\begingroup$

EDIT: This seems to be a case of improper MPI/OpenMP defaults, but the general information below still applies in general.

MPI And OpenMP

The two major parallel programming methods in HPC software are MPI and OpenMP. MPI parallelizes across processes, OpenMP across threads. (The difference is explained in this StackOverflow question.)

Both MPI and OpenMP must be programmed into the software by the developers and compiled correctly. An end-user can't affect that process, but it is their responsibility to call the parallelized program with the right settings.

MPI settings are usually set through flags to the mpirun program. For example, this runs my_program with the number of processes set to 4:

mpirun -np 4 my_program

(With SLURM's srun the -np flag can sometimes be omitted, as SLURM will try to deduce the number of processes from the resources allocated. Please AYSA whether you should use srun or mpirun -- often one option is correct and the other will crash or slow your program.)

OpenMP settings are usually set as environment variables. For example, writing this before the mpirun or srun command sets each process to use one thread:

export OMP_NUM_THREADS=1

Now, for a compute-intensive job one thread needs one core*. Suppose two threads are "squished" onto one core -- the core is forced to multitask between those two threads, and each time it switches threads it has a reset course (plus the core's own scheduler's time spent "juggling" the threads).

In Camilla's answer she noted the following info in the program logs:

Parallel version (MPI & OpenMP), running on       8 processor cores
     Number of MPI processes:                 4
     Threads/MPI process:                     2

while her SLURM script read:

#SBATCH --ntasks=4
#SBATCH --ntasks-per-core=1

So SLURM allocated 4 cores, but her OMP_NUM_THREADS seems to have defaulted to 2, which together with mpirun -np 4 results in 8 threads in total, which will run incredibly slowly when "juggled" on 4 cores. Setting OMP_NUM_THREADS to 1 using the above command seems to have fixed it.

*Unless your cluster and the software you're using supports hyperthreading. AYSA.


Original Answer

Ask. Your. System. Administrator. :)

When you see "AYSA" throughout this answer it means you need to ask your system administrator how to best run jobs on the cluster. We don't know what cluster you use, and even if we did, we don't run it. Your administrators do. If you are good friends with them, they can make your life much better.

And if you say to your system administrators "okay so you're telling me to do XYZ, but a stranger on the Internet said ABC instead", you will not be good friends with them.

So, please AYSA. Having said that, some sysadmins are allergic to what I'd call Google questions -- the sort of questions that, in theory, any reasonably smart scientist can answer for themselves on Google. (The truth is Google sucks these days, and it's not getting any better.) So what I'll do is give you the Google Q&A and also give you more detailed questions to AYSA. Make sense? Here we go.


The First Rule of using a cluster is that The Cluster Is Not Just A Bigger Faster Computer. Here's an analogy: a 4-bedroom house is basically the same as a 2-bedroom house but twice as big. But a 2000-bedroom building isn't a house scaled up 1000x, it's a hotel or an apartment block, and it's a completely different building with completely different principles.

Firstly, each individual core is likely slower than anything that's on your PC. If your job can finish running on a two-processor laptop it should take longer on two cluster cores. That makes sense when you think about the economics: your cluster couldn't afford to buy hundreds of thousands of very fast cores, and it doesn't need to, because the job of the cluster is to run jobs on hundreds or thousands of cores which could never be run on just two cores, no matter how fast. If you're using your cluster because you have to run thousands of small little jobs, AYSA about how to "batch" jobs together so the scheduler can block off the batch like a regular big job.

Secondly, the file system is distributed. Just as cluster cores are not just Big Fast Processors, the cluster's files don't live in One Big Hard Drive. (Imagine if they did -- that one hard drive would have thousands of processes trying to write or read from it at once and it would instantly crash and burn!) Instead there is a distributed file system which reliably collates changes made by all these processors and then makes them consistent. As a tradeoff, you shouldn't always expect an output file to be instantly written the moment a process has produced it -- you may have some delay, especially with a HPC-optimised program that only writes to hard disk infrequently if used with a distributed file system. So don't just keep refreshing your home directory. AYSA what to expect and how to best use the file system.

Thirdly, parallelizing your application is tricky. There's MPI, OpenMP, various Python solutions, and all of them need the correct options to be set -- otherwise you risk allocating 16 cores to an application only to have it run on one or two cores. AYSA, but before you do, you should run scaling benchmarks. If you go up from 2 cores to 4 cores, does it finish twice as fast? What about from 4 to 8? What if you go from 2 cores to 8 cores, but also make the problem four times larger? The first two are examples of strong scaling, which usually drops off more quickly than the weak scaling of the third example. If you go from 2 to 4 to 8 cores and see the time stay the same -- you're probably not asking correctly for cores.

And one important thing to understand is your cluster's chiplet and node sizes. Please don't AYSA until you've done some basic searching, since most clusters publicly describe what processors they use and how many are in each node. Node sizes are relevant to larger jobs -- for example, if each node has 64 cores and you're running a big job, it may be worth using either 64 or 128 cores and booking out whole nodes rather than asking for 96 cores (the scheduler might not even allow you to ask for that).

Chiplet sizes are more relevant to smaller jobs, especially if your cluster uses the recent AMD EPYC processors. Here's an example from documentation of a cluster I use:

Left, design of EPYC Gen2 SoC showing eight compute dies around a central I/O die; right, zoom-in of compute die showing eight cores around central L3 cache

See how, on the right, there are eight cores sharing each L3 cache (essentially superfast RAM used directly during number crunching)? On this cluster, you do best asking for groups of 8 cores so that your program has that L3 cache all to itself, making sure the scheduler does actually give you the group of 8 instead of splashing your job all over various cores. (If you're using SLURM, AYSA about using the -m block:block:block directive or whether they recommend alternatives.) Otherwise, if your program is sharing cache with another job, there may be contention slowing your job down (or if your program is aggressive you will slow their job).

All the best! Please report back on your findings.

$\endgroup$
3
  • 1
    $\begingroup$ Excuse me, but why do you recommend to ask the system administrator in this situation? $\endgroup$ Jul 22 at 10:59
  • 4
    $\begingroup$ Because (1) running jobs on a HPC is very different from running on your own laptop and you need specialised advice (2) that specialised advice will be very different from cluster to cluster, so outside advice like mine is less valuable compared to cluster-specific advice. "HPC" is very general -- there are differences between running on Perlmutter, Lumi, Bunya, Gadi, Setonix ... $\endgroup$ Jul 22 at 23:08
  • $\begingroup$ @ShernRenTee Wow that's incredible :) thank you so much. $\endgroup$
    – Camilla
    Jul 25 at 10:28
7
$\begingroup$

Almost all codes are written to take advantage of symmetry that is available in the system, to reduce the number of calculations that needs to be performed. If you explicitly say that symmetry must not be considered, it will take more time since now the code is stuck doing the calculations which would have otherwise been generated by symmetry.

When you are doing calculations on a cluster with multiple nodes, there is an overhead for splitting the jobs into chunks, collecting the data back to master and related steps. In your case, this overhead is significantly larger than the actual time required to do the calculations on a single node.

$\endgroup$
2
  • $\begingroup$ But how do you explain the fact that it takes more time on the HPC than a normal laptop? $\endgroup$
    – Camilla
    Jul 21 at 14:27
  • 2
    $\begingroup$ I have added that to the answer $\endgroup$ Jul 21 at 14:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .