6
$\begingroup$

I'm running my calculations using Quantum ESPRESSO on a 10th generation i7 computer (Dell) with Ubuntu.

When I run any calculation using the command mpirun '/directory/ pw.x' -in file.in> file.out, only 8 processors are used. However the output of nproc gives me 16.

I want my calculations to run faster. How can I increase the number CPUs used? Is it safe to do that?

PS: I use the computer just for DFT calculations at the moment.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Exactly how are you running Quantum Espresso? How do you start it? And how many cores are in the machine you are using? To answer that if you are using Linux the output of lscpu would be useful, if relevant please edit the question to include it. I'm afraid I can't help you with Windows. $\endgroup$
    – Ian Bush
    Commented Mar 1, 2023 at 11:29
  • $\begingroup$ @IanBush I edited the question can you check it please? $\endgroup$
    – Camilla
    Commented Mar 1, 2023 at 13:04
  • 4
    $\begingroup$ The output of nproc is not that useful. Can you show us the output of lscpu as requested. $\endgroup$
    – Ian Bush
    Commented Mar 1, 2023 at 13:40

2 Answers 2

Reset to default
4
$\begingroup$

Answer to the question in the title

"How to increase the number of CPUs in my computer?"

You can either remove your current i7 microprocessor from the motherboard and replace it with one that has more CPUs, or you could connect multiple motherboards together to form an HPC "cluster". You might find the following threads useful if you're interesting in going in either of those directions:

However, the rest of your question suggests that the title was not chosen so accurately. It seems that you're not trying to increase the number of CPUs in your computer but you're trying to make use of more of the CPUs that already exist in your comptuer.

Nick Papior's comment pointed out that if "8 processors are used" (as your question says), then you're already using all of your processors! This is because the Intel i7 microprocessor does something called hyperthreading which means that each core can have two threads, so 8 physical cores means a total of 16 possible threads.

Is it safe to oversubscribe a computer?

"How can I increase the number CPU used? Is it safe to do that?"

I do not recommend for you to oversubscribe your machine. Nick Papior's above-linked comment mentioned that you can use --oversubscribe, but it came with a big warning:

"but please test for something simple to see if this actually gives you anything!!!!"

I want to add to that warning, not to just test if it gives you speed-up, but also to test how safe this is: I wouldn't want you to crash your hardware. In this question: Is this "internal error" reproducible when compiling OpenMolcas on other Intel compilers?, I warned users not to even "fully subscribe" (let alone "over subscribe") a personal computer. In that specific case of compiling OpenMolcas, the compilation took about 20 minutes when using 2 cores, but took more than an hour when using 4 cores (which also caused me a big scare because the entire computer froze during that hour, and the fan seemed overworked as if the computer was going to overheat and permanently damage CPU components).

Conclusion

Welcome to the wonderful world of high-performance matter modeling :) It's really nice to have you in this community. Some of my greatest joys have been with doing calculations that gave me experiences like the one you're currently having. At some point we need to accept that we are doing the best we can with the resources that we have available, and we just need to find more resources (e.g. run the calculations on a supercomputer that has 1000s of cores), reduce the computational demands in the project (e.g. don't aim for such high accuracy), or wait longer for the calculations to run.

If you are open to the idea of jumping off the bicycle (8-core CPU) and hopping into a racecar (supercomputer with 1000s of cores), the following threads might be helpful for you (also I have over 1000 CPU years available in total in case you want to try some calculations here):

Improve this answer
$\endgroup$
2
  • $\begingroup$ If your code is not well optimized, there might be something to gain from using both logical cores per physical core. e.g. if you sum an array with simplistic code like for(...) sum += arr[i]; and don't use -ffast-math to let the compiler unroll with multiple accumulators to hide FP latency. Two threads per physical core would let the CPU interleave work of two dependency chains each bottlenecked on one add per 4 clock cycles (e.g. Skylake). Of course, it's even better to optimize your code so one thread can mostly saturate its execution units and/or memory bandwidth. $\endgroup$
    – Peter Cordes
    Commented Mar 1, 2023 at 21:44
  • 3
    $\begingroup$ Yeah, for most HPC work, you only want one thread per physical core. But a good personal computer should have no problem running all cores, especially a desktop. e.g. a prime95 stress test on my system (i7-6700k) gets my 4c8t machine up to 75C at 3.9GHz (EPP = balance_performance) or 80C at 4.0 GHz (performance), according to sensors. (With "small" FFTs, otherwise it wants to allocate so much RAM that it's triggering the OOM killer :/) At worst a non-broken system won't be able to turbo beyond its base frequency, which can be quite low in a laptop if there are many cores and low TDP. $\endgroup$
    – Peter Cordes
    Commented Mar 1, 2023 at 22:04
6
$\begingroup$

The specific details of how you are running QE are not clear. But here is a simple general overview of how to run QE in parallel settings.

If you are using mpirun command, then do the following:

mpirun -np 8 pw.x < file.in > file.out

Here, the -np switch specifies the number of processes of QE to start. You can increase it from 8 to as long as you are not oversubscribing. But make sure you have enough available processors. For other programs such as dos.x, bands.x, etc., the syntax remains the same. For windows, you might have to change .x to .exe. Also, if the standard input symbol < doesn't work, try -inp or -in or simply -i.

Also, you can specify the number of processors used in each group by adding additional switches (-nk, -nb, -nd etc). See this user guide for details.

Improve this answer
$\endgroup$
5
  • $\begingroup$ when i run a calculation I use the command mpirun '/directory/ pw.x' -in file.in> file.out and when I check the file.out i see that the calculations were performed using 8 processors. However when I type the command nproc in the terminal it gives me 16, means I have 16 processors. Am I right? If yes I don't know how to benefit from the remaining 8 processors $\endgroup$
    – Camilla
    Commented Mar 1, 2023 at 12:57
  • $\begingroup$ You can try the -np 16 switch after mpirun to use all your 16 processing units $\endgroup$
    – Abdul Muhaymin -Free Palestine
    Commented Mar 1, 2023 at 13:05
  • $\begingroup$ When I try to do that I get the following message in the terminal There are not enough slots available in the system to satisfy the 9 slots that were requested by the application........ $\endgroup$
    – Camilla
    Commented Mar 1, 2023 at 13:15
  • 8
    $\begingroup$ This is because you have 16 processing units, but only 8 physical cores. Generally MPI checks for physical cores vs threads, and nproc gives you the threads. For qe you probably won't see a performance benefit from using the threads, but you can do so with mpirun -np 16 --oversubscribe ... (but please test for something simple to see if this actually gives you anything!!!!) $\endgroup$
    – nickpapior
    Commented Mar 1, 2023 at 13:54
  • $\begingroup$ This conversation is also taking place in chat. $\endgroup$
    – Nike Dattani - No Free Time
    Commented Mar 1, 2023 at 16:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .