7
$\begingroup$

Context

I saw some examples of SLURM scripts on the internet like the one below:

#!/bin/bash
#SBATCH -J QE68
#SBATCH --ntasks=16
#SBATCH --ntasks-per-core=1
#SBATCH --partition=defq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err

Problem

I am having a hard time understanding the specific values that are initialized to the various directives. For example, ntasks=16 and ntasks-per-cor=1.

How can I know in prior that ntasks=16 not 17, why ntasks-per-cor=1 and not 2?

I don't know how to arrive at those numbers, or which commands I should use to get them before constructing a SLURM script.

Question(s):

  1. How can I determine what numbers I need to use for the various directives?

  2. Are there any good rules of thumb to determine these numbers?

$\endgroup$
4
  • $\begingroup$ In this page, you will find the description of all sbatch directives. $\endgroup$ May 26, 2023 at 20:57
  • $\begingroup$ @AbdulMuhaymin I modified my post can you check it please? $\endgroup$
    – Camilla
    May 29, 2023 at 9:13
  • $\begingroup$ that link contains a description of all possible slurm directives. I guess, first you need to know how many nodes, CPUs, cores, threads, etc. you have. Then for example --ntasks-per-core can be set to any number between 1 to the number of threads per core you have. See Vandan's answer below for a way to get the initial estimates. $\endgroup$ May 29, 2023 at 10:46
  • 1
    $\begingroup$ I'm not a QE guy, but I know from the code I develop for, that nowadays these codes have multiple levels of MPI, OMP, and maybe GPU parallelism. This implies that the user should understand the different parallelization layers to select a parallelization scheme resulting in performant calculations. Often this is a calculation-dependent mixture of the different parallelization approaches or leaves some compute cores unoccupied. I would assume that all SLURM scripts I saw in the answers up to now are pretty naive. Please read quantum-espresso.org/Doc/user_guide/node17.html $\endgroup$ May 29, 2023 at 11:45

2 Answers 2

5
$\begingroup$

Since the question was edited, here is an updated answer.

1. How to estimate the numbers which we initialize the SLURM directives:

As far as I understand, the way to decide the numbers that you initialise the SLURM directives are by empirical means.

So one way to estimate those numbers are by running your script on a very small subset on your local PC and logging various specifications of your PC. This could help you to calculate how long, how much memory you might need for a larger population (more molecules, more stronger basis sets etc).

For example, if you run a CCSD_T calculation for 1 molecule containing 20 atoms on your local PC. And suppose it takes 14 mins on your local PC which has 32 GB RAM, 6 cores.

So if you wanted to run the calculations for 100 such molecules on the supercomputer, with 32GB RAM and 6 cores, you can estimate that it would take 100*14 = 1400 mins, which is about 23 hours !

So perhaps you would want to increase the RAM to 64GB or maybe 128GB. Which would mostly half and quarter your time. So maybe at 128GB it would take 6 hours.

So in that way you can say:

#SBATCH --time=6:00:00
#SBATCH --cpus-per-task=6
#SBATCH --mem=128000

2. Good rules of thumb for determining the values for SLURM directives:

For the --mem= directive, use values which are powers of 2.



To Print out what values are initialized when you run the shell script

You can access the various parameters using the SLURM environment variables. A complete list of the environment variables are found on the SLURM webpage.

input-environment-variables

output-environment-variables

Here is an example of how to print information regarding these parameters using SLURM environment variables:

Say we have a test.sh with the following contents:

#!/bin/bash
#SBATCH -J QE68
#SBATCH --ntasks=16
#SBATCH --ntasks-per-core=1
#SBATCH --partition=defq
#SBATCH --mem=800
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
printf "N tasks per core %s\n" "$SLURM_NTASKS_PER_CORE"
printf "Number of tasks  %s\n" "$SLURM_NTASKS"
printf "Memory per node  %s\n" "$SLURM_MEM_PER_NODE"

Upon scheduling this sh script with: sbatch test.sh

In the output file, we would see the following output:

N tasks per core 1
Number of tasks  16
Memory per node  800
$\endgroup$
1
  • $\begingroup$ I modified my post can you check it please? $\endgroup$
    – Camilla
    May 29, 2023 at 9:13
5
$\begingroup$

The SLURM general directives are given on the SLURM webpage, as indicated in the other comments.

The Digital Research Alliance of Canada uses the script below to launch QuantumEspresso jobs:

#!/bin/bash
#SBATCH --account=def-someuser
#SBATCH --time=0-1:00           # DD-HH:MM
#SBATCH --nodes=1
#SBATCH --tasks-per-node=32     # MPI tasks
#SBATCH --mem=0                 # all memory on node
module load StdEnv/2020  intel/2020.1.217  openmpi/4.0.3
module load quantumespresso/6.6
srun pw.x < si.scf.in > si.scf.out

There are no specific directives for specific software.

$\endgroup$
2
  • $\begingroup$ I modified my post can you check it please? $\endgroup$
    – Camilla
    May 29, 2023 at 9:10
  • 2
    $\begingroup$ That numbers ( 'ntasks=16' and 'ntasks-per-cor=1') are selected accordingly to the number of CPUs you want to use to run a parallel job. Remember that not always more CPUs imply your job will run faster! The number of CPUs also will depend on the type of hardware you have and how the SLURM variable are linked to the package you are running. In my answer above, the number of CPUs that will be used is setup only in the variable 'tasks-per-node'. $\endgroup$
    – Camps
    May 29, 2023 at 13:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .