# How can I submit jobs to an HPC scheduler?

It is my first time trying to use an HPC system and I am a bit overwhelmed. I have gone through a course on using HPC systems on Coursera and have a basic idea about using Linux. Could you guys tell me what the general procedure is for running ab initio packages such as Quantum ESPRESSO?

For the information about my particular HPC system you can have a look here. It says that we use a system called SGE, but I don't know how this works.

You can find the coursera course here

• This conversation has been moved to chat. Aug 2 at 21:25
• There really is not information in the question; the correct command depends on the scheduler installed on the HPC (for SLURM, see Anoop A Nair's answer), as well as on the shell you are using, possibly the Linux distro that is installed, and a number of other factors. That is why the only people who can give you a reliable answer are the administrators for your specific HPC. Aug 3 at 17:07

# SLURM

I'll provide you with a general script to run QE on your HPC. This is with the assumption that the HPC facility is using SLURM as your workload manager and also with the assumption that you have the essential modules installed like openmpi (for parallel execution of the QE code) and intel compiler (in the example given below):

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --job-name=scf
#SBATCH --nodes=2
#SBATCH --time=5-00:00:00

mpirun -np 56 ~/path to bin folder of QE/pw.x -inp QE_input_file.in > QE_output_file.out


Here the --nodes=2 indicates the number of nodes allocate to QE to parallelize over and the number 56 after -np is the number of processors the QE is allowed to parallelize over. In my case each node in the HPC facility i used has 28 cores and thats why I'm able to parallelize QE over 56 CPU cores. Usually there would be a restriction on the number of nodes a person can use, for me it was 8 (hence I could parallelize QE over 224 CPU cores at the most). The module load <package> line loads in the required packages for your task at hand. The ~/path to bin folder of QE/pw.x indicates that if you have compiled QE by yourself on the HPC provided to you just add the path to the required QE routine there and your good to go!!.

If QE is installed in the HPC (and if its added as an environment variable) you could just specify pw.x (assuming that you are running a SCF calculation) in place of ~/path to bin folder of QE/pw.x

In order to run the job, copy the code into a shell script (say submit.sh) and submit it using the command :

sbatch submit.sh


You could read more about SLURM in the documentation link I've attached above. I hope it helps :)

• This conversation has been moved to chat. Aug 2 at 22:42
• Heads up! The SBATCH lines, despite beginning with "#", are not comments. I've been bitten by that in the past... (This may look obvious here but in shell syntax # introduces a comment.) Aug 3 at 22:08
• @LoremIpsum That's very true. It took me a long time to get used to that (and I'm still not completely used to it). Two # characters together is enough for the line to be treated as a comment though :) Aug 4 at 0:47
• @LoremIpsum: IMO probably the best way to think of them is that they are comments (as far as the shell is concerned), but they're a special kind of comments that sbatch (which peeks at the script before handing it off to the shell / OS to execute) will notice and (try to) parse. Just like the #! line at the top is also treated as a comment by the shell, but the OS will look at it to decide which shell (or other script interpreter) to run the script with. It's a very good point to mention, though. Aug 4 at 9:10

I would recommend to save the following shell script file with a name like submission.sh:

#! /bin/ksh
# @ job_name   = Calculation_for_Parmeet
# @ job_type=MPICH
# @ error = job.err.$(jobid) # @ node = 2 # @ tasks_per_node = 20 # @ wall_clock_limit = 23:58:00 # @ notification = complete # @ notify_user = nike@hpqc.org # @ queue module load qe/6.5 module load intel mpirun -np 40 ~/path_to_bin_folder_of_QE/pw.x -inp QE_input_file.in > QE_output_file.out  Now running ./submission.sh in the folder where submission.sh was saved, will run a job: • named Calculation_for_Parmeet, • with 2 nodes and 20 cores/node (40 cores in total), • a time limit of just under 24 hours (since making it exactly 24 hours might send your job to a queue for longer jobs, which might make it take longer for your job to run, depending on the HPC system's configurations), • with an email notification to nike@hpqc.org, • with error messages printed to job.err.$(jobid) where jobid is the ID number for the job, assigned by the scheduler,
• with an input file called QE_input_file.in and output file QE_output_file.out,
• with qe/6.5 installed as a module and compiled with the HPC system's intel module, and the QE program pw.x located in ~/path_to_bin_folder_of_QE/.

## SGE

Disclaimer: I haven't ever used SGE before, so there may be some inaccuracies that other users can correct. However, it seems similar to other systems and the instructions from your university page seem sufficient.

The example input is very similar to those already given:

#!/bin/bash
#$$-S /bin/bash #specifies shell #$$ -cwd          #where to find input/place output
#$$-N JobName #$$ -pe mpich 32   #-pe=Parallel environment, 32 is number of CPU
#$$-e$$JOB_ID.err  #Error file, named with variable $$JOB_ID #$$ -o $$JOB_ID.out #Output file, should contain prints to console #$$ -q all.q     #submission queue
#$-V #Says to load your environment variables for the job module load qe/6.5 module load intel mpirun -np 40 ~/path_to_bin_folder_of_QE/pw.x -inp QE_input_file.in > QE_output_file.out  There are a couple things you will need to look out for with your university's system. • For the number of CPUs, your administrators say all nodes have 16, so you should mainly specify jobs in increments of 16. • They seem to have several queue: all.q, largejob.q, and test.q. These have different uses, permissions, and restrictions associated with them. As the name suggests, you should probably test your job with test.q first to see if it at least starts. Then you can run it with all.q, which will give access to more time/CPUs. You can then submit using qsub ScriptName where ScriptName can be whatever you want (I tend to just use job) if its something I can use generally that isn't terribly specific to one type of job. • Is the shell redirection (>) necessary? Wouldn't it conflict with the option -o$JOB_ID.out? I have seen that if redirection is used, the scheduler output is empty. Aug 5 at 2:37
• @SRMaiti I suppose not. In general, it could be used to capture other output that isn't directly from QE, but if they stick with this minimal job script it should just lead to an empty file.
– Tyberius
Aug 5 at 2:46