2
$\begingroup$

Question

I submitted 2 jobs for the first time in a HPC to test it. It's a very light calculation that takes almost 5 minutes or less in my desktop. But it has been almost 1 hour since I submitted the job in the HPC with SLURM using the command sbatch script.sl and I still didn't see an output from my calculations.

When I use the command squeue -u username I get the folloming :

JOBID  PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) 
192137    shortq      one  username PD       0:00      2 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions) 
192138    shortq      one  username PD       0:00      2 (Priority) 

I tried to verify the status of the calculation by the command sacct -j 192138 -o start,end but I got the following result (same thing for the other job):

              Start                 End 
------------------- ------------------- 
            Unknown             Unknown 

Scripts that I used

For Job 192137:

#!/bin/bash
#SBATCH -J one
#SBATCH --ntasks=64
#SBATCH --ntasks-per-core=1
#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err

#load QuantumESPRESSO
module load QuantumESPRESSO/intel/7.0
export OMP_NUM_THREADS=1

#prepare work dir
export WORK_DIR=/data/$USER/Qe${SLURM_JOB_ID}
export INPUT_DIR=$PWD/myInput

[[ -z $INPUT_DIR ]] && { echo "Error: Input Directory (INPUT_DIR) is not defined "; exit 1; }
[[ ! -d $INPUT_DIR ]] && { echo "Error:Input Directory (INPUT_DIR) does not exist "; exit 1; }

export ESPRESSO_PSEUDO=/data/$USER/pseudo
mkdir -p $WORK_DIR
cp -R $INPUT_DIR/* $WORK_DIR
cd $WORK_DIR

#run
echo "Running Quantum Espresso with $SLURM_NTASKS at $WORK_DIR"

mpirun -np $SLURM_NTASKS pw.x -i 1.in > 1.out

echo "Done"

For Job 192138:

#!/bin/bash
#SBATCH -J one
#SBATCH --ntasks=64
#SBATCH --ntasks-per-core=1
#SBATCH --time=00:10:00

#SBATCH --partition=shortq
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err

#load QuantumESPRESSO
module load QuantumESPRESSO/intel/7.0
export OMP_NUM_THREADS=1

#prepare work dir
export WORK_DIR=/data/$USER/Qe${SLURM_JOB_ID}
export INPUT_DIR=$PWD/myInput

[[ -z $INPUT_DIR ]] && { echo "Error: Input Directory (INPUT_DIR) is not defined "; exit 1; }
[[ ! -d $INPUT_DIR ]] && { echo "Error:Input Directory (INPUT_DIR) does not exist "; exit 1; }

export ESPRESSO_PSEUDO=/data/$USER/pseudo
mkdir -p $WORK_DIR
cp -R $INPUT_DIR/* $WORK_DIR
cd $WORK_DIR

#run
echo "Running Quantum Espresso with $SLURM_NTASKS at $WORK_DIR"

mpirun -np $SLURM_NTASKS pw.x -i 1.in > 1.out

echo "Done"
$\endgroup$
5
  • 1
    $\begingroup$ For the squeue output, the column labeled NODELIST (REASON) lists either the nodes a calculation is running on or the reason it isn't running yet. Both your job submissions seemed to be marked as having low priority, so they haven't started yet. $\endgroup$
    – Tyberius
    May 30, 2023 at 14:17
  • $\begingroup$ @Tyberius How can I have a high prority? Based on what they give a low or high priority? $\endgroup$
    – Camilla
    May 30, 2023 at 14:31
  • 2
    $\begingroup$ Different HPC centers or clusters can use different factors to determine priority, depending in part on what kind of jobs they want to promote. Consulting your cluster's documentation or an admin would be two ways to find out. We could just guess. $\endgroup$
    – Anyon
    May 30, 2023 at 17:00
  • $\begingroup$ @Tyberius and @Anyon now in the NODELIST (REASON) it's giving me QOSMaxCpuPerUserLimit what does that mean, please? $\endgroup$
    – Camilla
    Jun 2, 2023 at 9:13
  • 1
    $\begingroup$ QOSMaxCpuPerUserLimit would normally indicate that the job doesn't start, because if it did, you would exceed your assigned max number of CPU cores. See e.g. example 2 here $\endgroup$
    – Anyon
    Jun 2, 2023 at 13:29

1 Answer 1

4
$\begingroup$

"I still didn't see an output from my calculations from the HPC."

This is because your jobs had not yet started. I can see this based on the squeue output that you provided:

JOBID  PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) 
192137    shortq      one  username PD       0:00      2 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)        
192138    shortq      one  username PD       0:00      2 (Priority) 

As mentioned in the comment by Tyberius, job 192137 did not start because the nodes (computers) required to run the calculation are either "down, drained or reserved for jobs in higher priority partitions", and job 192138 did not start yet because other jobs (maybe even your own job 192137, but I did later notice that job 192138 had #SBATCH --time=00:10:00 which job 192137 did not have, and 10-minute jobs would have a lower priority than longer jobs on some systems, because HPC clusters are not meant for such short calculations) have a priority over this one.

When you ran sacct -j 192138 -o start,end and it gave you unknown for the start and end times, it would have been more informative if they instead told you that the job has not started yet, but the point is that the job had not yet started running, so there was no "start time" or "end time".

$\endgroup$
1
  • $\begingroup$ The ST column (next to USER column) indicates the status of the job, and PD means pending. When your job starts, it will show R $\endgroup$
    – S R Maiti
    Dec 1, 2023 at 21:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .