I wrote a SLURM job script to run a computational chemistry calculation using the CREST program (part of the xtb software package). In the script, I create a temporary directory on the local storage of the compute node. The files from the submission directory are copied to this temporary directory, after which I run the CREST calculation in the background. The script contains a trap to handle SIGTERM signals (for job termination). If terminated, it attempts to archive results and copy the archive back to the original submission directory.
The functions are:
- wait_for_allocated_time: Calculates and waits for the job's time limit
- report_crest_status: Reports the status of the CREST calculation
- archiving: Creates an archive of the output files
- handle_sigterm: Handles premature job termination
The script is designed to:
- Utilize local storage on compute nodes for better I/O performance
- Handle job time limits gracefully
- Attempt to save results even if the job is terminated prematurely
- Provide detailed logging of the job's progress and any issues encountered
The problem with the script is that it fails to create an archive because sometimes the local directory is cleaned up before archiving can occur (see output below).
- Running xtb crest calculation...
- xtb crest calculation interrupted. Received SIGTERM signal. Cleaning up...
- Sat Jul 6 16:24:20 CEST 2024: Creating output archive...
- Sat Jul 6 16:24:20 CEST 2024: LOCAL_DIR /tmp/job-11235125
- total 0
- Sat Jul 6 16:24:20 CEST 2024: ARCHIVE_PATH /tmp/job-11235125/output-11235125.tar.gz
- tar: Removing leading `/' from member names
- tar: /tmp/job-11235125: Cannot stat: No such file or directory
- tar (child): /tmp/job-11235125/output-11235125.tar.gz: Cannot open: No such file or directory
- tar (child): Error is not recoverable: exiting now
- tar: Child returned status 2
- tar: Error is not recoverable: exiting now
- Sat Jul 6 16:24:20 CEST 2024: Failed to create output archive.
- Job finished.
I hoped to prevent this by running a parallel process in the background and wait for it to monitor the job's allocated time. This process will sleep until the allocated time is nearly up. Only when the archiving took place, the complete job script will end and thus preventing the clean up of the local directory. However, somehow this did not work and I do not know how to prevent cleanup of the local directory in case of termination/cancellation/error of the job.
Can someone help me?
#!/bin/bash
dos2unix $1
dos2unix *
pwd=$(pwd)
#echo "0) Submitting SLURM job..." >> "$pwd/output.log"
#SBATCH --time=0-00:30:00
#SBATCH --partition=regular
#SBATCH --nodes=1
#SBATCH --ntasks=12
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
module purge
module load OpenMPI
LOCAL_DIR="$TMPDIR/job-${SLURM_JOBID}"
SIGTERM_RECEIVED=0
function wait_for_allocated_time () {
local start_time=$(date +%s)
local end_time
local time_limit_seconds
time_limit_seconds=$(scontrol show job $SLURM_JOB_ID | grep TimeLimit | awk '{print $2}' |
awk -F: '{ if (NF==3) print ($1 * 3600) + ($2 * 60) + $3; else print ($1 * 60) + $2 }')
end_time=$((start_time + time_limit_seconds))
echo "Job started at: $(date -d @$start_time)" >> "$pwd/time.log"
echo "Expected end time: $(date -d @$end_time)" >> "$pwd/time.log"
echo "Job time limit: $((time_limit_seconds / 60)) minutes" >> "$pwd/time.log"
current_time=$(date +%s)
sleep_duration=$((end_time - current_time))
if [ $sleep_duration -gt 0 ]; then
echo "Sleeping for $sleep_duration seconds..." >> "$pwd/time.log"
sleep $sleep_duration
echo "Allocated time has ended at: $(date)" >> "$pwd/time.log"
else
echo "Job has already exceeded its time limit." >> "$pwd/time.log"
fi
}
function report_crest_status () {
local exit_code=$1
if [ $SIGTERM_RECEIVED -eq 1 ]; then
echo "xtb crest calculation interrupted. Received SIGTERM signal. Cleaning up..." >> "$pwd/output.log"
elif [ $exit_code -eq 0 ]; then
echo "xtb crest calculation completed successfully." >> "$pwd/output.log"
else
echo "xtb crest calculation failed or was terminated. Exit code: $exit_code" >> "$pwd/output.log"
fi
}
function archiving () {
echo "$(date): Creating output archive..." >> "$pwd/output.log"
cd "$LOCAL_DIR" >> "$pwd/output.log" 2>&1
echo "$(date): LOCAL_DIR $LOCAL_DIR" >> "$pwd/output.log"
ls -la >> "$pwd/output.log" 2>&1
ARCHIVE_NAME="output-${SLURM_JOBID}.tar.gz"
ARCHIVE_PATH="$LOCAL_DIR/$ARCHIVE_NAME"
echo "$(date): ARCHIVE_PATH $ARCHIVE_PATH" >> "$pwd/output.log"
tar cvzf "$ARCHIVE_PATH" --exclude=output.log --exclude=slurm-${SLURM_JOBID}.out $LOCAL_DIR >> "$pwd/output.log" 2>&1
if [ -f "$ARCHIVE_PATH" ]; then
echo "$(date): Output archive created successfully." >> "$pwd/output.log"
else
echo "$(date): Failed to create output archive." >> "$pwd/output.log"
return 1
fi
echo "$(date): Copying output archive to shared storage..." >> "$pwd/output.log"
cp "$ARCHIVE_PATH" "$pwd/" >> "$pwd/output.log" 2>&1
if [ $? -eq 0 ]; then
echo "$(date): Output archive copied to shared storage successfully." >> "$pwd/output.log"
else
echo "$(date): Failed to copy output archive to shared storage." >> "$pwd/output.log"
fi
}
function handle_sigterm () {
SIGTERM_RECEIVED=1
report_crest_status 1
archiving
kill $SLEEP_PID
}
trap 'handle_sigterm' SIGTERM #EXIT #USR1
echo "1) Creating temporary directory $LOCAL_DIR on node's local storage..." >> "$pwd/output.log"
mkdir -p "$LOCAL_DIR" >> "$pwd/output.log" 2>&1
if [ $? -eq 0 ]; then
echo "Temporary directory created successfully." >> "$pwd/output.log"
else
echo "Failed to create temporary directory." >> "$pwd/output.log"
exit 1
fi
echo "2) Copying files from $pwd to temporary directory..." >> "$pwd/output.log"
cp "$pwd"/* "$LOCAL_DIR/" >> "$pwd/output.log" 2>&1
if [ $? -eq 0 ]; then
echo "Files copied successfully." >> "$pwd/output.log"
else
echo "Failed to copy files." >> "$pwd/output.log"
exit 1
fi
cd "$LOCAL_DIR" || exit 1
echo "3) Running xtb crest calculation..." >> "$pwd/output.log"
srun crest Bu-Em_RR_OPT.xyz --T 12 --sp > crest.out &
MAIN_PID=$!
wait_for_allocated_time &
SLEEP_PID=$!
wait $MAIN_PID #wait -n $MAIN_PID $SLEEP_PID
CREST_EXIT_CODE=$?
if [ $SIGTERM_RECEIVED -eq 0 ]; then
report_crest_status $CREST_EXIT_CODE
if [ $CREST_EXIT_CODE -eq 0 ]; then
archiving
fi
kill $SLEEP_PID
fi
wait $SLEEP_PID
echo "Job finished." >> "$pwd/output.log"
