OpenMPI not running under SLURM in UBUNTU with ORCA?

Question

I have ORCA 5.0.4, OpenMPI 4.1.1 and SLURM in UBUNTU 22 in a single PC with 16 threads.

I have setup this pc and, when I run orca alone, openmpi works. But, when I submit orca through slurm, the calculation fails. In this image I can show what is happening:

It seems that SLURM does not recognize OpenMPI

How can I resolve it?

Files:

Orca.sh

#!/bin/bash -l
#SBATCH -J p_amf2
#SBATCH -n 1
#SBATCH --cpus-per-task=8
#SBATCH --ntasks-per-node=1
#SBATCH --output="output.out"
#SBATCH --error="error.err"
#SBATCH --mem=1000

export LD_LIBRARY_PATH="$HOME/orca:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/usr/local/lib/openmpi/:$LD_LIBRARY_PATH"
export MPIROOT="/usr/local/bin/:$MPIROOT"
export OMP_PROC_BIND=true OMP_PLACES=sockets
export OMPI_MCA_mpi_yield_when_idle=1

cd $SLURM_SUBMIT_DIR

/home/david/orca/orca Benzene.inp > Test.out 

ORCA input:

# Benzene RHF Opt Calculation
#%pal nprocs 8 end
! RHF TightSCF PModel
! opt
                               # Molecule specification
* xyz 0 1
     C    0.000000000000     1.398696930758     0.000000000000
     C    0.000000000000    -1.398696930758     0.000000000000
     C    1.211265339156     0.699329968382     0.000000000000
     C    1.211265339156    -0.699329968382     0.000000000000    
     C   -1.211265339156     0.699329968382     0.000000000000
     C   -1.211265339156    -0.699329968382     0.000000000000
     H    0.000000000000     2.491406946734     0.000000000000
     H    0.000000000000    -2.491406946734     0.000000000000
     H    2.157597486829     1.245660462400     0.000000000000
     H    2.157597486829    -1.245660462400     0.000000000000
     H   -2.157597486829     1.245660462400     0.000000000000
     H   -2.157597486829    -1.245660462400     0.000000000000
*

slurm.conf

# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=localcluster
SlurmctldHost=localhost
MpiDefault=none
ProctrackType=proctrack/linuxproc
ReturnToService=2
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
#
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
#
# COMPUTE NODES
NodeName=localhost CPUs=24 RealMemory=31772 State=UNKNOWN
PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE     State=UP

Answer 1

In order for Orca to launch an 8-task MPI run, Slurm must assign 8 tasks. This is achieved by the --ntasks-per-node flag. So you need the following settings in your job script:

--ntasks-per-node=8
--cpus-per-task=1

Which allows Orca to execute an 8-task MPI job (i.e., mpirun -np 8 ...). See also this web page, which has a sample Slurm script for Orca.

Your initial setting of

--ntasks-per-node=1
--cpus-per-task=8

Does not allow for spawning multiple MPI tasks but, rather, for a single multi-threaded process. It can be used if you're running an OpenMP-parallelized program.

Answer 2

Just for completeness, I will copy over my answer to this question in the Orca forum:

Assuming you don't have an Orca 'module' set up, you may need to add a few extra lines to your script. The ones I have on mine are:

export MPIROOT="/opt/local/bin/:$MPIROOT"  # modified from /usr/local/openmpi 

export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"  # contains my libmpi.40.dylib file

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/Applications/orca/orca_5_0_4_macosx_arm64_openblas_openmpi411"  # my Orca directory

While, as you can tell, I am using a Mac, you definitely need to specify the LD_LIBRARY_PATH appropriately, and possibly the OpenMPI path (not sure). Maybe add them and see if that helps?

(I suspect the second line in that example isn't necessary if you have your LD_LIBRARY_PATH variable set in your .bashrc or .bash_profile files, but I didn't, so I don't know for sure.)

All this being said, the error in your screenshot does not necessarily mean that OpenMPI is not activating. That error often pops up if you do not have enough RAM available for the calculation you want to run. I just noticed that your job script does not actually ask for any RAM, so I am not sure if that will mean Slurm gives you as much memory as you need, or absolutely no memory. Either way, I would suggest you add this line to your Slurm script:

#SBATCH --mem=xxx  # amount of RAM requested, in MB

So, if you have an Orca input file with:

%mem 1000
%pal
nprocs 4
end

you would use:

#SBATCH --mem=5000

or something like that. (Personally, I think it is always better to include a value for %mem in the Orca input file, just to be on the safe side.)