5
$\begingroup$

I have ORCA 5.0.4, OpenMPI 4.1.1 and SLURM in UBUNTU 22 in a single PC with 16 threads.

I have setup this pc and, when I run orca alone, openmpi works. But, when I submit orca through slurm, the calculation fails. In this image I can show what is happening:

enter image description here

It seems that SLURM does not recognize OpenMPI

How can I resolve it?

Files:

Orca.sh

#!/bin/bash -l
#SBATCH -J p_amf2
#SBATCH -n 1
#SBATCH --cpus-per-task=8
#SBATCH --ntasks-per-node=1
#SBATCH --output="output.out"
#SBATCH --error="error.err"
#SBATCH --mem=1000

export LD_LIBRARY_PATH="$HOME/orca:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/usr/local/lib/openmpi/:$LD_LIBRARY_PATH"
export MPIROOT="/usr/local/bin/:$MPIROOT"
export OMP_PROC_BIND=true OMP_PLACES=sockets
export OMPI_MCA_mpi_yield_when_idle=1

cd $SLURM_SUBMIT_DIR

/home/david/orca/orca Benzene.inp > Test.out 

ORCA input:

# Benzene RHF Opt Calculation
#%pal nprocs 8 end
! RHF TightSCF PModel
! opt
                               # Molecule specification
* xyz 0 1
     C    0.000000000000     1.398696930758     0.000000000000
     C    0.000000000000    -1.398696930758     0.000000000000
     C    1.211265339156     0.699329968382     0.000000000000
     C    1.211265339156    -0.699329968382     0.000000000000    
     C   -1.211265339156     0.699329968382     0.000000000000
     C   -1.211265339156    -0.699329968382     0.000000000000
     H    0.000000000000     2.491406946734     0.000000000000
     H    0.000000000000    -2.491406946734     0.000000000000
     H    2.157597486829     1.245660462400     0.000000000000
     H    2.157597486829    -1.245660462400     0.000000000000
     H   -2.157597486829     1.245660462400     0.000000000000
     H   -2.157597486829    -1.245660462400     0.000000000000
*

slurm.conf

# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=localcluster
SlurmctldHost=localhost
MpiDefault=none
ProctrackType=proctrack/linuxproc
ReturnToService=2
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
#
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
#
# COMPUTE NODES
NodeName=localhost CPUs=24 RealMemory=31772 State=UNKNOWN
PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE     State=UP
$\endgroup$
11
  • 2
    $\begingroup$ +1 but please include all input files and submission scripts in text format rather than a screensot. I'm not going to type all of that to try to reproduce and debug your error. $\endgroup$ Feb 7 at 14:09
  • $\begingroup$ @NikeDattani The post has been updated. Thank you for any insight. $\endgroup$ Feb 7 at 15:29
  • 2
    $\begingroup$ wouldn't using --ntasks-per-node=8 be more appopriate? given that MPI creates multiple tasks $\endgroup$ Feb 9 at 12:29
  • $\begingroup$ @KristofBal I tried but it did not work, What I understand about the issue is that slurm does not communicate with openmpi. $\endgroup$ Feb 9 at 12:36
  • 1
    $\begingroup$ @ShernRenTee thanks for adding the fact that --cpus-per-task=1 is also needed, I implicitly assumed the OP would try this, but without your comment they didn't $\endgroup$ Feb 13 at 15:09

2 Answers 2

4
+200
$\begingroup$

In order for Orca to launch an 8-task MPI run, Slurm must assign 8 tasks. This is achieved by the --ntasks-per-node flag. So you need the following settings in your job script:

--ntasks-per-node=8
--cpus-per-task=1

Which allows Orca to execute an 8-task MPI job (i.e., mpirun -np 8 ...). See also this web page, which has a sample Slurm script for Orca.

Your initial setting of

--ntasks-per-node=1
--cpus-per-task=8

Does not allow for spawning multiple MPI tasks but, rather, for a single multi-threaded process. It can be used if you're running an OpenMP-parallelized program.

$\endgroup$
2
$\begingroup$

Just for completeness, I will copy over my answer to this question in the Orca forum:

Assuming you don't have an Orca 'module' set up, you may need to add a few extra lines to your script. The ones I have on mine are:

export MPIROOT="/opt/local/bin/:$MPIROOT"  # modified from /usr/local/openmpi 

export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"  # contains my libmpi.40.dylib file

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/Applications/orca/orca_5_0_4_macosx_arm64_openblas_openmpi411"  # my Orca directory

While, as you can tell, I am using a Mac, you definitely need to specify the LD_LIBRARY_PATH appropriately, and possibly the OpenMPI path (not sure). Maybe add them and see if that helps?

(I suspect the second line in that example isn't necessary if you have your LD_LIBRARY_PATH variable set in your .bashrc or .bash_profile files, but I didn't, so I don't know for sure.)

All this being said, the error in your screenshot does not necessarily mean that OpenMPI is not activating. That error often pops up if you do not have enough RAM available for the calculation you want to run. I just noticed that your job script does not actually ask for any RAM, so I am not sure if that will mean Slurm gives you as much memory as you need, or absolutely no memory. Either way, I would suggest you add this line to your Slurm script:

#SBATCH --mem=xxx  # amount of RAM requested, in MB

So, if you have an Orca input file with:

%mem 1000
%pal
nprocs 4
end

you would use:

#SBATCH --mem=5000

or something like that. (Personally, I think it is always better to include a value for %mem in the Orca input file, just to be on the safe side.)

$\endgroup$
11
  • $\begingroup$ Thank you for your response, it works with orca alone. The problem is that slurm is not activating openmpi. I think, the solution is more related with the set up of slurm. +1 for your try. cheers, David $\endgroup$ Feb 9 at 8:41
  • $\begingroup$ @Another.Chemist the HPC I use has implemented a series of 'modules' (I assume it is some kind of bash procedure: module load orca. Maybe if you set it up like that...? (See askubuntu.com/questions/358414/what-does-module-load-do) Unless... are you saying openmpi isn't activating at all? $\endgroup$ Feb 9 at 14:00
  • $\begingroup$ Thank you, I am installed slurm, openmpi and orca in a single computer. You are right, indicating to load the module in a HPC job works. But, if you see the screen photo, orca is working without calling the module, because is a single pc. in resume, the problem is related with slurm, someway, is not calling openmpi $\endgroup$ Feb 9 at 14:04
  • $\begingroup$ @Another.Chemist I edited my answer—take a look and see if it helps? $\endgroup$ Feb 10 at 4:20
  • $\begingroup$ Thank you, but remember that the issue is that slurm does not activate openmpi $\endgroup$ Feb 12 at 8:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .