I wonder, if someone has run (runs) parallel Orca calculations on at least two nodes without a work load manager (scheduler)? I am asking because I have several interconnected servers (nodes) I'd like to run Orca on but there is no scheduler and I could not make Orca run on more than one server (node with several CPU-cores) so far.
1 Answer
Not sure if ORCA can be run using a user-chosen MPI runner but there are a few steps that need to be done in order to run ORCA (or any other MPI program) on a distributed/interconnect cluster (assuming that you are using an Ubuntu server with OpenMPI already installed (check again that the version of your OpenMPI is supported by ORCA). You have root permission or are sudoer).
- You need to deal with a
hosts
file on the master node of your cluster. In this file, you can specify the name of the worker machines you would like to do the calculation, for example:
$ cat /etc/hosts
127.0.0.1 localhost
172.xx.xx.1 worker1
172.xx.xx.2 worker2
172.xx.xx.3 worker3
172.xx.xx.4 worker4
Note that, make sure that each worker's manager
file has exactly the same content as the hosts
file of the master node.
- Setup SSH for it
The next step would be setting up SSH because this is the way that all machines communicate with each other. This can be done simply as you can find elsewhere on the internet with SSH tutorial:
Install openssh-server
and generate a new key
sudo apt-get install openssh-server
ssh-keygen -t dsa
Add the newly generated key to each worker, e.g
ssh-copy-id worker1
Then enable a passwordless SSH connection
eval `ssh-agent`
ssh-add ~/.ssh/id_dsa
- Setup NFS in
manager
which the worker nodes mount to exchange data (You can skip this step if the NFS is already set up)
On NFS-server
# Install the required package
sudo apt-get install nfs-kernel-server
cd $HOME/
mkdir scratch
# Edit `/etc/exports` and add the following
/home/YOUR_USER_NAME/scratch *(rw,sync,no_root_squash,no_subtree_check)
exportfs -a
sudo service nfs-kernel-server restart
You can easily find the meaning of each keyword on the internet
On NFS-worker (actually you do almost the same as NFS-server)
# Install required package
sudo apt-get install nfs-common
cd $HOME/
mkdir scratch
sudo mount -t nfs manager:/home/YOUR_USER_NAME/scratch ~/scratch
You can use df -h
to check whether the directory is already mounted.
- Running an MPI program
mpirun -np 8 --hosts manager,worker1,worker2 FULL_PATH_TO_YOUR_ORCA
You should try running a simple MPI program, e.g. printing Hello world, to make sure the process above works. You can find more details on the internet, like this website: http://selkie.macalester.edu/csinparallel/modules/Patternlets/build/html/MessagePassing/RunningMPI.html.
FWIW, hope this helps
-
$\begingroup$ Hi, many thanks for your response. The point is, Orca has its own (parallel) launcher (one starts a calculation just simply with FULL_PATH_TO_YOUR_ORCA without mpirun or mpiexec), which takes all needed information from the scheduler environment, if I understand it correctly. At least, it works problem-free on clusters with scheduler (SLURM or whatever). One can feed the launcher with a file containing names of the workers (sort of the host file). Well, one also has to take care of the nodes being in the same directory. Maybe something else is needed ... $\endgroup$– EvGeniyMay 23 at 13:34
-
$\begingroup$ @EvGeniy I see your point. Maybe you had better ask in ORCA forum about this. $\endgroup$ May 26 at 5:33
mpirun
ormpiexec
didn't work? $\endgroup$