Almost all HPC servers use Linux. Therefore, in almost all cases, familiarity with Linux is crucial, especially the basic commands (e.g.
Connecting to the server
Connecting to a Linux-based HPC server is now always in my experience done through SSH (in the 1990s I used Telnet to log into the University of Toronto servers, but I can't recall using anything other than SSH in the mid-2000s or later). To log into an HPC server with SSH, you can run the following command in the default terminal in Linux, MacOS or Windows Subsystem for Linux, or in something like PuTTY in pure Windows (assuming your username is
camilla and the server's name is
materials.stackexchange.com, for a hypothetical example):
ssh [email protected]
As long as there's no problems, you will next be asked for your password, and upon entering it correctly you will be able to access the server. Usually you will be given a temporary password at the beginning and will be highly encouraged (or forced) to change it immediately. The passwd command will allow you to do this if you are not already forced to do it upon your first successful login. The spelling of that command is a bit strange, and it can be hard to remember whether or leave out the letter "a" or the letters "or", so I spelled it correctly for you in the hyperlinked words in this paragraph.
Transfer of files
It is very likely that you will have input scripts or data files on a computer that you're currently using, and will want those files to be accessible on the HPC server. A very simple way to transfer files to and from an HPC server is using
scp (in the 1990s I used FTP, which stands for "file transfer protocol" but now exclusively use
scp which is just a "secure" analog of the previously mentioned and very basic
cp command). The following command is run on your local machine and is for copying files from your local machine to the above hypothetical server (the important part to remember is the colon symbol
: which is easy to forget because we far more often use
cp without it, than we use
scp with it):
scp file.txt [email protected]:
From your local machine you can also copy files from the server to your current directory on your local machine, for example with the following command (the main difference from the above command is the order):
scp [email protected]:/path/to/file.txt .
Most HPC servers do not allow you to have full control of the server, so you are limited in how you can install things. For this reason, and because it's wasteful for 100 users to each install their own copy of a 10GB software, servers often come with most major programs pre-installed. These pre-installed programs are often made available with the lmod system. The following
lmod commands are the most commonly used ones, and my example below is for Python but could be used for other programs by changing the last word in the each command accordingly:
module load python # load the system's default version of Python
module unload python # unload whatever version of Python was loaded
module spider python # learn information about the various versions of Python modules that are made available to you
To see which modules you already have loaded, you can run:
and to see which ones are available to load via
module load, you can run:
When programs are not available via the
lmod system, then your next option might be
pip, but this can be significantly more complicated than using
pip on your local machine, for example Compute Canada which serves almost every Canadian academic researcher has a "wheel house" for Python wheels, which is described here, and
pip install has to actually be run every time you run a job (so it's part of the "job submission script") as described here: How can I build the wheels necessary for a quick installation of PySCFad on a "compute node"?. Likewise, compiling software the "traditional way" from "source" can be significantly more complicated than on a local machine as you may have to "load" compilers as modules (among other things), and you can see an example of the sequence of commands for this type of installation here: Is this "internal error" reproducible when compiling OpenMolcas on other Intel compilers?
Explanations of how to use SLURM, LoadLeveler, and SGE are provided in answers here: How can I submit jobs to an HPC scheduler?. This discussion is also related: Job scheduler alternatives. The job-scheduler tag here on MMSE is for asking questions about schedulers like the ones mentioned in this paragraph.
Logging out of the server
When you run the command:
the terminal that you have been using all this time will return you to the directory on your local machine at which you were present at the time of logging into the server.