EDIT: This seems to be a case of improper MPI/OpenMP defaults, but the general information below still applies in general.
MPI And OpenMP
The two major parallel programming methods in HPC software are MPI and OpenMP. MPI parallelizes across processes, OpenMP across threads. (The difference is explained in this StackOverflow question.)
Both MPI and OpenMP must be programmed into the software by the developers and compiled correctly. An end-user can't affect that process, but it is their responsibility to call the parallelized program with the right settings.
MPI settings are usually set through flags to the mpirun program. For example, this runs my_program with the number of processes set to 4:
mpirun -np 4 my_program
(With SLURM's srun the -np flag can sometimes be omitted, as SLURM will try to deduce the number of processes from the resources allocated. Please AYSA whether you should use srun or mpirun -- often one option is correct and the other will crash or slow your program.)
OpenMP settings are usually set as environment variables. For example, writing this before the mpirun or srun command sets each process to use one thread:
export OMP_NUM_THREADS=1
Now, for a compute-intensive job one thread needs one core*. Suppose two threads are "squished" onto one core -- the core is forced to multitask between those two threads, and each time it switches threads it has a reset course (plus the core's own scheduler's time spent "juggling" the threads).
In Camilla's answer she noted the following info in the program logs:
Parallel version (MPI & OpenMP), running on 8 processor cores
Number of MPI processes: 4
Threads/MPI process: 2
while her SLURM script read:
#SBATCH --ntasks=4
#SBATCH --ntasks-per-core=1
So SLURM allocated 4 cores, but her OMP_NUM_THREADS seems to have defaulted to 2, which together with mpirun -np 4 results in 8 threads in total, which will run incredibly slowly when "juggled" on 4 cores. Setting OMP_NUM_THREADS to 1 using the above command seems to have fixed it.
*Unless your cluster and the software you're using supports hyperthreading. AYSA.
Original Answer
Ask. Your. System. Administrator. :)
When you see "AYSA" throughout this answer it means you need to ask your system administrator how to best run jobs on the cluster. We don't know what cluster you use, and even if we did, we don't run it. Your administrators do. If you are good friends with them, they can make your life much better.
And if you say to your system administrators "okay so you're telling me to do XYZ, but a stranger on the Internet said ABC instead", you will not be good friends with them.
So, please AYSA. Having said that, some sysadmins are allergic to what I'd call Google questions -- the sort of questions that, in theory, any reasonably smart scientist can answer for themselves on Google. (The truth is Google sucks these days, and it's not getting any better.) So what I'll do is give you the Google Q&A and also give you more detailed questions to AYSA. Make sense? Here we go.
The First Rule of using a cluster is that The Cluster Is Not Just A Bigger Faster Computer. Here's an analogy: a 4-bedroom house is basically the same as a 2-bedroom house but twice as big. But a 2000-bedroom building isn't a house scaled up 1000x, it's a hotel or an apartment block, and it's a completely different building with completely different principles.
Firstly, each individual core is likely slower than anything that's on your PC. If your job can finish running on a two-processor laptop it should take longer on two cluster cores. That makes sense when you think about the economics: your cluster couldn't afford to buy hundreds of thousands of very fast cores, and it doesn't need to, because the job of the cluster is to run jobs on hundreds or thousands of cores which could never be run on just two cores, no matter how fast. If you're using your cluster because you have to run thousands of small little jobs, AYSA about how to "batch" jobs together so the scheduler can block off the batch like a regular big job.
Secondly, the file system is distributed. Just as cluster cores are not just Big Fast Processors, the cluster's files don't live in One Big Hard Drive. (Imagine if they did -- that one hard drive would have thousands of processes trying to write or read from it at once and it would instantly crash and burn!) Instead there is a distributed file system which reliably collates changes made by all these processors and then makes them consistent. As a tradeoff, you shouldn't always expect an output file to be instantly written the moment a process has produced it -- you may have some delay, especially with a HPC-optimised program that only writes to hard disk infrequently if used with a distributed file system. So don't just keep refreshing your home directory. AYSA what to expect and how to best use the file system.
Thirdly, parallelizing your application is tricky. There's MPI, OpenMP, various Python solutions, and all of them need the correct options to be set -- otherwise you risk allocating 16 cores to an application only to have it run on one or two cores. AYSA, but before you do, you should run scaling benchmarks. If you go up from 2 cores to 4 cores, does it finish twice as fast? What about from 4 to 8? What if you go from 2 cores to 8 cores, but also make the problem four times larger? The first two are examples of strong scaling, which usually drops off more quickly than the weak scaling of the third example. If you go from 2 to 4 to 8 cores and see the time stay the same -- you're probably not asking correctly for cores.
And one important thing to understand is your cluster's chiplet and node sizes. Please don't AYSA until you've done some basic searching, since most clusters publicly describe what processors they use and how many are in each node. Node sizes are relevant to larger jobs -- for example, if each node has 64 cores and you're running a big job, it may be worth using either 64 or 128 cores and booking out whole nodes rather than asking for 96 cores (the scheduler might not even allow you to ask for that).
Chiplet sizes are more relevant to smaller jobs, especially if your cluster uses the recent AMD EPYC processors. Here's an example from documentation of a cluster I use:
See how, on the right, there are eight cores sharing each L3 cache (essentially superfast RAM used directly during number crunching)? On this cluster, you do best asking for groups of 8 cores so that your program has that L3 cache all to itself, making sure the scheduler does actually give you the group of 8 instead of splashing your job all over various cores. (If you're using SLURM, AYSA about using the -m block:block:block directive or whether they recommend alternatives.) Otherwise, if your program is sharing cache with another job, there may be contention slowing your job down (or if your program is aggressive you will slow their job).
All the best! Please report back on your findings.
