# Job scheduler alternatives

The power of computation required to perform a DFT calculation increases with the size of the system. Therefore, for some systems, is mandatory to use high-performance-computers, clusters, and grids. One of the main ingredients in their administration is a job scheduler. There are some possibilities, such as Slurm and PBS but the configuration is not a simple and intuitive task, even with some tutorials spread on-line. Are there alternatives to more "user-friendly" schedulers, capable to perform basic queue administration?

• SLURM is now the most popular, but 10 years ago it was PBS/TORQUE. I also find LoadLeveller really easy to use but it might be only for IBM machines? I've also used other schedulers in the past, like the one by Sun microsystems which doesn't exist anymore. May 18 '20 at 15:33
• @Cody he might be a SysAdmin. May 18 '20 at 15:34
• @CodyAldaz It is maybe a question that can be useful for a wider audience that HPC sysadmins. There are several labs where small clusters are built and managed often by people with no formal HPC training. Also, there are projects where you want to automate the calculation of maybe 10s-100s of similar structures on a bigger workstation.
– Greg
May 18 '20 at 15:40
• I have never had the urge to research that, but there actually is a list on Wikipedia. It's short, and of course it doesn't say anything about ease of use. May 18 '20 at 17:23

This depends on what you want to do:

If you have any kind of distributed computing (this might be 5 PCs in a lab with a central node), you will not be able to avoid using a scheduler to distribute jobs.

Running jobs locally, people have written "first come first served" schedulers in for example bash. However I would suggest that this is not the best choice.

Even for just your local usage, setting up a scheduler is the better way to go. (I have done this in the past using slurm. I also remember using the SunGrid Scheduler but I don't think I configured it... Or maybe I did and then it died in an Upgrade and was replaced by SLURM?) For example using a scheduler allows you to dedicate only a part of the available CPU resources to computations while retaining some resources for day-to day tasks. Especially with today's multi-core CPUs, this is an enticing proposition. Schedulers can also be more clever than a first come first served algorithm, slipping in smaller jobs while a large calculation awaits its execution which a simple shell script will typically no do.

It turns out that for a local installation, SLURM is really not that bad in terms of configuration. The biggest issue is that you (in my past experience pre-2018/19 I have not set it up since) cannot use the loopback IP or localhost for the calculation node. Thus you are obliged to assign a static IP address (in the router) or to update the DHCP assigned IP-address in the config file when it changes in addition to using a specific host name for your PC.

Another advice is to install SLURM from the repository of your Linux distribution - this will ensure the correct dependencies are included.

A major simplification on a "personal" computer is, that you can assume that resource demands will be honest. -> So while the scheduler takes the requested resources into account, there is no need to monitor resource usage.

For the details of the configuration I will point at the documentation as this is eventually the only up-to date source: https://slurm.schedmd.com (There was a funny bug about 3 years ago in how threads and processor cores were handled between openSUSE and Ubuntu which was not identical which made some parameter choices non-intuitive...)

Another quick PS: SLURM offer a config tool in the form of a website to generate a configuration file. I believe it is also installed with the scheduler, the online version is here: https://slurm.schedmd.com/configurator.html

In the end, every IT tool will seem daunting at first until the language used becomes clear. Setting up a scheduler may seem daunting at first, but once the process is clear, it is not actually a difficult task to execute. (Balancing user shares and fair scheduling is yet another topic...) There also isn't going to be a major difference between different scheduler as they invariably need to carry out similar task. The scheduler needs to know what resources are available on the computer node and where they are. - The computer node and head node can be the same. Thus the differences in the configuration are in the end cosmetic.

Selecting a scheduler which is widely used is insofar advisable as support is not likely to disappear soon. I used a cluster that employed the Sun Grid Scheduler (SGE) which no longer exists. There is a spiritual successor, called Son of Grid Engine (SGE) but it might not be non-trivial to run on a modern Linux-Distribution. (Icon libraries, other libraries.) Selecting a mainstream scheduler such as SLURM or PBS (same syntax as SGE in the submission scripts, thanks for the comment pointing it out) will mean that you are familiar with a standard tool. I.e. If you bring along experience with a standard scheduler, this is valuable to others both from being able to use established systems elsewhere to helping new users.

On the warning side: Be careful that you make good use of your resources. I used to run calculations on an old AMD PC that had a total of 48 CPU cores but only one internal harddrive... - It turned out that running many jobs in parallel was slower than running then with more threads per job in fewer jobs, even leaving parts of the CPU unused due to the I/O bottleneck. To get the best performance, you will need to actually determine what kind of calculation you wish to carry out and what bottlenecks your work will hit. For example, I used a code that even with DFT used temporary files in the calculations. And then the DLPNO-CCSD(T) calculations wrote large temporary files that had a huge I/O impact... - Of course minimising the number of parallel jobs was important in this case.

PS: I anybody wants to turn this into a Wiki and update it, let me know and I'll tick the box.

I hope this slightly rambly post helps.

• Oracle acquired Sun, so I guess the commercial successor to the sun grid engine would be oracle grid engine. May 18 '20 at 20:57
• I wish I could run slurm under macOS, but it's not presently supported May 21 '20 at 13:24
• @jeguyer I am not surprised, given that large scale calculations are performed nearly always on a flavour of Linux. In the top500 clusters of the world, you have the choice of Linux top500.org/statistics/sublist - There are some people who insist on using for example Windows servers, but they are the rarity and thus get less support. The first post links to a Wikipedia post on schedulers, and it seems some for Macs do exist. May 21 '20 at 15:46
• @DetlevCM Oh, I understand. Similarly to the OP, though, there are times when I'm just running jobs on my local machine and it would be nice to be able to use the same workflow. May 21 '20 at 18:14
• @Martin-マーチン sure but Oracle Grid Engine is also obsolete since it got sold to Univa; see en.wikipedia.org/wiki/Oracle_Grid_Engine, and Univa Grid Engine is proprietary. May 26 '20 at 14:55

I fully endorse @DetlevCM's advocacy for using "proper" job schedulers, but for comparison, there are some tools geared toward intensive DFT runs that might be more approachable for your application:

• Materials Project's FireWorks is a "code for defining, managing, and executing workflows". It looks like it sits on top of schedulers like SLURM, but may be more approachable.
• OQMD from the Wolverton group at Northwestern includes facilities for "automating density-functional calculations". I don't know how generalizable those facilities are.
• NIST's JARVIS "is a repository designed to automate materials discovery using classical force-field, density functional theory, machine learning calculations and experiments". I'm not sure how adaptable its internal scripting and workflow is to other applications.
• signac from the Glotzer group at UMich has workflow and data pipelining capabilities. Their applications are MD-centric, but my impression of signac is that it's more general.
• Missing from this list are Atomate and AiiDA, although the former is closely tied to FireWorks. Also, the tool associated with the OQMD is qmpy. May 25 '20 at 7:09
• Thanks. I've edited my answer to remove any suggestion that my list was complete. I'm not a DFT modeler, so haven't used any of these tools, but know people involved in all of them. May 26 '20 at 14:42

One of my colleagues uses a very hacky solution to this. I don't recommend using it, but I can imagine it being useful in certain narrow applications, so I'll post it here anyways.

Let's call this person Alex. Alex is a PI and admin for a cluster used by a small research group. The cluster has a normal queuing system, but unlike most clusters, it is not set up to kill jobs running outside the queue on the compute nodes. Instead, it just sets them to nice -19. Alex has a lot of very long (months+) low-memory, low-priority simulations. Instead of running them through the queue, they log onto a not-too-busy node and start their jobs manually. They don't keep track of where any particular job is running. The jobs regularly check a text file that has a flag in it, to kill that job they just edit that file to have the 'kill' flag.

• Since jobs are run with nice -19 priority, if the queue starts a job on that node, the job is effectively paused until the node frees up again (although it will run in the background when there are spare cycles).