Job scheduler alternatives

Question

The power of computation required to perform a DFT calculation increases with the size of the system. Therefore, for some systems, is mandatory to use high-performance-computers, clusters, and grids. One of the main ingredients in their administration is a job scheduler. There are some possibilities, such as Slurm and PBS but the configuration is not a simple and intuitive task, even with some tutorials spread on-line. Are there alternatives to more "user-friendly" schedulers, capable to perform basic queue administration?

Answer 1

This depends on what you want to do:

If you have any kind of distributed computing (this might be 5 PCs in a lab with a central node), you will not be able to avoid using a scheduler to distribute jobs.

Running jobs locally, people have written "first come first served" schedulers in for example bash. However I would suggest that this is not the best choice.

Even for just your local usage, setting up a scheduler is the better way to go. (I have done this in the past using slurm. I also remember using the SunGrid Scheduler but I don't think I configured it... Or maybe I did and then it died in an Upgrade and was replaced by SLURM?) For example using a scheduler allows you to dedicate only a part of the available CPU resources to computations while retaining some resources for day-to day tasks. Especially with today's multi-core CPUs, this is an enticing proposition. Schedulers can also be more clever than a first come first served algorithm, slipping in smaller jobs while a large calculation awaits its execution which a simple shell script will typically no do.

It turns out that for a local installation, SLURM is really not that bad in terms of configuration. The biggest issue is that you (in my past experience pre-2018/19 I have not set it up since) cannot use the loopback IP or localhost for the calculation node. Thus you are obliged to assign a static IP address (in the router) or to update the DHCP assigned IP-address in the config file when it changes in addition to using a specific host name for your PC.

Another advice is to install SLURM from the repository of your Linux distribution - this will ensure the correct dependencies are included.

A major simplification on a "personal" computer is, that you can assume that resource demands will be honest. -> So while the scheduler takes the requested resources into account, there is no need to monitor resource usage.

For the details of the configuration I will point at the documentation as this is eventually the only up-to date source: https://slurm.schedmd.com (There was a funny bug about 3 years ago in how threads and processor cores were handled between openSUSE and Ubuntu which was not identical which made some parameter choices non-intuitive...)

Another quick PS: SLURM offer a config tool in the form of a website to generate a configuration file. I believe it is also installed with the scheduler, the online version is here: https://slurm.schedmd.com/configurator.html

In the end, every IT tool will seem daunting at first until the language used becomes clear. Setting up a scheduler may seem daunting at first, but once the process is clear, it is not actually a difficult task to execute. (Balancing user shares and fair scheduling is yet another topic...) There also isn't going to be a major difference between different scheduler as they invariably need to carry out similar task. The scheduler needs to know what resources are available on the computer node and where they are. - The computer node and head node can be the same. Thus the differences in the configuration are in the end cosmetic.

Selecting a scheduler which is widely used is insofar advisable as support is not likely to disappear soon. I used a cluster that employed the Sun Grid Scheduler (SGE) which no longer exists. There is a spiritual successor, called Son of Grid Engine (SGE) but it might not be non-trivial to run on a modern Linux-Distribution. (Icon libraries, other libraries.) Selecting a mainstream scheduler such as SLURM or PBS (same syntax as SGE in the submission scripts, thanks for the comment pointing it out) will mean that you are familiar with a standard tool. I.e. If you bring along experience with a standard scheduler, this is valuable to others both from being able to use established systems elsewhere to helping new users.

On the warning side: Be careful that you make good use of your resources. I used to run calculations on an old AMD PC that had a total of 48 CPU cores but only one internal harddrive... - It turned out that running many jobs in parallel was slower than running then with more threads per job in fewer jobs, even leaving parts of the CPU unused due to the I/O bottleneck. To get the best performance, you will need to actually determine what kind of calculation you wish to carry out and what bottlenecks your work will hit. For example, I used a code that even with DFT used temporary files in the calculations. And then the DLPNO-CCSD(T) calculations wrote large temporary files that had a huge I/O impact... - Of course minimising the number of parallel jobs was important in this case.

PS: I anybody wants to turn this into a Wiki and update it, let me know and I'll tick the box.

I hope this slightly rambly post helps.

Answer 2

I fully endorse @DetlevCM's advocacy for using "proper" job schedulers, but for comparison, there are some tools geared toward intensive DFT runs that might be more approachable for your application:

• Materials Project's FireWorks is a "code for defining, managing, and executing workflows". It looks like it sits on top of schedulers like SLURM, but may be more approachable.
• OQMD from the Wolverton group at Northwestern includes facilities for "automating density-functional calculations". I don't know how generalizable those facilities are.
• NIST's JARVIS "is a repository designed to automate materials discovery using classical force-field, density functional theory, machine learning calculations and experiments". I'm not sure how adaptable its internal scripting and workflow is to other applications.
• signac from the Glotzer group at UMich has workflow and data pipelining capabilities. Their applications are MD-centric, but my impression of signac is that it's more general.

Answer 3

One of my colleagues uses a very hacky solution to this. I don't recommend using it, but I can imagine it being useful in certain narrow applications, so I'll post it here anyways.

Let's call this person Alex. Alex is a PI and admin for a cluster used by a small research group. The cluster has a normal queuing system, but unlike most clusters, it is not set up to kill jobs running outside the queue on the compute nodes. Instead, it just sets them to nice -19. Alex has a lot of very long (months+) low-memory, low-priority simulations. Instead of running them through the queue, they log onto a not-too-busy node and start their jobs manually. They don't keep track of where any particular job is running. The jobs regularly check a text file that has a flag in it, to kill that job they just edit that file to have the 'kill' flag.

Advantages of this system:

• Since jobs are run with nice -19 priority, if the queue starts a job on that node, the job is effectively paused until the node frees up again (although it will run in the background when there are spare cycles).
• This allows for super long simulations without tying up the queue or preventing anyone else from running jobs.
• It's basically like a slow cooker.

Disadvantages:

• Complicated and labor-intensive to maintain and keep track of all the running jobs, know how to turn them off, etc.
• Only sustainable for one person to do this on a given system (and really only when that person is in charge).
• Also really only makes sense for low-memory simulations where you really don't care when you get the final result.
• Probably has other weird side effects. It's easy to lose track of your running jobs or not know which is which.
• Seriously don't do this.