15
$\begingroup$

The power of computation required to perform a DFT calculation increases with the size of the system. Therefore, for some systems, is mandatory to use high-performance-computers, clusters, and grids. One of the main ingredients in their administration is a job scheduler. There are some possibilities, such as Slurm and PBS but the configuration is not a simple and intuitive task, even with some tutorials spread on-line. Are there alternatives to more "user-friendly" schedulers, capable to perform basic queue administration?

$\endgroup$
  • 1
    $\begingroup$ SLURM is now the most popular, but 10 years ago it was PBS/TORQUE. I also find LoadLeveller really easy to use but it might be only for IBM machines? I've also used other schedulers in the past, like the one by Sun microsystems which doesn't exist anymore. $\endgroup$ – Nike Dattani May 18 at 15:33
  • $\begingroup$ @Cody he might be a SysAdmin. $\endgroup$ – Nike Dattani May 18 at 15:34
  • 4
    $\begingroup$ @CodyAldaz It is maybe a question that can be useful for a wider audience that HPC sysadmins. There are several labs where small clusters are built and managed often by people with no formal HPC training. Also, there are projects where you want to automate the calculation of maybe 10s-100s of similar structures on a bigger workstation. $\endgroup$ – Greg May 18 at 15:40
  • $\begingroup$ I have never had the urge to research that, but there actually is a list on Wikipedia. It's short, and of course it doesn't say anything about ease of use. $\endgroup$ – Martin - マーチン May 18 at 17:23
14
$\begingroup$

This depends on what you want to do:

If you have any kind of distributed computing (this might be 5 PCs in a lab with a central node), you will not be able to avoid using a scheduler to distribute jobs.

Running jobs locally, people have written "first come first served" schedulers in for example bash. However I would suggest that this is not the best choice.

Even for just your local usage, setting up a scheduler is the better way to go. (I have done this in the past using slurm. I also remember using the SunGrid Scheduler but I don't think I configured it... Or maybe I did and then it died in an Upgrade and was replaced by SLURM?) For example using a scheduler allows you to dedicate only a part of the available CPU resources to computations while retaining some resources for day-to day tasks. Especially with today's multi-core CPUs, this is an enticing proposition. Schedulers can also be more clever than a first come first served algorithm, slipping in smaller jobs while a large calculation awaits its execution which a simple shell script will typically no do.

It turns out that for a local installation, SLURM is really not that bad in terms of configuration. The biggest issue is that you (in my past experience pre-2018/19 I have not set it up since) cannot use the loopback IP or localhost for the calculation node. Thus you are obliged to assign a static IP address (in the router) or to update the DHCP assigned IP-address in the config file when it changes in addition to using a specific host name for your PC.

Another advice is to install SLURM from the repository of your Linux distribution - this will ensure the correct dependencies are included.

A major simplification on a "personal" computer is, that you can assume that resource demands will be honest. -> So while the scheduler takes the requested resources into account, there is no need to monitor resource usage.

For the details of the configuration I will point at the documentation as this is eventually the only up-to date source: https://slurm.schedmd.com (There was a funny bug about 3 years ago in how threads and processor cores were handled between openSUSE and Ubuntu which was not identical which made some parameter choices non-intuitive...)

Another quick PS: SLURM offer a config tool in the form of a website to generate a configuration file. I believe it is also installed with the scheduler, the online version is here: https://slurm.schedmd.com/configurator.html

In the end, every IT tool will seem daunting at first until the language used becomes clear. Setting up a scheduler may seem daunting at first, but once the process is clear, it is not actually a difficult task to execute. (Balancing user shares and fair scheduling is yet another topic...) There also isn't going to be a major difference between different scheduler as they invariably need to carry out similar task. The scheduler needs to know what resources are available on the computer node and where they are. - The computer node and head node can be the same. Thus the differences in the configuration are in the end cosmetic.

Selecting a scheduler which is widely used is insofar advisable as support is not likely to disappear soon. I used a cluster that employed the Sun Grid Scheduler (SGE) which no longer exists. There is a spiritual successor, called Son of Grid Engine (SGE) but it might not be non-trivial to run on a modern Linux-Distribution. (Icon libraries, other libraries.) Selecting a mainstream scheduler such as SLURM or PBS (same syntax as SGE in the submission scripts, thanks for the comment pointing it out) will mean that you are familiar with a standard tool. I.e. If you bring along experience with a standard scheduler, this is valuable to others both from being able to use established systems elsewhere to helping new users.

On the warning side: Be careful that you make good use of your resources. I used to run calculations on an old AMD PC that had a total of 48 CPU cores but only one internal harddrive... - It turned out that running many jobs in parallel was slower than running then with more threads per job in fewer jobs, even leaving parts of the CPU unused due to the I/O bottleneck. To get the best performance, you will need to actually determine what kind of calculation you wish to carry out and what bottlenecks your work will hit. For example, I used a code that even with DFT used temporary files in the calculations. And then the DLPNO-CCSD(T) calculations wrote large temporary files that had a huge I/O impact... - Of course minimising the number of parallel jobs was important in this case.

PS: I anybody wants to turn this into a Wiki and update it, let me know and I'll tick the box.

I hope this slightly rambly post helps.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Oracle acquired Sun, so I guess the commercial successor to the sun grid engine would be oracle grid engine. $\endgroup$ – Martin - マーチン May 18 at 20:57
  • $\begingroup$ I wish I could run slurm under macOS, but it's not presently supported $\endgroup$ – jeguyer May 21 at 13:24
  • $\begingroup$ @jeguyer I am not surprised, given that large scale calculations are performed nearly always on a flavour of Linux. In the top500 clusters of the world, you have the choice of Linux top500.org/statistics/sublist - There are some people who insist on using for example Windows servers, but they are the rarity and thus get less support. The first post links to a Wikipedia post on schedulers, and it seems some for Macs do exist. $\endgroup$ – DetlevCM May 21 at 15:46
  • $\begingroup$ @DetlevCM Oh, I understand. Similarly to the OP, though, there are times when I'm just running jobs on my local machine and it would be nice to be able to use the same workflow. $\endgroup$ – jeguyer May 21 at 18:14
  • 1
    $\begingroup$ @Martin-マーチン sure but Oracle Grid Engine is also obsolete since it got sold to Univa; see en.wikipedia.org/wiki/Oracle_Grid_Engine, and Univa Grid Engine is proprietary. $\endgroup$ – Susi Lehtola May 26 at 14:55
9
$\begingroup$

I fully endorse @DetlevCM's advocacy for using "proper" job schedulers, but for comparison, there are some tools geared toward intensive DFT runs that might be more approachable for your application:

  • Materials Project's FireWorks is a "code for defining, managing, and executing workflows". It looks like it sits on top of schedulers like SLURM, but may be more approachable.
  • OQMD from the Wolverton group at Northwestern includes facilities for "automating density-functional calculations". I don't know how generalizable those facilities are.
  • NIST's JARVIS "is a repository designed to automate materials discovery using classical force-field, density functional theory, machine learning calculations and experiments". I'm not sure how adaptable its internal scripting and workflow is to other applications.
  • signac from the Glotzer group at UMich has workflow and data pipelining capabilities. Their applications are MD-centric, but my impression of signac is that it's more general.
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Missing from this list are Atomate and AiiDA, although the former is closely tied to FireWorks. Also, the tool associated with the OQMD is qmpy. $\endgroup$ – Andrew Rosen May 25 at 7:09
  • $\begingroup$ Thanks. I've edited my answer to remove any suggestion that my list was complete. I'm not a DFT modeler, so haven't used any of these tools, but know people involved in all of them. $\endgroup$ – jeguyer May 26 at 14:42
4
$\begingroup$

One of my colleagues uses a very hacky solution to this. I don't recommend using it, but I can imagine it being useful in certain narrow applications, so I'll post it here anyways.

Let's call this person Alex. Alex is a PI and admin for a cluster used by a small research group. The cluster has a normal queuing system, but unlike most clusters, it is not set up to kill jobs running outside the queue on the compute nodes. Instead, it just sets them to nice -19. Alex has a lot of very long (months+) low-memory, low-priority simulations. Instead of running them through the queue, they log onto a not-too-busy node and start their jobs manually. They don't keep track of where any particular job is running. The jobs regularly check a text file that has a flag in it, to kill that job they just edit that file to have the 'kill' flag.

Advantages of this system:

  • Since jobs are run with nice -19 priority, if the queue starts a job on that node, the job is effectively paused until the node frees up again (although it will run in the background when there are spare cycles).
  • This allows for super long simulations without tying up the queue or preventing anyone else from running jobs.
  • It's basically like a slow cooker.

Disadvantages:

  • Complicated and labor-intensive to maintain and keep track of all the running jobs, know how to turn them off, etc.
  • Only sustainable for one person to do this on a given system (and really only when that person is in charge).
  • Also really only makes sense for low-memory simulations where you really don't care when you get the final result.
  • Probably has other weird side effects. It's easy to lose track of your running jobs or not know which is which.
  • Seriously don't do this.
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.