16
$\begingroup$

I've found a page dedicated to quantum chemistry packages benchmarks, on GitHub, qmspeedtest. But most results there are several years old, and so probably outdated. Is there some place where we can find comparisons like these, but updated often, or at least more recently?

I specified quantum chemistry in the question because I'm more interested in molecular systems, modeled with atom-centered gaussian function basis sets, for example. I have almost no familiarity with software that deals with periodic systems, plane-wave based. But I think it could be a good idea if someone with more familiarity with periodic systems opened a similar question for the respective packages.

$\endgroup$
  • 2
    $\begingroup$ There is CCCBDB: cccbdb.nist.gov, but the numbers are old and members of the public can't contribute to it. I made a GitHub database called "AI Energies" where AI means ab initio but the goal is that eventually there would be enough to do AI machine learning: github.com/HPQC-LABS/AI_ENERGIES. It was only me working on it so it's very limited, but any calculation you do, can be pushed into this repository easily, and every other quantum chemist in the world can do the same, so it has potential to grow. Is there a specific molecule youre interested in comparing to a benchmark? $\endgroup$ – Nike Dattani Jun 19 at 23:16
  • $\begingroup$ Thank you, @NikeDattani. No specific molecule. $\endgroup$ – ksousa Jun 19 at 23:22
  • 1
    $\begingroup$ So what exactly do you need or want? Okay now I see that the example you gave was only showing speed tests, not energy comparisons. This means CCCBDB and AI_ENERGIES are both not what you're looking for. The example you gave is the only one I've ever seen for comparing speeds. Actually I recorded runtimes and RAM usage for almost 100 coupled cluster calculations here and over 50 FCI calculations in a similar file (in the same repo). Calculations were added as recently as 3 months ago! But let's see if someone knows a better answer. $\endgroup$ – Nike Dattani Jun 20 at 0:35
  • 1
    $\begingroup$ I noticed. Let's see if there's anything more recent or more widely used than the GitHub repo in your question. I'm not aware of anything. It's something that I think would be very valuable. My databases (linked in the previous comment) that contain CPU times and RAM usage, only contain results from my own calculations, so they are limited. If no similar database exists though, I'd be keen to polish the style of mine and encourage others to add to it. $\endgroup$ – Nike Dattani Jun 20 at 0:49
  • 1
    $\begingroup$ In my experience with DFT materials modelling programs it is surprisingly difficult to get different programs to do exactly the same calculation, which makes it very hard to give a meaningful benchmark. There are some obvious things to check (e.g. that the number of k-points is the same) but many are not so clear (e.g. how many non-local projectors are in the pseudopotentials). $\endgroup$ – Phil Hasnip Nov 12 at 16:32
19
$\begingroup$

The problem is that this is a highly volatile question. In order to meaningfully benchmark programs, you have to use the exact same compiler flags (may require heavy hacking) and use the same algorithms and parameters (accuracy, cutoffs, quadrature grids, etc). But, if a program supports many kinds of algorithms, then each of them would have to be benchmarked. In contrast, qmspeedtest is comparing apples to oranges; it is making no effort to actually ensure that the core algorithms and parameters are the same. It is for good reason that some programs explicitly ban publishing benchmark comparisons.

If you still intend to proceed, a good benchmark should look at these two core questions first:

  • speed of a single Fock build i.e. how quickly do you get a single-point energy from a given density
  • speed of gradient evaluation i.e. how quickly do you evaluate forces from a converged wave function

These are well-posed problems which are reproducible and where there is a single meaningful answer. This also means that the energy and Fock matrix / the nuclear gradient you get out from the benchmarks should agree numerically exactly between different codes. (You still do have several choices in the way to evaluate the final solution, e.g. density fitting, Cholesky decomposition, fast multipoles, etc, which may give different answers!)

Now, running a full calculation also depends on these issues:

  • cycles taken until SCF convergence i.e. how good is the default SCF guess and the default convergence accelerator for the system you're looking at
  • steps taken until geometry optimization converges i.e. how sophisticated is the geometry optimizer (use of internal coordinates? empirical force constants / exact second derivatives?)

While the first two issues, which are purely a question of speed, are somewhat important in practical applications, it is actually the latter two issues that in many cases are the most important for a workflow. If you're studying challenging molecules, you may face cases of poor SCF convergence, and this is where a flexible algorithm makes all the difference. You shouldn't care if program A solves an easy molecule in 5 steps while program B takes 7 steps to solve it, if for a challenging case program A takes 3000 steps but program B only 40. But, these issues are highly system dependent, and depend heavily on the algorithm. Using a second-order algorithm (e.g. trust region) yields more robust convergence, but even though the calculation now may converge in few steps they are much more expensive than with a simple gradient descent method; this is why you should compare apples to apples and use the exact same algorithms in all programs, and study a large variety of systems to try to cover a large sample of both "easy" and "difficult" cases.

I would note last that speed is not everything. Also the ease of use of the program and its general availability are key questions in determining which tool to use. If program A is 3x faster than program B, but B is easier/safer to use, most people would opt for program B.

Programs have also become more modular than before; this may also affect your choice: if it's easy to modify one program to do exactly what you want, it becomes your tool of choice even if it's not as fast as its competitors.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ Beautiful and elegant $\endgroup$ – Thomas Jun 20 at 14:51
  • 2
    $\begingroup$ Thanks for a great answer. In terms of using the software for a research problem, you probably also want to see how well the program scales with process count and/or threads (and/or GPGPUs etc). You might also care about memory usage, it doesn't matter how fast it will run in principle if you can't run it in practice! $\endgroup$ – Phil Hasnip Jun 29 at 1:40
  • $\begingroup$ Worth adding to this answer that different programs might be better at different problems (both in intentional and accidental ways). $\endgroup$ – taciteloquence Oct 23 at 16:36
  • $\begingroup$ Beyond that, the programs will vary on how much of your time is required to set them up for your problem, and your time is more valuable than CPU time. $\endgroup$ – taciteloquence Oct 23 at 16:36
5
$\begingroup$

I don't mean this answer to criticize your question in any way because it's actually a great question. My opinion, however, is that this is sort of the wrong question.

I think a much easier and more relevant test is not how fast some calculation is on, say, a single core, but how well the implementation scales across many nodes, each with many cores. This is because many people doing quantum chemistry have access to very large computing clusters (either through their university, national lab, company, the cloud, etc.). Despite this, many electronic structure packages do not scale well beyond even a few nodes. Sometimes this is because the method does not scale well, in which case the code cannot be blamed, and sometimes it is because the code was not written to scale well. Usually, this is because the code was originally written in like 1970.

I hesitate to be too specific because I have not used every electronic structure package to do large calculations. I have, however, done some very large calculations with NWChem and have found that the MP2 and CCSD(T) implementations scale linearly with the number of cores for a very long time. The triples part of CCSD(T) is actually known to scale linearly to the entire size of the Cori supercomputer at Nersc as implemented in NWChem.

My personal experience is that Gaussian does not scale particularly well with the number of nodes. I think most people use Gaussian for DFT though, which I have never done, so take this statement with a big grain of salt.

I have also used Molpro and out-of-the-box, it seems to be a very fast code. So, if you are only interested in single-core speed, I would guess that Molpro will fare very well. Their MCSCF implementation is famously good in my experience as well.

Also, Psi4 is an excellent, modern electronic structure package which seems to have been made with parallelism in mind, so I would think it will scale better than many packages.

This is why generally, for gas-phase ground state electronic structure I think Psi4 and NWChem are the way to go. They seem to be well-written and are free.


Generally, though, getting fair comparisons of the speed of two programs which implement the same method is very difficult. If you want to benchmark DFT, you need to use the same grid for each calculation, you need to run them on the exact same core of the same CPU. You need to make sure nothing is happening in the background of the computer you're running on that could interfere. You should run each calculation many times.

Something like HF is even harder to benchmark fairly because HF is an iterative method. So, the initial guess you use, as well as things that accelerate iterations such as DIIS, make a big difference in how long the calculation takes. Also, when comparing, you need to make sure the integral thresholds are identical since most electronic structure programs will throw out certain integrals which are guaranteed to be smaller than some value. Also, for a large calculation, you need to be careful to make sure that the integrals are stored in an identical manner since sometimes the integrals are stored in RAM and other times they are stored partially on disk and still other times they just aren't stored and get re-calculated.

For all of these reasons, performing a good-faith comparison of the speed of these packages is nearly impossible. Also, I would argue that the scaling matters much more than the zero-order speed.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Nice answer. Are you also familiar with Gamess-US? I wonder how well does it scale. I tried to install NWChem twice, on Ubuntu and Fedora. But it seems the version on both repositories is broken. I still didn't try to compile it from source. $\endgroup$ – ksousa Oct 22 at 0:09
  • 1
    $\begingroup$ Why is HF harder than DFT to benchmark? Both are generally solved with an iterative SCF approach, so DFT has the same issues with the picking a guess, DIIS, etc. $\endgroup$ – Tyberius Oct 22 at 1:27
  • $\begingroup$ @tyberius ya your right. Generally I think of DFT as being a bit easier to converge because the integrals are done over a grid, but I think that's actually not even relevant. I'll try to remember to edit to not be specific to HF only. I think the integral thresholding for DFT and HF might be different. $\endgroup$ – jheindel Oct 22 at 4:08
  • 1
    $\begingroup$ @ksousa NWChem works on Fedora. It's just an MPI program so you need to run it with MPI. Fedora comes with versions compiled for OpenMPI and for MPICH. $\endgroup$ – Susi Lehtola Oct 28 at 17:08
  • $\begingroup$ Thank you, @SusiLehtola , I will do the test again. If I still don't suceed, I will try to write a question. $\endgroup$ – ksousa Oct 28 at 21:49
4
$\begingroup$

Since I leave academia (where I used the popular gaussian package) but I want to continue to do some research in my free time, I spend the last months trying to chose the best software for quantum chemistry calculations. I think the first thing one need to know is how much the software is updated with new methods, new DFT functionals.... After a first selection based on this, one need to know what is the system that will be used for running the calculation (i.e. laptop, desktop pc, workstation, cluster) to evaluate also the parallelization of the software (see for example the scaling of nwchem in a cluster with thousands of nodes). I will do my work on a desktop pc, so it's obvious that I need a well written code to run the calculation faster. However, like others said, you can compare different packages only if you use the same parameters (grid size, convergence threshold....). In my research I also discover that most packages use external libraries (BLAS, LAPACK) for the hardest part of calculations such as matrix multiplications, integral evaluations... So I think the choice of the software may be done mostly on the basis of the frequency of the update. In the end, I think that most of the performance for a calculation with the same parameters could be attributed to : the optimization in the compilation phase (optimization flags); the choice of good libraries (ATLAS vs OPENBLAS vs MKL...; look for some benchmark and you will see how much they perform very very different); the system you are using (linux version, linux scheduler, used file system, optimized kernel)... I' m still studying about this topic and these are my actual findings. What I would like to do in the near future is try to optimize all these parameters to see how much one can gain in term of time.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Yes, we are working on the performance benchmark of many quantum chemistry program packages. If you're interested in, you can visit our Github repository: https://github.com/r2compchem/benchmark-qm.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ The benchmarks are senseless since they're comparing different calculations. E.g. Gaussian'16 shows up much slower than Gaussian'09, since Gaussian Inc changed their default grids in Gaussian'16. The results you get with Gaussian'16 are more accurate than the ones with Gaussian'09. (Also, IIRC publishing benchmarking data with Gaussian is strictly against their user agreement.) $\endgroup$ – Susi Lehtola Oct 21 at 16:10
  • 1
    $\begingroup$ Thank you for the comment. Please check "Note for computational details", and of course more reasonable compare will come out later. :) Any default settings in packages should already be taken into consideration by developers. $\endgroup$ – r2compchem Oct 22 at 18:24
  • $\begingroup$ @r2compchem which "developers"? Developers of your benchmark set, or or the quantum chemistry software? If you mean the latter, speed should not be the priority when determining default settings. It is much more important that software gives a reasonably accurate answer than a quick answer. Users may miss important parameters that they should investigate, and it is better that the effect is that they waste computer time rather than their results being rendered meaningless. $\endgroup$ – Phil Hasnip Nov 12 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.