Is there a set of updated, comprehensive benchmarks for speed comparison between different quantum chemistry packages?

Question

I've found a page dedicated to quantum chemistry packages benchmarks, on GitHub, qmspeedtest. But most results there are several years old, and so probably outdated. Is there some place where we can find comparisons like these, but updated often, or at least more recently?

I specified quantum chemistry in the question because I'm more interested in molecular systems, modeled with atom-centered gaussian function basis sets, for example. I have almost no familiarity with software that deals with periodic systems, plane-wave based. But I think it could be a good idea if someone with more familiarity with periodic systems opened a similar question for the respective packages.

Answer 1

The problem is that this is a highly volatile question. In order to meaningfully benchmark programs, you have to use the exact same compiler flags (may require heavy hacking) and use the same algorithms and parameters (accuracy, cutoffs, quadrature grids, etc). But, if a program supports many kinds of algorithms, then each of them would have to be benchmarked. In contrast, qmspeedtest is comparing apples to oranges; it is making no effort to actually ensure that the core algorithms and parameters are the same. It is for good reason that some programs explicitly ban publishing benchmark comparisons.

If you still intend to proceed, a good benchmark should look at these two core questions first:

• speed of a single Fock build i.e. how quickly do you get a single-point energy from a given density
• speed of gradient evaluation i.e. how quickly do you evaluate forces from a converged wave function

These are well-posed problems which are reproducible and where there is a single meaningful answer. This also means that the energy and Fock matrix / the nuclear gradient you get out from the benchmarks should agree numerically exactly between different codes. (You still do have several choices in the way to evaluate the final solution, e.g. density fitting, Cholesky decomposition, fast multipoles, etc, which may give different answers!)

Now, running a full calculation also depends on these issues:

• cycles taken until SCF convergence i.e. how good is the default SCF guess and the default convergence accelerator for the system you're looking at
• steps taken until geometry optimization converges i.e. how sophisticated is the geometry optimizer (use of internal coordinates? empirical force constants / exact second derivatives?)

While the first two issues, which are purely a question of speed, are somewhat important in practical applications, it is actually the latter two issues that in many cases are the most important for a workflow. If you're studying challenging molecules, you may face cases of poor SCF convergence, and this is where a flexible algorithm makes all the difference. You shouldn't care if program A solves an easy molecule in 5 steps while program B takes 7 steps to solve it, if for a challenging case program A takes 3000 steps but program B only 40. But, these issues are highly system dependent, and depend heavily on the algorithm. Using a second-order algorithm (e.g. trust region) yields more robust convergence, but even though the calculation now may converge in few steps they are much more expensive than with a simple gradient descent method; this is why you should compare apples to apples and use the exact same algorithms in all programs, and study a large variety of systems to try to cover a large sample of both "easy" and "difficult" cases.

I would note last that speed is not everything. Also the ease of use of the program and its general availability are key questions in determining which tool to use. If program A is 3x faster than program B, but B is easier/safer to use, most people would opt for program B.

Programs have also become more modular than before; this may also affect your choice: if it's easy to modify one program to do exactly what you want, it becomes your tool of choice even if it's not as fast as its competitors.

Answer 2

Yes, we are working on the performance benchmark of many quantum chemistry program packages. If you're interested in, you can visit our Github repository: https://github.com/r2compchem/benchmark-qm.

Answer 3

I don't mean this answer to criticize your question in any way because it's actually a great question. My opinion, however, is that this is sort of the wrong question.

I think a much easier and more relevant test is not how fast some calculation is on, say, a single core, but how well the implementation scales across many nodes, each with many cores. This is because many people doing quantum chemistry have access to very large computing clusters (either through their university, national lab, company, the cloud, etc.). Despite this, many electronic structure packages do not scale well beyond even a few nodes. Sometimes this is because the method does not scale well, in which case the code cannot be blamed, and sometimes it is because the code was not written to scale well. Usually, this is because the code was originally written in like 1970.

I hesitate to be too specific because I have not used every electronic structure package to do large calculations. I have, however, done some very large calculations with NWChem and have found that the MP2 and CCSD(T) implementations scale linearly with the number of cores for a very long time. The triples part of CCSD(T) is actually known to scale linearly to the entire size of the Cori supercomputer at Nersc as implemented in NWChem.

My personal experience is that Gaussian does not scale particularly well with the number of nodes. I think most people use Gaussian for DFT though, which I have never done, so take this statement with a big grain of salt.

I have also used Molpro and out-of-the-box, it seems to be a very fast code. So, if you are only interested in single-core speed, I would guess that Molpro will fare very well. Their MCSCF implementation is famously good in my experience as well.

Also, Psi4 is an excellent, modern electronic structure package which seems to have been made with parallelism in mind, so I would think it will scale better than many packages.

This is why generally, for gas-phase ground state electronic structure I think Psi4 and NWChem are the way to go. They seem to be well-written and are free.

---

Generally, though, getting fair comparisons of the speed of two programs which implement the same method is very difficult. If you want to benchmark DFT, you need to use the same grid for each calculation, you need to run them on the exact same core of the same CPU. You need to make sure nothing is happening in the background of the computer you're running on that could interfere. You should run each calculation many times.

Something like HF is even harder to benchmark fairly because HF is an iterative method. So, the initial guess you use, as well as things that accelerate iterations such as DIIS, make a big difference in how long the calculation takes. Also, when comparing, you need to make sure the integral thresholds are identical since most electronic structure programs will throw out certain integrals which are guaranteed to be smaller than some value. Also, for a large calculation, you need to be careful to make sure that the integrals are stored in an identical manner since sometimes the integrals are stored in RAM and other times they are stored partially on disk and still other times they just aren't stored and get re-calculated.

For all of these reasons, performing a good-faith comparison of the speed of these packages is nearly impossible. Also, I would argue that the scaling matters much more than the zero-order speed.

Answer 4

Since I leave academia (where I used the popular gaussian package) but I want to continue to do some research in my free time, I spend the last months trying to chose the best software for quantum chemistry calculations. I think the first thing one need to know is how much the software is updated with new methods, new DFT functionals.... After a first selection based on this, one need to know what is the system that will be used for running the calculation (i.e. laptop, desktop pc, workstation, cluster) to evaluate also the parallelization of the software (see for example the scaling of nwchem in a cluster with thousands of nodes). I will do my work on a desktop pc, so it's obvious that I need a well written code to run the calculation faster. However, like others said, you can compare different packages only if you use the same parameters (grid size, convergence threshold....). In my research I also discover that most packages use external libraries (BLAS, LAPACK) for the hardest part of calculations such as matrix multiplications, integral evaluations... So I think the choice of the software may be done mostly on the basis of the frequency of the update. In the end, I think that most of the performance for a calculation with the same parameters could be attributed to : the optimization in the compilation phase (optimization flags); the choice of good libraries (ATLAS vs OPENBLAS vs MKL...; look for some benchmark and you will see how much they perform very very different); the system you are using (linux version, linux scheduler, used file system, optimized kernel)... I' m still studying about this topic and these are my actual findings. What I would like to do in the near future is try to optimize all these parameters to see how much one can gain in term of time.