14
$\begingroup$

I've found a page dedicated to quantum chemistry packages benchmarks, on GitHub, qmspeedtest. But most results there are several years old, and so probably outdated. Is there some place where we can find comparisons like these, but updated often, or at least more recently?

I specified quantum chemistry in the question because I'm more interested in molecular systems, modeled with atom-centered gaussian function basis sets, for example. I have almost no familiarity with software that deals with periodic systems, plane-wave based. But I think it could be a good idea if someone with more familiarity with periodic systems opened a similar question for the respective packages.

$\endgroup$
  • 2
    $\begingroup$ There is CCCBDB: cccbdb.nist.gov, but the numbers are old and members of the public can't contribute to it. I made a GitHub database called "AI Energies" where AI means ab initio but the goal is that eventually there would be enough to do AI machine learning: github.com/HPQC-LABS/AI_ENERGIES. It was only me working on it so it's very limited, but any calculation you do, can be pushed into this repository easily, and every other quantum chemist in the world can do the same, so it has potential to grow. Is there a specific molecule youre interested in comparing to a benchmark? $\endgroup$ – Nike Dattani Jun 19 at 23:16
  • $\begingroup$ Thank you, @NikeDattani. No specific molecule. $\endgroup$ – ksousa Jun 19 at 23:22
  • 1
    $\begingroup$ So what exactly do you need or want? Okay now I see that the example you gave was only showing speed tests, not energy comparisons. This means CCCBDB and AI_ENERGIES are both not what you're looking for. The example you gave is the only one I've ever seen for comparing speeds. Actually I recorded runtimes and RAM usage for almost 100 coupled cluster calculations here and over 50 FCI calculations in a similar file (in the same repo). Calculations were added as recently as 3 months ago! But let's see if someone knows a better answer. $\endgroup$ – Nike Dattani Jun 20 at 0:35
  • $\begingroup$ I edited the question title to clarify the speed aspect. $\endgroup$ – ksousa Jun 20 at 0:44
  • 1
    $\begingroup$ I noticed. Let's see if there's anything more recent or more widely used than the GitHub repo in your question. I'm not aware of anything. It's something that I think would be very valuable. My databases (linked in the previous comment) that contain CPU times and RAM usage, only contain results from my own calculations, so they are limited. If no similar database exists though, I'd be keen to polish the style of mine and encourage others to add to it. $\endgroup$ – Nike Dattani Jun 20 at 0:49
16
$\begingroup$

The problem is that this is a highly volatile question. In order to meaningfully benchmark programs, you have to use the exact same compiler flags (may require heavy hacking) and use the same algorithms and parameters (accuracy, cutoffs, quadrature grids, etc). But, if a program supports many kinds of algorithms, then each of them would have to be benchmarked. In contrast, qmspeedtest is comparing apples to oranges; it is making no effort to actually ensure that the core algorithms and parameters are the same. It is for good reason that some programs explicitly ban publishing benchmark comparisons.

If you still intend to proceed, a good benchmark should look at these two core questions first:

  • speed of a single Fock build i.e. how quickly do you get a single-point energy from a given density
  • speed of gradient evaluation i.e. how quickly do you evaluate forces from a converged wave function

These are well-posed problems which are reproducible and where there is a single meaningful answer. This also means that the energy and Fock matrix / the nuclear gradient you get out from the benchmarks should agree numerically exactly between different codes. (You still do have several choices in the way to evaluate the final solution, e.g. density fitting, Cholesky decomposition, fast multipoles, etc, which may give different answers!)

Now, running a full calculation also depends on these issues:

  • cycles taken until SCF convergence i.e. how good is the default SCF guess and the default convergence accelerator for the system you're looking at
  • steps taken until geometry optimization converges i.e. how sophisticated is the geometry optimizer (use of internal coordinates? empirical force constants / exact second derivatives?)

While the first two issues, which are purely a question of speed, are somewhat important in practical applications, it is actually the latter two issues that in many cases are the most important for a workflow. If you're studying challenging molecules, you may face cases of poor SCF convergence, and this is where a flexible algorithm makes all the difference. You shouldn't care if program A solves an easy molecule in 5 steps while program B takes 7 steps to solve it, if for a challenging case program A takes 3000 steps but program B only 40. But, these issues are highly system dependent, and depend heavily on the algorithm. Using a second-order algorithm (e.g. trust region) yields more robust convergence, but even though the calculation now may converge in few steps they are much more expensive than with a simple gradient descent method; this is why you should compare apples to apples and use the exact same algorithms in all programs, and study a large variety of systems to try to cover a large sample of both "easy" and "difficult" cases.

I would note last that speed is not everything. Also the ease of use of the program and its general availability are key questions in determining which tool to use. If program A is 3x faster than program B, but B is easier/safer to use, most people would opt for program B.

Programs have also become more modular than before; this may also affect your choice: if it's easy to modify one program to do exactly what you want, it becomes your tool of choice even if it's not as fast as its competitors.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Beautiful and elegant $\endgroup$ – Thomas Jun 20 at 14:51
  • 1
    $\begingroup$ Thanks for a great answer. In terms of using the software for a research problem, you probably also want to see how well the program scales with process count and/or threads (and/or GPGPUs etc). You might also care about memory usage, it doesn't matter how fast it will run in principle if you can't run it in practice! $\endgroup$ – Phil Hasnip Jun 29 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.