The problem is that this is a highly volatile question. In order to meaningfully benchmark programs, you have to use the exact same compiler flags (may require heavy hacking) and use the same algorithms and parameters (accuracy, cutoffs, quadrature grids, etc). But, if a program supports many kinds of algorithms, then each of them would have to be benchmarked. In contrast, qmspeedtest is comparing apples to oranges; it is making no effort to actually ensure that the core algorithms and parameters are the same. It is for good reason that some programs explicitly ban publishing benchmark comparisons.
If you still intend to proceed, a good benchmark should look at these two core questions first:
- speed of a single Fock build i.e. how quickly do you get a single-point energy from a given density
- speed of gradient evaluation i.e. how quickly do you evaluate forces from a converged wave function
These are well-posed problems which are reproducible and where there is a single meaningful answer. This also means that the energy and Fock matrix / the nuclear gradient you get out from the benchmarks should agree numerically exactly between different codes. (You still do have several choices in the way to evaluate the final solution, e.g. density fitting, Cholesky decomposition, fast multipoles, etc, which may give different answers!)
Now, running a full calculation also depends on these issues:
- cycles taken until SCF convergence i.e. how good is the default SCF guess and the default convergence accelerator for the system you're looking at
- steps taken until geometry optimization converges i.e. how sophisticated is the geometry optimizer (use of internal coordinates? empirical force constants / exact second derivatives?)
While the first two issues, which are purely a question of speed, are somewhat important in practical applications, it is actually the latter two issues that in many cases are the most important for a workflow. If you're studying challenging molecules, you may face cases of poor SCF convergence, and this is where a flexible algorithm makes all the difference. You shouldn't care if program A solves an easy molecule in 5 steps while program B takes 7 steps to solve it, if for a challenging case program A takes 3000 steps but program B only 40. But, these issues are highly system dependent, and depend heavily on the algorithm. Using a second-order algorithm (e.g. trust region) yields more robust convergence, but even though the calculation now may converge in few steps they are much more expensive than with a simple gradient descent method; this is why you should compare apples to apples and use the exact same algorithms in all programs, and study a large variety of systems to try to cover a large sample of both "easy" and "difficult" cases.
I would note last that speed is not everything. Also the ease of use of the program and its general availability are key questions in determining which tool to use. If program A is 3x faster than program B, but B is easier/safer to use, most people would opt for program B.
Programs have also become more modular than before; this may also affect your choice: if it's easy to modify one program to do exactly what you want, it becomes your tool of choice even if it's not as fast as its competitors.