Which method gives the most accurate electron density, and how can it be verified experimentally?

Question

I read a paper from Science (DOI https://doi.org/10.1126/science.aah5975), “Density functional theory is straying from the path toward the exact functional”. They were trying to make a point that accurate energy prediction does not always imply accurate electron density. In this study, they used the CCSD as a reference method for electron density predictions.

But CCSD’s total energy prediction is much less accurate than DFT’s (DOI https://doi.org/10.3390/molecules25153485), so how can we be sure that CCSD’s electron density would be so accurate that it can be safely used as a reference?

Is there another way to address the accuracy of the calculated electron density?

Answer 1

The coupled-cluster hierarchy is a systematic approach to the exact many-body solution to the electronic Schrödinger equation, which yields size extensive energies and often converges extremely rapidly with respect to the maximum rank of excitations included in the model.

CCSD(T) is widely known as the "golden standard of quantum chemistry", since it has been shown to yield excellent agreement for e.g. atomization energies of small molecules, see Fig 2 in J. Chem. Phys. 112, 9229 (2000) for a powerful demonstration. I would like to note here that the reference energies the Chachiyo paper is using come from nothing but CCSD(T) calculations.

However, whenever you have a molecule that is not dominated by weak i.e. dynamical correlation effects, you need to also include higher-order excitations; see e.g. J. Chem. Phys. 149, 034102 (2018) for a recent benchmark study.

The accuracy of the CCSD energy and density can be verified by going up the ladder of CC theory, to CCSD(T), CCSDT, CCSDT(Q), CCSDTQ, CCSDTQ(5), CCSDTQ5, etc. However, every rung of the ladder means a significant increase in the computational cost. If your molecule is well-behaved, then the density should converge rather rapidly when you go up the ladder. It's pretty easy to find counterexamples, too, see e.g. J. Chem. Phys. 147, 154105 (2017); however, by the point where you have included all possible excitations, i.e. full coupled-cluster theory, then you have reached exactness i.e. full agreement with the full configuration interaction model.

One should note here that the density can be expected to converge less rapidly than the energy: if the wave function is variational, the error in the energy is second-order in the wave function, while the error in the density is only first-order in the wave function! Still, by the point when you get to full coupled-cluster, your density is exact.

It's also important to note that whenever one is discussing the post-HF level of theory like coupled-cluster, the one-electron basis set is hugely important. For instance, while coupled-cluster calculations yield worse energies than density functional approximation in small basis sets, J. Chem. Theory Comput. 11, 2036 (2015), the disagreements between coupled-cluster theory and experiment goes away when you are careful about what you are doing, see e.g. J. Chem. Theory Comput. 13, 1044 (2017) and J. Chem. Theory Comput. 13, 1057 (2017).

As a final point, agreement with experiment is not always simple: experiments often have several sources of error, which may not be obvious. In fact, there are several examples of cases where calculations have shown errors in the experiments, see e.g. Physics Today 61, 4, 58 (2008).