What major advances in theoretical and computational chemistry have been made in recent decades?

Question

Computational modeling has become a mainstay in chemistry, materials science and physics. However, the methods that are typically employed are already decades old: for instance, the B3LYP and PBE functionals that dominate simulation in molecular and solid-state contexts date back to 1994 and 1996, respectively; some 30 years ago!

To convince the general public on the importance of continued investment in methods development, I want to ask what major breakthroughs have been made in theoretical and computational chemistry in the last 30 years. What can we do now that we couldn't do before? What kinds of methods are scientists unaware of, yet which could significantly affect the way chemistry research is carried out in academia and the private sector?

Answer 1

The GW miracle

Intro: What sets GW developments apart from others?

This is a great question! The top answer to "What are some recent developments in density functional theory?" mentioned double hybrid functionals, but these and the other answers don't constitute methods that were nearly as prolific as the introduction of hybrid functinals (1993, Becke) or TD-DFT (1984, Runge & Gross) or CPMD (1985, Car & Parinello). Double hybrids can significantly improve accuracy while significantly increasing computational expense, but also having largely the same shortfalls as previously existing DFT methods (the major one being the lack of a way to systematically improve functionals). Similarly, other recent methods that come to mind, such as arbitrary-order coupled cluster (2001, Kállay & Surján) and newer approaches to FCI starting with FCIQMC (2009, Both, Thom & Alavi) offer previously unimaginable accuracy but at an expense that is too great for the type of popularity that your examples of B3LYP and PBE have enjoyed. Also, it feels hard to think of something entirely "new", as I have so-far mentioned DFT, CC, and CI, which were all well-known since far before the 1990s, until I consider DMRG which is very different from all of those approaches, but again it's too computationally expensive to reach the level of popularity of B3LYP and PBE.

Using explicitly correlated Gaussians on systems with more than 2 electons, or abandoning Gaussians altogether with multiresolution appraoches, still fall into the category of improving accuracy for small systems but failing to help with treating the size of systems that B3LYP and PBE can. The invention of better Hamiltonians such as the X2C, SFDC or eQED Hamiltonians, and post-HF implementations of DBOC, were great achievements but don't have the type of impact that B3LYP and PBE did, because their importance is limited to a smaller subset of systems, and "decent" alternatives did previously exist (DK1 from 1974, or HF level DBOC from 1986).

The GW method offers superior accuracy to B3LYP and PBE, but with a lower cost than double-hybrids, coupled cluster, DMRG, and FCIQMC or other approaches to FCI; and the improvement in accuracy offered by GW methods exceed in magnitude or applicability, what we get from X2C vs DK1 or other mentioned method developments from the last 30 years.

G₀W₀ was introduced in 1965, so what's different now?

The original G₀W₀ method appeared in 1965 here, and it was used for electron gases in the 1960s, on periodic solids through the 1970s and 1980s, and on atoms in 1993 but it's application to non-periodic molecules first appeared around 2001 and has started to become very popular since the introduction of the GW100 benchmark set (along wtih its recent implementation in ABINIT, BerkeleyGW, FHI-AIMS, GPAW, MolGW, PySCF, Quantum ESPRESSO, TURBOMOLE, VASP, YAMBO, and many other software packages).

Why is GW becoming so popular for chemistry?

Nearly 5 years ago, it was already possible to do GW calculations on the GW5000 dataset (5239 molecules, some of them with more than 100 nuclei, which is beyond what can comforatably be done even with coupled-cluster, even in packages like ORCA that specialize in doing CC for systems wtih dozens of nuclei). GW is very computationally facile compared to other attempts to go beyond B3LYP or PBE in accuracy. Furthermore, GW methods were traditionally benchmarked against the CCSD(T) "gold standard" for the GW100 benchmark set, but in the latest comparisons (see here and papers that cite it), for many molecules it has become hard to tell whether the CCSD(T) or GW calculations are more accurate, as the properties calculated with GW methods seem to match experimental data close to as well as CCSD(T) does (and in many cases the GW calculations seem to match experiments even better than the "benchmark" CCSD(T) calculations).

Also, new GW infrastructure seems to be constantly developed these days, for example a relativistic GW implementation recently appeared on arXiv in 2024.

What about systematic improvement?

One of the main criticisms of DFT, which I mentioned before, is that it's not very systematically improvable. GW methods fall in the category of "perturbation theories", which can be systematically improved with higher perturbation orders. In the paper in which we introduced the term "GW miracle", we complemented GW in five different ways: with second-order exchange (SOX), with second-order screened exchange (SOSEX), with interacting electron-hole pairs, and with a GW density matrix. The "GW miracle" is that although we immediately know how to improve GW if we desire (unlike DFT), comparison to such complements showed that the "basic" GW method is already in a sweet-spot in terms of the balance between accuracy and speed. Also, the paper showed that GW calculations can be improved simply by improving the functional used for the non-interacting Green's function, meaning that whatever improvements in DFT that the future brings us, will likely also further improve GW calculations.

Answer 2

One big step forward is training machine learning algorithms on the output of DFT calculations, given appropriately rich feature vectors that are insensitive to atom ordering (i.e. that work even across graph isomorphism, given the covalent bond graph of a molecule).

You can see this at work in this paper (I am co-first author -- the paper is out of date by now, there has been a lot of followup work by many labs, but it may be a useful starting point). We were able to speed up predictions of a range of quantum molecular properties by orders of magnitude relative to DFT, while maintaining high levels of accuracy. This allows you to search the molecular space for candidates much more quickly.

Answer 3

The Rise of Teraflop scale Supercomputing (One Trillion operations per second) in 1997 with advent of Intel' ASCI Red spearheaded high speed developments of engineered physics models. As companies like NVIDIA installed their own chips into Graphics engines and physics engines for more realistic gaming, the same technology was used by Pharmaceutical industry to build molecular models and determined how they would behave with proteins/cells interactions before it was ever manufactured; but in relatively short timeframes often months per prototype cycle. However this still required renting computer time at a Cost of 57 Million USD per Teraflop. By 2010 Teraflop scale computing was possible with Off the shelf hardware. By 2013 Sony' PLaystation 4 was peak performance of 1.8 Teraflops/second at a cost .30 USD worth of Electricity. Drug companies could build molecular models in mere weeks. What once took decades.

Is now doable in weeks/months