What does machine learning learn about DFT?

Question

I'm a student and now studying quantum chemistry but also interested in machine learning (ML) and materials informatics (MI).

In order to understand an ML method for MI, I tried to use the smooth overlap of atomic positions (SOAP), which is a widely-used descriptor for molecules. Using the SOAP descriptor as the input, I implemented a simple neural network (NN) model in PyTorch (see Fig.1 (a)). After training the NN model with the energy, the accuracy was very good, and then I transferred the trained NN model to the HOMO-LUMO gap. I obtained the results as Fig.1 (b) and (c).

Fig.1: (a) The SOAP descriptor input-based NN model. (b) Learning curve for the atomization energy (eV) provided by the QM9 database, which is a famous benchmark database in MI. Test error of energy is about 2.0 eV. (c) Transfer learning for the HOMO-LUMO gap with the pre-trained model.

I found that the transfer result was very poor (the random and pre-trained accuracies (errors) are the same, about 1.2 eV as shown in Fig.1 (c))... For my study, I tried to use other ML models (e.g., graph neural network and neural network potential), but the results were also very poor or much worse...

Here, I have a question; does this mean that the above NN model just fitted to only the energy, did not learn physics at all? In addition, do most of the ML models that just fit only the energy make sense from the view point of DFT? In DFT, as far as I have studied, solving the Kohn-Sham equation yields the ground state energy and density by variational principle, and also provides the eigenvalues (i.e., orbital energies), so we can know various physical properties such as the HOMO and LUMO.

My impression is that ML models just predict "similar molecules have similar energies (see Fig.2)", and it is very easy if we have a large number of similar molecules and their energies in the database. However, even in this case, can we say that "ML can approximate the DFT calculation"? (In fact, most of the research papers in ML and MI claim so, and I'm honestly very confused…)

Fig.2: The energy of the right-side molecule must be close to about -86.7X eV (the actual energy is -86.75 eV), but this is very easy to predict because we already know the energies of the left-side molecules as the training data samples, which are very similar to the right-side molecule.

Postscript: actually, I'm now interested in the "foundation model" in AI/ML research (https://arxiv.org/pdf/2108.07258.pdf). I would like to know whether AI/ML can solve various tasks (e.g., not only predict the total energy but also provide the HOMO LUMO eigenvalues, density, and other physical properties) with one (pre-trained) model. I would like to ask about the possibility as another question.

Answer 1

Total energies and HOMO-LUMO gaps are very different quantities, and naturally necessitate very different neural network designs (including the choice of descriptors and architectures) in order to describe efficiently and accurately. For example:

1. The total energy is a size-extensive property, while the HOMO-LUMO gap is size-intensive. Two non-interacting copies of the same molecule have exactly twice the energy as the single molecule, and exactly the same HOMO-LUMO gap (not twice of it) as the single molecule. Many neural networks designed to predict total energies work by expressing the total energy as a sum of atomic energies, and predict the atomic energies; this guarantees size-extensivity by construction, but which also means that the same neural networks are completely unsuitable for predicting size-intensive properties.
2. The total energy is a relatively local property, but the HOMO-LUMO gap is very non-local. Imagine a large molecule where the HOMO is localized on one side while the LUMO is localized on the other side. A neural network cannot correctly estimate the HOMO-LUMO gap of the system by only looking at a portion of it, not even when it computes estimates for many small portions of the molecule and then pools the results. At some point of the neural network prediction, there has to be a subtraction of two quantities computed from two arbitrarily far parts of the molecule. This also rules out many common neural network architectures. By contrast, all through-space interactions involved in the total energy decay at least as $1/R$, and much faster than that if the system does not contain ions.

Besides, there are some issues that affect both total energy and HOMO-LUMO gap predictions, but are much more severe for the latter:

3. The HOMO-LUMO gap is a non-differentiable function of the molecular geometry. Consider a system whose ground state and the first excited state form a conical intersection (e.g. a first-order Jahn-Teller system), and we vary the geometry continuously from one side of the conical intersection, till we pass the intersection point. The HOMO-LUMO gap will decrease approximately linearly. Then at the intersection point, the HOMO and LUMO suddenly swap, and the HOMO-LUMO gap increases approximately linearly again. No neural network built from smooth activation functions can reproduce this behavior. While neural networks that use, e.g. ReLU activation functions do possess derivative discontinuities, it is a difficult task to ensure that the derivative discontinuities only occur at (or at least near) conical intersections.
4. The HOMO-LUMO gap sometimes depend on whether there are an even number or an odd number of a certain functional group. This is best exemplified by the Hückel 4n+2 rule: conjugated rings with 4n+2 electrons are aromatic and have relatively large HOMO-LUMO gaps, while those with 4n electrons are antiaromatic and have small HOMO-LUMO gaps. It is extremely challenging to learn this fact with a neural network except by hard-coding the rule and/or intentionally including a large number of different aromatic and antiaromatic systems in the training set.

These may explain why it is difficult to predict the HOMO-LUMO gap to high accuracy. I would personally recommend predicting the Fock matrix instead, and then obtain the HOMO-LUMO gap by diagonalizing the Fock matrix. The Fock matrix is a much more well-behaved object than the HOMO-LUMO gap: the matrix elements are mostly determined from nearby atoms, do not have strong derivative discontinuities, and do not have the even-odd alternations as the HOMO-LUMO gap has for conjugated rings. Moreover, I believe that there are well-established techniques for predicting the Fock matrix via machine learning.

Answer 2

It could just be that the features you are using are well suited for describing total energies in these sorts of systems, but not in describing the differences of eigenvalues. When people do ML to try to capture trends from DFT calculations, usually they do so with specific quantities in mind, and don’t expect to get everything from the DFT calculation correct. For example, neural network potentials can very accurately give you the total energy and forces for a range of systems, but they would likely be poor for giving you the charge density or eigenvalues unless you trained on those quantities in particular. When people say that the ML agrees with DFT, they aren’t necessarily saying that it does so for everything that DFT outputs, instead they usually have a more limited scope.

Second, the gap between occupied and unoccupied states is not a physical quantity that DFT should be expected to get accurate, even with the unknown exact exchange correlation functional. The fundamental gap and the optical gap are not ground state quantities, and the DFT orbitals and eigenvalues don’t rigorously correspond to molecular orbitals or molecular orbital energies. In practice, DFT can often give reasonable values for the band gap of solids, or things like molecular orbitals in molecules, but unlike with the total energy and the electron density, quantities like the wavefunctions and eigenvalues aren’t really meant to be compared with physical observables.

The reason why people often do this anyway, and why often the DFT results aren’t bad in many cases is that the Hamiltonian from DFT and the Hamiltonian you get from a many body perturbation theory (typically in solids) or wave function based method (like coupled cluster) ends up being somewhat similar. In solids, a method like GW produces eigenvalues that rigorously correspond to the band gap. It just so happens that in GW, the difference between the exchange correlation energy and the GW self-energy is small enough in some systems that the DFT results can be sort of interpreted as giving you a value for the band gap.