What are the physical reasons if the SCF doesn't converge?

Question

I've been mostly using ab initio methods as a routine calculation tool, so even though I know some things from the theoretical side, I still have mostly hands-on experience. One of the things that I know is if your molecular geometry does not make sense, then SCF crashes. One example would be if I use angstroms instead of bohrs in the geometry definition (who hasn't been there!). I assume that if my molecular geometry makes little chemical sense, SCF would also crash.

What would be some physical/numerical reasons for these failures?

Answer 1

I have written a section in the BDF user manual on this issue. It is in Chinese but I'll roughly translate the key parts to English as below.

Common reasons for the SCF procedure to fail to converge include:

1. A too small HOMO-LUMO gap, which causes repetitive changes of the frontier orbital occupation numbers. Imagine two orbitals $\psi_1$ and $\psi_2$, where the former is occupied and the latter is unoccupied at the $N$-th SCF iteration. If the orbital energies of $\psi_1$ and $\psi_2$ do not differ much, then it may well be that at the $N+1$-th SCF iteration, the orbital energy of $\psi_1$ becomes higher than that of $\psi_2$, resulting in the electron(s) on $\psi_1$ transferring to the orbital $\psi_2$. However, this will inevitably cause a large change in the density matrix, and therefore the Fock matrix. Thus chances are that if one diagonalizes the new Fock matrix, one finds that $\psi_1$ is again lower in energy than $\psi_2$. In this unfortunate scenario, the orbital occupation number oscillates between two patterns, which prevents convergence. Typical signatures of this scenario include an oscillating SCF energy (with an amplitude of $10^{-4} \sim 1$ Hartree), and a clearly wrong occupation pattern printed at the end of the SCF calculation.
2. The HOMO-LUMO gap is relatively small but not excessively small, so that the occupation numbers of the orbitals do not change, but the orbital shape oscillates (this is what the physicists call "charge sloshing"). (The polarizability of a system is inversely proportional to the HOMO-LUMO gap, and if the polarizability is high, a small error in the Kohn-Sham potential may result in a large distortion of the electronic density. When the HOMO-LUMO gap shrinks beyond a certain point, the distorted density may give a Kohn-Sham potential that is even more erroneous than the original one, which marks the onset of divergence.) Typical signatures include an oscillating SCF energy with a slightly smaller magnitude, and a qualitatively correct occupation pattern.
3. Numerical noise caused by a too small grid or a too loose integral cutoff threshold. Typical signatures include an oscillating SCF energy with a very small magnitude ($<10^{-4}$ Hartree), and a qualitatively correct occupation pattern.
4. The basis set (orbital basis set, auxiliary basis sets, etc.) is close to linearly dependent, or the grid is so small that the projection of the basis set on the grid is close to linearly dependent (this problem does not exist for plane wave basis sets). Typical signatures include a wildly oscillating or unrealistically low SCF energy (error > 1 Hartree), and a qualitatively wrong occupation pattern.

Note that a wrong geometry or a wrong charge merely increase the likelihood that problems 1, 2 and 4 arise, and are themselves not separate reasons for SCF non-convergence. In particular, too long bonds make problems 1 and 2 more likely, while too short bonds tend to invoke problem 4. Many people tend to ignore or underestimate the importance of problems 3 and 4, such that they play around hopelessly with damp and level shift (or in the physicists' language, different mixing schemes), which actually only work for problems 1 and 2, when the true cause of the non-convergence is actually numerical noise or basis set linear dependence. (For the sake of this particular question, however, one may argue that only problems 1 and 2 are "physical reasons", since problems 3 and 4 are purely numerical artifacts and are not due to any physical property of the system being studied.)

Answer 2

There are several good answers. One concern that's not currently mentioned in the other answers is when the initial guess is poor.

While most programs have several methods to perform an initial guess, and these are usually fairly good, they can also fail.

As a simple example, let's consider the superposition of atomic potentials: J. Chem. Theory Comput. 2019, 15, 3, 1593–1604. It works well and it's conceptually easy to understand both why it should work and cases when it can fail.

If I want to converge this calculation of benzene, I'd start with the atom potentials from six carbon atoms and six hydrogen atoms:

Not surprisingly, this is a fairly good initial guess and the SCF can converge.

This one is harder to converge. I stretched all the bonds by $3\times$:

In this case, how do I treat the atoms? Are they individual non-interacting atoms? No, the bonds are stretched a lot, but there's still some shared electron density.

I can probably still converge this "stretched benzene," but it's harder because the initial guess is often tailored for covalently bonded carbon.

Problems with poor initial guess can happen more for unusual charge or spin states, or with metal centers. For the latter, consider that small differences in geometry could lead to different spin states, etc.

(Some programs I will not use for inorganic or organometallic calculations because the initial guess are routinely poor -- although most programs continue to improve.)

Answer 3

There are countless reasons why the SCF might be difficult to converge. Some physically interpretable scenarios include a system that is not appropriately charge-balanced, closely overlapping atoms, or charge sloshing. As noted here, "charge sloshing generally refers to the long-wavelength oscillations of the output charge density due to some small changes in the input density during the iterations, and results in a slow convergence or even divergence."

Answer 4

Let me add one more: (incorrectly high) symmetry, which can lead to zero HOMO-LUMO gap.

One scenario is when one poses too high symmetry to a system, compared to the real symmetry of the electronic system. Another common scenario is that the method cannot describe properly the electronic structure of the systems, e.g. DFT calculations on low-spin Fe(II) in the octahedral field, therefore using high symmetry, while chemically correct, will lead to convergence problems.

Neither of these is just SCF convergence error, but the error and the small HOMO-LUMO gap can be an indicator that something is wrong.