# What are the physical reasons if the SCF doesn't converge?

I've been mostly using ab initio methods as a routine calculation tool, so even though I know some things from the theoretical side, I still have mostly hands-on experience. One of the things that I know is if your molecular geometry does not make sense, then SCF crashes. One example would be if I use angstroms instead of bohrs in the geometry definition (who hasn't been there!). I assume that if my molecular geometry makes little chemical sense, SCF would also crash.

What would be some physical/numerical reasons for these failures?

• The choice of coordinates can make a difference. When I use psi4 for dimers, I have to make sure it uses mixed cartesian and redundant coordinates or about 40% of the dimers will fail to converge. I don't have this problem for single ideal gas calculations. On the same note, starting too far from optimal geometry (perhaps you used molecular mechanics to make the initial coordinate estimate, can lead you astray too). The choice of solver also can make a big difference, particularly for dimers. the geomeTRIC solver in psi4 works well. Feb 3, 2022 at 19:48
• @B.Kelly - that sounds like converging the geometry optimization? I think the poster is asking about converging the SCF in any particular geometry single-point. Feb 4, 2022 at 0:43
• @GeoffHutchison You are probably right, but it is just a comment. For a solely SCF calculation, I suppose one would have to resort to the numerical methods aspects, pointing out DIIS etc. On paper finding a minimum is easy, in practice, pretty tough. Other than improper geometry, I am not sure what physical reason would cause an SCF failure. Smarter people than me have told me it is possible to guarantee SCF convergence, but it (the algorithm) takes longer and isn't widely implemented. Feb 4, 2022 at 2:35

I have written a section in the BDF user manual on this issue. It is in Chinese but I'll roughly translate the key parts to English as below.

Common reasons for the SCF procedure to fail to converge include:

1. A too small HOMO-LUMO gap, which causes repetitive changes of the frontier orbital occupation numbers. Imagine two orbitals $$\psi_1$$ and $$\psi_2$$, where the former is occupied and the latter is unoccupied at the $$N$$-th SCF iteration. If the orbital energies of $$\psi_1$$ and $$\psi_2$$ do not differ much, then it may well be that at the $$N+1$$-th SCF iteration, the orbital energy of $$\psi_1$$ becomes higher than that of $$\psi_2$$, resulting in the electron(s) on $$\psi_1$$ transferring to the orbital $$\psi_2$$. However, this will inevitably cause a large change in the density matrix, and therefore the Fock matrix. Thus chances are that if one diagonalizes the new Fock matrix, one finds that $$\psi_1$$ is again lower in energy than $$\psi_2$$. In this unfortunate scenario, the orbital occupation number oscillates between two patterns, which prevents convergence. Typical signatures of this scenario include an oscillating SCF energy (with an amplitude of $$10^{-4} \sim 1$$ Hartree), and a clearly wrong occupation pattern printed at the end of the SCF calculation.
2. The HOMO-LUMO gap is relatively small but not excessively small, so that the occupation numbers of the orbitals do not change, but the orbital shape oscillates (this is what the physicists call "charge sloshing"). (The polarizability of a system is inversely proportional to the HOMO-LUMO gap, and if the polarizability is high, a small error in the Kohn-Sham potential may result in a large distortion of the electronic density. When the HOMO-LUMO gap shrinks beyond a certain point, the distorted density may give a Kohn-Sham potential that is even more erroneous than the original one, which marks the onset of divergence.) Typical signatures include an oscillating SCF energy with a slightly smaller magnitude, and a qualitatively correct occupation pattern.
3. Numerical noise caused by a too small grid or a too loose integral cutoff threshold. Typical signatures include an oscillating SCF energy with a very small magnitude ($$<10^{-4}$$ Hartree), and a qualitatively correct occupation pattern.
4. The basis set (orbital basis set, auxiliary basis sets, etc.) is close to linearly dependent, or the grid is so small that the projection of the basis set on the grid is close to linearly dependent (this problem does not exist for plane wave basis sets). Typical signatures include a wildly oscillating or unrealistically low SCF energy (error > 1 Hartree), and a qualitatively wrong occupation pattern.

Note that a wrong geometry or a wrong charge merely increase the likelihood that problems 1, 2 and 4 arise, and are themselves not separate reasons for SCF non-convergence. In particular, too long bonds make problems 1 and 2 more likely, while too short bonds tend to invoke problem 4. Many people tend to ignore or underestimate the importance of problems 3 and 4, such that they play around hopelessly with damp and level shift (or in the physicists' language, different mixing schemes), which actually only work for problems 1 and 2, when the true cause of the non-convergence is actually numerical noise or basis set linear dependence. (For the sake of this particular question, however, one may argue that only problems 1 and 2 are "physical reasons", since problems 3 and 4 are purely numerical artifacts and are not due to any physical property of the system being studied.)

• Thank you, this answer seems very thorough! Just as a follow-up: are you aware of a possibility of detecting a small HOMO-LUMO gap before doing an SCF calculation?
– user4626
Feb 4, 2022 at 16:29
• @LaBelleCroissant In most cases it is possible to have a very rough estimate of HOMO-LUMO gap by looking at the structure and resorting to experience. Otherwise I think the most straightforward method that does not require any chemical intuition is to perform a semiempirical calculation and read out the HOMO-LUMO gap from it. As semiempirical calculations are very cheap, they take basically no time, and you can add huge level shifts (e.g. 1.0 au) to ensure convergence. Feb 4, 2022 at 18:37

There are several good answers. One concern that's not currently mentioned in the other answers is when the initial guess is poor.

While most programs have several methods to perform an initial guess, and these are usually fairly good, they can also fail.

As a simple example, let's consider the superposition of atomic potentials: J. Chem. Theory Comput. 2019, 15, 3, 1593–1604. It works well and it's conceptually easy to understand both why it should work and cases when it can fail.

If I want to converge this calculation of benzene, I'd start with the atom potentials from six carbon atoms and six hydrogen atoms: Not surprisingly, this is a fairly good initial guess and the SCF can converge.

This one is harder to converge. I stretched all the bonds by $$3\times$$: In this case, how do I treat the atoms? Are they individual non-interacting atoms? No, the bonds are stretched a lot, but there's still some shared electron density.

I can probably still converge this "stretched benzene," but it's harder because the initial guess is often tailored for covalently bonded carbon.

Problems with poor initial guess can happen more for unusual charge or spin states, or with metal centers. For the latter, consider that small differences in geometry could lead to different spin states, etc.

(Some programs I will not use for inorganic or organometallic calculations because the initial guess are routinely poor -- although most programs continue to improve.)

• It's hard to say whether the convergence difficulty in this example is primarily due to a poor guess or a too small HOMO-LUMO gap. I believe that atomic density/potential-based initial guesses are not much worse for stretched benzene than for normal benzene, because the density of the former is just as well approximated as a superposition of atomic densities as the latter is. In contrast, the reduction of HOMO-LUMO gap upon stretching all the bonds of benzene is enormous. Feb 8, 2022 at 10:14

There are countless reasons why the SCF might be difficult to converge. Some physically interpretable scenarios include a system that is not appropriately charge-balanced, closely overlapping atoms, or charge sloshing. As noted here, "charge sloshing generally refers to the long-wavelength oscillations of the output charge density due to some small changes in the input density during the iterations, and results in a slow convergence or even divergence."

Let me add one more: (incorrectly high) symmetry, which can lead to zero HOMO-LUMO gap.

One scenario is when one poses too high symmetry to a system, compared to the real symmetry of the electronic system. Another common scenario is that the method cannot describe properly the electronic structure of the systems, e.g. DFT calculations on low-spin Fe(II) in the octahedral field, therefore using high symmetry, while chemically correct, will lead to convergence problems.

Neither of these is just SCF convergence error, but the error and the small HOMO-LUMO gap can be an indicator that something is wrong.