# When do we abandon ab initio methods?

This question is related to (and was originally asked in) another post about "quantum protectorates" I made here.

Ab initio methods are nice because they directly solve a sort of "theory of everything" for many-body systems, and they are systematically improvable. However, having been part of a physics department I rarely ever saw them used at all before moving to chemistry.

At what point do we abandon ab initio methods? I have typically thought the main reason other methods are used is because of computational constraints. I have also heard that ab initio methods tend to fail in strongly correlated systems (as does DFT), and just about everyone I know who works in this area relies on renormalisation methods. Thijssen's Computational Physics textbook also briefly discusses ab initio vs semi-empirical methods,

A distinction can be made between ab initio methods, which use no experimental input, and semi-empirical methods, which do. The latter should not be considered as mere fitting procedures: by fitting a few numbers to a few experimental data, many new results may be predicted, such as full band structures. Moreover, the power of ab initio methods should not be exaggerated: there are always approximations inherent to them, in particular the reliance on the Born–Oppenheimer approximation separating the electronic and nuclear motion, and often the local density approximation for exchange and correlation.

So when do we know when and when not to use ab initio methods? The list of potential reasons I have come up with are:

• Ab initio methods tend to be much slower.
• Ab initio methods are better for small systems (i.e. they don't scale nicely), although I have heard that very large systems have been solved with them, and there do exist periodic methods, so I do not know how true this is.
• There are extra approximations in any ab initio calculation in order to make the many-body Schrödinger equation tractable. For many applications, these approximations (especially the Born-Oppenheimer approximation) are unacceptable.
• Ab initio methods do not capture the physics of strongly correlated systems properly for some reason.
• Model Hamiltonians are simpler and tend to be "baked" for the problem at hand, and therefore results are easier to interpret, especially for a scientist who is used to working with it.
• Physicists are not so familiar with them (compared to chemists, say), so the current paradigm of condensed matter physics does not involve ab initio methods.
• Maybe there is some deep reason about "quantum protectorates" and how the many-body Schrödinger equation is not sufficient to describe some systems, and a "higher" theory is needed (this seems quite philosophical and hard to prove, though).
• Well, ab-initio is great research, parameter fitting is great engineering. – B. Kelly Feb 6 at 5:42

There's three scenarios that come to my mind, for when ab initio methods get abandoned:

• The cost becomes prohibitive (e.g. too many electrons)
• The insight is lost
• It is simply not required for what we want to do

Prohibitive cost: If solving the Schrödinger equation (for example) is no longer possible, we may very well still wish we could solve the Schrödinger equation, but we are unfortunately forced to make simplifications. For example, you can switch to a Hückel model of the Hamiltonian, and the benefits and uses of Hückel theory were discussed in this mini-publication that Etienne Palos wrote for his answer to this question: Where is the extended Hückel method (EHM) still used today?

Insight is lost: A major disadvantage of ab initio methods, is akin to the disadvantage of solving a differential equation, or finding the eigenvalues of a matrix numerically rather than analytically. When you solve a differential equation or find matrix eigenvalues analytically, you can see explicitly the dependence of the answer on each parameter. For example if the lowest eigenvalue of a matrix is $$\lambda = \frac{5D^2}{\sqrt{\pi}}$$ for some diffusion constant $$D$$, then not only can you now calculate the answer for various other values of $$D$$ easily, you can also see right away that $$\lambda = \mathcal{O}(D^2)$$ which can be a very powerful thing to know. To find the eigenvalue numerically you will need to substitute a number for $$D$$ and then you'll just get a number such as $$\lambda=5.22195$$. To discover the $$D^2$$ dependence, you will have to re-do the calculation for many other values of $$D$$. But there is an advantage to numerically finding the eigenvalue, and that is that you can do this for arbitrary diagonalizable matrices of size $$1000\times 1000$$ (for example) extremely quickly (so perhaps re-consider your first statement, that ab initio methods are "slow") whereas it in general would be impossible to solve the problem analytically. The situation is very similar for model Hamiltonians vs ab initio treatments: when you solve for the ground state energy of a 50-electron Schrödinger equation, you just get a number, but if you do this for an exactly solvable Ising or Hubbard Hamiltonian, you might get a formula.

Not required: In this paper in the field of "quantum biology", I modeled quantum mechanically the FMO pigment-protein complex, which contains 24 fairly large molecules called chromophores, embedded within a protein which is dissolved in water. The system contains thousands of particles. Did I solve the Schrödinger equation for thousands of particles? No, because the purpose was to study the rate of energy transfer from the first chromophore to three specific "end" chromophores and it could easily be done in the following way: each of the 24 chromophores can either be in its ground state or electronic excited state, but there is exactly enough energy in the system for only one of the chromophores to be excited at any given point in time, so we can ignore ground states that have zero excitations and we can ignore the second, third, fourth, etc. excited states. We therefore have a single-excitation subspace of 24 possible states (one for each possible chromophore containing the single excitation). Each of these states has an energy relative to the lowest one: these are diagonals of my single-excitation model Hamiltonian. There's a tunelling amplitude between each pair of states: these are the off-diagonals of my single-excitation model Hamiltonian. The vibrations of the protein and water around the chromophores will certainly affect the dynamics of energy transfer, and these involve thousands of atoms, but we roughly know the "spectral distribution functions" $$J_m(\omega)$$ for each single-excitation state $$m$$, where $$J_m(\omega)$$ tells us how strongly state $$m$$ interacts with vibrations of frequency $$\omega$$. We then have a $$24 \times 24$$ matrix describing the electronic degrees of freedom and twenty-four $$J(\omega)$$ functions describing the effect of nuclear vibrations on these, and we use the numerically exact Feynman-Vernon equation to calculate how the excitation evolves. If we tried to solve the entire problem ab initio we could get much more detailed information about the system's wavefunction, but we really do not need those details in this case, and the difference we would get for the overall result would unlikely provide any further valuable information to the study.