When do we abandon ab initio methods?

Question

This question is related to (and was originally asked in) another post about "quantum protectorates" I made here.

Ab initio methods are nice because they directly solve a sort of "theory of everything" for many-body systems, and they are systematically improvable. However, having been part of a physics department I rarely ever saw them used at all before moving to chemistry.

At what point do we abandon ab initio methods? I have typically thought the main reason other methods are used is because of computational constraints. I have also heard that ab initio methods tend to fail in strongly correlated systems (as does DFT), and just about everyone I know who works in this area relies on renormalisation methods. Thijssen's Computational Physics textbook also briefly discusses ab initio vs semi-empirical methods,

A distinction can be made between ab initio methods, which use no experimental input, and semi-empirical methods, which do. The latter should not be considered as mere fitting procedures: by fitting a few numbers to a few experimental data, many new results may be predicted, such as full band structures. Moreover, the power of ab initio methods should not be exaggerated: there are always approximations inherent to them, in particular the reliance on the Born–Oppenheimer approximation separating the electronic and nuclear motion, and often the local density approximation for exchange and correlation.

So when do we know when and when not to use ab initio methods? The list of potential reasons I have come up with are:

• Ab initio methods tend to be much slower.
• Ab initio methods are better for small systems (i.e. they don't scale nicely), although I have heard that very large systems have been solved with them, and there do exist periodic methods, so I do not know how true this is.
• There are extra approximations in any ab initio calculation in order to make the many-body Schrödinger equation tractable. For many applications, these approximations (especially the Born-Oppenheimer approximation) are unacceptable.
• Ab initio methods do not capture the physics of strongly correlated systems properly for some reason.
• Model Hamiltonians are simpler and tend to be "baked" for the problem at hand, and therefore results are easier to interpret, especially for a scientist who is used to working with it.
• Physicists are not so familiar with them (compared to chemists, say), so the current paradigm of condensed matter physics does not involve ab initio methods.
• Maybe there is some deep reason about "quantum protectorates" and how the many-body Schrödinger equation is not sufficient to describe some systems, and a "higher" theory is needed (this seems quite philosophical and hard to prove, though).

Answer 1

There's three scenarios that come to my mind, for when ab initio methods get abandoned:

• The cost becomes prohibitive (e.g. too many electrons)
• The insight is lost
• It is simply not required for what we want to do

Prohibitive cost: If solving the Schrödinger equation (for example) is no longer possible, we may very well still wish we could solve the Schrödinger equation, but we are unfortunately forced to make simplifications. For example, you can switch to a Hückel model of the Hamiltonian, and the benefits and uses of Hückel theory were discussed in this mini-publication that Etienne Palos wrote for his answer to this question: Where is the extended Hückel method (EHM) still used today?

Insight is lost: A major disadvantage of ab initio methods, is akin to the disadvantage of solving a differential equation, or finding the eigenvalues of a matrix numerically rather than analytically. When you solve a differential equation or find matrix eigenvalues analytically, you can see explicitly the dependence of the answer on each parameter. For example if the lowest eigenvalue of a matrix is $\lambda = \frac{5D^2}{\sqrt{\pi}}$ for some diffusion constant $D$, then not only can you now calculate the answer for various other values of $D$ easily, you can also see right away that $\lambda = \mathcal{O}(D^2)$ which can be a very powerful thing to know. To find the eigenvalue numerically you will need to substitute a number for $D$ and then you'll just get a number such as $\lambda=5.22195$. To discover the $D^2$ dependence, you will have to re-do the calculation for many other values of $D$. But there is an advantage to numerically finding the eigenvalue, and that is that you can do this for arbitrary diagonalizable matrices of size $1000\times 1000$ (for example) extremely quickly (so perhaps re-consider your first statement, that ab initio methods are "slow") whereas it in general would be impossible to solve the problem analytically. The situation is very similar for model Hamiltonians vs ab initio treatments: when you solve for the ground state energy of a 50-electron Schrödinger equation, you just get a number, but if you do this for an exactly solvable Ising or Hubbard Hamiltonian, you might get a formula.

Not required: In this paper in the field of "quantum biology", I modeled quantum mechanically the FMO pigment-protein complex, which contains 24 fairly large molecules called chromophores, embedded within a protein which is dissolved in water. The system contains thousands of particles. Did I solve the Schrödinger equation for thousands of particles? No, because the purpose was to study the rate of energy transfer from the first chromophore to three specific "end" chromophores and it could easily be done in the following way: each of the 24 chromophores can either be in its ground state or electronic excited state, but there is exactly enough energy in the system for only one of the chromophores to be excited at any given point in time, so we can ignore ground states that have zero excitations and we can ignore the second, third, fourth, etc. excited states. We therefore have a single-excitation subspace of 24 possible states (one for each possible chromophore containing the single excitation). Each of these states has an energy relative to the lowest one: these are diagonals of my single-excitation model Hamiltonian. There's a tunelling amplitude between each pair of states: these are the off-diagonals of my single-excitation model Hamiltonian. The vibrations of the protein and water around the chromophores will certainly affect the dynamics of energy transfer, and these involve thousands of atoms, but we roughly know the "spectral distribution functions" $J_m(\omega)$ for each single-excitation state $m$, where $J_m(\omega)$ tells us how strongly state $m$ interacts with vibrations of frequency $\omega$. We then have a $24 \times 24$ matrix describing the electronic degrees of freedom and twenty-four $J(\omega)$ functions describing the effect of nuclear vibrations on these, and we use the numerically exact Feynman-Vernon equation to calculate how the excitation evolves. If we tried to solve the entire problem ab initio we could get much more detailed information about the system's wavefunction, but we really do not need those details in this case, and the difference we would get for the overall result would unlikely provide any further valuable information to the study.

I'd like now to try to address some of the comments in your question:

• I would try not to get to caught up about "differences" between "physicists" and "chemists" here. It is not true that "physicists" do not do ab initio calculations, for example Krzysztof Pachucki does some of the most high-precision ab initio atomic calculations in the world, and also some molecular ab initio calculations, but more people would call him a "physicist" than a "chemist" (his degrees are also all in physics and he's in a physics department). Likewise Gordon W. F. Drake, who was the "Editor in Chief" of Physical Review A for decades, also got all his degrees in physics, worked in physics departments for his entire career, and was elected an APS Fellow but probably was never even considered for an ACS Fellow designation, does ab initio calculations on atoms, as do a fair number of other atomic physicists.
• On another front, ab initio methods are getting popular as a more rigorous way to study high-Tc superconductive materials in a first-principles way, see for example:
  • "New superexchange paths due to breathing-enhanced hopping in corner-sharing cuprates" and,
  • "Towards an exact description of electronic wavefunctions in real solids".
• "There are extra approximations in any ab initio calculation in order to make the many-body Schrödinger equation tractable. For many applications, these approximations (especially the Born-Oppenheimer approximation) are unacceptable." That's not really true. We do non-Born-Oppenheimer ab initio calculations frequently: Are there examples of ab initio predictions on small molecules without the "major approximations"?. There's excellent review papers on the subject by Ludwik Adamowicz, for example:
  • "Non-Born–Oppenheimer calculations of atoms and molecules",
  • "Born–Oppenheimer and Non-Born–Oppenheimer, Atomic and Molecular Calculations with Explicitly Correlated Gaussians".
• "Ab initio methods do not capture the physics of strongly correlated systems properly for some reason." Who said so? The ab initio method called CASSCF is used to (accurately) treat strong correlation in thousands of studies, and it is such a popular method that it's implemented in several open source software packages: Is there a free package with robust CASSCF functionality?
• "Maybe there is some deep reason about "quantum protectorates" and how the many-body Schrödinger equation is not sufficient to describe some systems, and a "higher" theory is needed (this seems quite philosophical and hard to prove, though)." If that's the case, what makes you think your model Hamiltonian such as an Ising or Hubbard Hamiltonian, will work? Those model Hamiltonians are just approximations of the Hamiltonian that Schrödinger told us. ab initio methods try to solve the "true" the Schrödinger equation, whereas model Hamiltonians approximate it a bit first, then solve.
• Just because it's in a book doesn't mean it's true. That quote alone is enough to tell us that the book's author is biased against ab initio methods, and doesn't even quite know what they are. It is not correct that "there are always approximations inherent to them, in particular the reliance on the Born–Oppenheimer approximation" as I already showed above two review papers on non-Born-Oppenheimer ab initio calculations. The author then goes on to say "and often the local density approximation for exchange and correlation" as if the GGA and meta-GGA and other DFT functionals that are not LDA are not approximations? Also, density functionals are almost always fitted to experimental data so a very large number of people don't consider density functional calculations to be ab initio, see: Can DFT be considered an ab initio method? and Is density functional theory an ab initio method?.