# How to overcome the exponential wall encountered in full configurational interaction methods?

Similar to how a molecular orbital, also known as a 1-electron wavefunction, can be represented with a linear combination of “basis” functions, e.g., atomic orbitals (LCAO): $$\Phi(\mathbf{r})=\sum_i^N c_i \phi(r).$$

A multi-electron wavefunction can be represented with a linear combination of multi-electron basis functions, also known as determinants, or electron configurations.

In the limit of all possible excited-determinants the solution is exact. The excited-electronic configurations describe both static electronic and dynamic electronic correlation.

However, the main challenge in solving this problem is that the number of determinants scales combinatorially. For even small molecules (e.g. >4 heavy atoms) and modest basis functions this problem is intractable. To make matters worse, solving for the coefficients $$c$$ requires diagonalizing the Hamiltonian, which scales as $$N^3$$.

What are some of the best approaches, generally speaking, for working around this problem (within the confines of configuration interaction approaches) while still capturing most of the static and dynamic electronic correlation?

• I recommend using displaystyle mathematics wherever possible, i.e. $$...$$ or \begin{align} ... \end{align}. – Martin - マーチン May 10 at 16:56
• Thanks Martin. I also saw your edits for references and will start to do that. – Cody Aldaz May 10 at 18:48

It is hard to claim that any FCI code overcomes the exponential wall, especially for strongly correlated systems. There are many algorithms, e.g. CDFCI, HCI, FCIQMC, ACI, etc., which significantly reduce the computational cost of direct FCI calculations and represent wavefunctions by sparse vectors. However, in my opinion, all of them only reduce the prefactor and none of them overcome the exponential wall.

DMRG is another method using a different ansatz. The scaling is indeed polynomial for weakly correlated systems (when it's possible to use small bandwidth). However, for strongly correlated systems, the scaling is difficult to say.

Another possible attempt is to rotate the molecular orbitals, like CASSCF, OptOrbFCI, etc. However, full accuracy of these methods is not expected if the orbital truncation is aggressive.

• Good answer, and in addition to that I will comment on the last paragraph: CASSCF essentially involves an FCI calculation on the active space, so there is the same "wall" to overcome in some sense. – Nike Dattani Apr 29 at 23:33
• Congrats on this paper btw, arxiv.org/pdf/2004.04205.pdf. I personally would not have been offended if you gave a bit more detail about OptOrbFCI. I've never heard of it and I think it deserves a little explanation. – Cody Aldaz May 7 at 2:05

Not all determinants contribute significant correlation, especially configurations for excitations of multiple electrons to higher energy orbitals, which is the basis for CAS methods; i.e. only choose the “important” orbitals in which to perform FCI. Doing so works acceptably for weakly correlated systems, but since FCI is still performed within the active space, as more orbitals are required to maintain an accurate description of electronic structure, the exponential growth once again becomes a problem. The only way to circumvent this is by excluding insignificant determinants, often for higher-level excitations, upon expansion of the active space. To my understanding, this approach is present to some extent in HCI, QMC, DMRG, and other algorithms, but each experiences their own problems with strongly correlated systems.

An emerging strategy that considers the entire orbital space with the means to systematically capture correlation without performing excessive excitations is the application of the Method of Increments to FCI (iFCI). This algorithm achieves polynomial scaling at low-order increments (up to n~4), where the FCI problem is broken up into portions to be solved individually. Of course, higher expansions are required for more strongly correlated systems, which again potentially encroaches on the problem of exponential scaling. However, utilizing the entire space to find the most strongly correlating orbital spaces without excessively including high-level excitations shows promise for approaching FCI-level accuracy without the intractable cost.

• Could you give a reference? And why does it scale polynomially? – Cody Aldaz May 6 at 16:39
• Just some initial comments: 1) "The only way to circumvent this is by excluding insignificant determinants" .. it might be possible to improve the phrasing here, since DMRG is not a determinant-based approach (it is different from the other methods listed in this sentence, namely HCI and FCIQMC). 2) There is a paragraph about HCI, QMC, DMRG, & other algorithms mitigating the cost of FCI but experiencing problems with strongly correlated systems, then a roughly equally long paragraph about the benefits iFCI, even though iFCI has its own problems, just like all the other methods. – Nike Dattani May 6 at 17:56