How to overcome the exponential wall encountered in full configurational interaction methods?

Question

Similar to how a molecular orbital, also known as a 1-electron wavefunction, can be represented with a linear combination of “basis” functions, e.g., atomic orbitals (LCAO): $$\Phi(\mathbf{r})=\sum_i^N c_i \phi(r).$$

A multi-electron wavefunction can be represented with a linear combination of multi-electron basis functions, also known as determinants, or electron configurations.

In the limit of all possible excited-determinants the solution is exact. The excited-electronic configurations describe both static electronic and dynamic electronic correlation.

However, the main challenge in solving this problem is that the number of determinants scales combinatorially. For even small molecules (e.g. >4 heavy atoms) and modest basis functions this problem is intractable. To make matters worse, solving for the coefficients $c$ requires diagonalizing the Hamiltonian, which scales as $N^3$.

What are some of the best approaches, generally speaking, for working around this problem (within the confines of configuration interaction approaches) while still capturing most of the static and dynamic electronic correlation?

Answer 1

It is hard to claim that any FCI code overcomes the exponential wall, especially for strongly correlated systems. There are many algorithms, e.g. CDFCI, HCI, FCIQMC, ACI, etc., which significantly reduce the computational cost of direct FCI calculations and represent wavefunctions by sparse vectors. However, in my opinion, all of them only reduce the prefactor and none of them overcome the exponential wall.

DMRG is another method using a different ansatz. The scaling is indeed polynomial for weakly correlated systems (when it's possible to use small bandwidth). However, for strongly correlated systems, the scaling is difficult to say.

Another possible attempt is to rotate the molecular orbitals, like CASSCF, OptOrbFCI, etc. However, full accuracy of these methods is not expected if the orbital truncation is aggressive.

Answer 2

Not all determinants contribute significant correlation, especially configurations for excitations of multiple electrons to higher energy orbitals, which is the basis for CAS methods; i.e. only choose the “important” orbitals in which to perform FCI. Doing so works acceptably for weakly correlated systems, but since FCI is still performed within the active space, as more orbitals are required to maintain an accurate description of electronic structure, the exponential growth once again becomes a problem. The only way to circumvent this is by excluding insignificant determinants, often for higher-level excitations, upon expansion of the active space. To my understanding, this approach is present to some extent in HCI, QMC, DMRG, and other algorithms, but each experiences their own problems with strongly correlated systems.

An emerging strategy that considers the entire orbital space with the means to systematically capture correlation without performing excessive excitations is the application of the Method of Increments to FCI (iFCI). This algorithm achieves polynomial scaling at low-order increments (up to n~4), where the FCI problem is broken up into portions to be solved individually. Of course, higher expansions are required for more strongly correlated systems, which again potentially encroaches on the problem of exponential scaling. However, utilizing the entire space to find the most strongly correlating orbital spaces without excessively including high-level excitations shows promise for approaching FCI-level accuracy without the intractable cost.

Reference: https://pubs.acs.org/doi/pdf/10.1021/acs.jpca.0c07624