6
$\begingroup$

I am aware of the fact that Full Configuration Interaction scales poorly. Taking into account that the number of $n$-times excited Slater determinants of an $N$-particle System with $M$ basis functions is

$$ \begin{pmatrix} N \\ n \end{pmatrix} \begin{pmatrix} 2M-N\\ n \end{pmatrix} $$

and that you can exchange up to $N$ orbitals of your reference determinant makes clear that we won't go far with it. Sure there is Brillouines-Theorem and there is at most two particle operators in the electronic Hamiltonian, which makes the FCI Hamiltonian matrix sparse, but that doesn't change that this method is doomed to run into the "exponential-wall."

What I want to know is where this wall lies exactly. I am vaguely aware that some calculations of 10 electron systems have been made in the past with this method, but it must be possible to conclusively argue where the exponential wall lies and how it comes to be.

Does anyone have any information on this or can point me to a publication?

Cheers,

J.Erhard

$\endgroup$
3
  • 1
    $\begingroup$ Are you interested in exact FCI calculations, or are you also interested in approximate FCI calculations (e.g. FCI calculations whose energy error is much larger than machine precision but still within e.g. 1kcal/mol)? For the latter, the exponential wall comes significantly later than the former. $\endgroup$
    – wzkchem5
    Aug 8, 2022 at 16:08
  • $\begingroup$ I am interested in a very accurate density matrix. If I could argue that the DM is converged accuracy that would be useful to me for a different reason. Here I am more looking for something that enables me to gauge where to stop with the FCI calculations. $\endgroup$ Aug 8, 2022 at 16:17
  • $\begingroup$ Related: chemistry.stackexchange.com/q/75910/27342, mattermodeling.stackexchange.com/q/64/5 $\endgroup$ Aug 9, 2022 at 5:52

2 Answers 2

7
$\begingroup$

Assuming that you have $N$ electrons and $K$ spatial orbitals, the total number of electron configurations that can be built is $$ N_{\rm confs} = {K \choose N_\alpha} {K \choose N_\beta} \approx {K \choose N/2}^2 $$ since typically $N_\alpha \approx N_\beta$.

The exact solution is found by diagonalizing the configuration interaction (CI) Hamiltonian matrix, $H_{IJ} = \langle I | \hat{H} | J\rangle$. Although the matrix is typically huge (this is an understatement!), $N_{\rm confs} \times N_{\rm confs}$, it is also extremely sparse due to the Slater-Condon rules; the matrix elements between any two configurations that differ by more than a double excitation vanish. Moreover, the Hamiltonian is diagonally dominant, because the diagonal elements $H_{II}$ are single-configuration (i.e. Hartree-Fock) energies.

As a result, the lowest eigenstates of the full CI Hamiltonian matrix can be routinely determined with iterative methods; the famous Davidson method was designed for this purpose in the 1970s. Thanks to development of computers and algorithms, matrix sizes of a billion $\times$ a billion became feasible in the late 1980s, see Olsen et al.

However, due to the unforgiving exponential scaling of the number of configurations, the limits have not moved much since then. Tabulating the number of configurations obtained for $N$ electrons in $N$ orbitals, typically denoted as ($N$e,$N$o), one finds the following

\begin{array}{ll|ll} N & N_\text{confs} & N & N_\text{confs} \\ \hline \hline 1 & 1 & 11 & 2.1\times10^5 \\ 2 & 4 & 12 & 8.5\times10^5 \\ 3 & 9 & 13 & 2.9\times10^6 \\ 4 & 36 & 14 & 1.2\times10^7 \\ 5 & 100 & 15 & 4.1\times10^7 \\ 6 & 400 & 16 & 1.7\times10^8 \\ 7 & 1225 & 17 & 5.9\times10^8 \\ 8 & 4900 & 18 & 2.4\times10^9 \\ 9 & 15876 & 19 & 8.5\times10^9 \\ 10 & 63504 & 20 & 3.4\times10^{10} \end{array}

The (18e,18o) problem was thus feasible in the early 1990s. However, due to the unforgiving exponential scaling of the number of configurations, the limits have not moved that much since then. If you look at the literature, people often write (depending on the code and hardware at their use!) that full CI is limited in practice to (14e,14o) or (16e,16o) or (18e,18o).

The largest full CI calculations I know were reported in J. Chem. Phys. 147, 184111 (2017). The authors write in the abstract that (20e,20o) is routine with their new implementation and that they did a (22e,22o) calculation and a single CI iteration for the (24e,24o) problem; however, the text in Section V looks like only a single CI iteration was attempted for the (22e,22o) problem as well.

So, in summary, FCI is still limited in practice to somewhere around (16e,16o) - unless you want to waste a lot of computational resources to correlate a few more electrons.

The exponential scaling wall can, however, be pushed back a bit by being smart and leveraging the baked-in sparsity of the FCI problem with stochastic or selected CI methods developed by various authors. (You can see our recent paper here)

Another round is, of course, to reformulate the theory. We've done approximate (192e,192o) CASSCF for polyacenes with subtensor truncations of CCSDTQ56, see Mol. Phys. 116, 547 (2018). For the (50e,50o) pi-space calculation, we captured over 95% of the exact correlation energy. The full-valence calculation for 12acene just adds in sigma correlation, which is more localized and thus should be pretty much exactly captured by the used method.

$\endgroup$
6
  • $\begingroup$ Thank you very much for your answer. There is something I am unclear about though. You say you have N electrons and N orbitals. But if you have only N orbitals at your disposal, how do you generate configurations from your reference determinant? $\endgroup$ Aug 9, 2022 at 16:17
  • $\begingroup$ You maybe mean number of active orbitals? But it wouldn't be Full CI if it wouldn't equal the number of electrons, right? What I am missing here is what the basis set sizes were, i.e. the actual number of orbitals. $\endgroup$ Aug 9, 2022 at 18:11
  • $\begingroup$ Jannis, he means number of spatial orbitals, which is 2x the number of spin orbitals. $\endgroup$ Aug 9, 2022 at 22:56
  • $\begingroup$ So, it is implied that {$\phi_{\alpha}$} = {$\phi_{\beta}$}, i.e. every spatial orbital is occupied by 2 electrons ? $\endgroup$ Aug 10, 2022 at 14:13
  • $\begingroup$ @JannisErhard the N electrons in N orbitals splits into (Nalpha electrons in N orbitals) x (Nbeta electrons in N orbitals). You need to generate the individual spin strings, since the alpha electrons can distribute independently of the beta electrons. $\endgroup$ Aug 11, 2022 at 4:14
4
$\begingroup$

"it must be possible to conclusively argue where the exponential wall lies and how it comes to be."

The location of the "wall" depends on which "wall". With FCI you will hit a wall if you:

  • try to correlate too many electrons (even if you keep everything else the same), or
  • try to have too many orbitals in your basis set (even if you keep everything else the same).

For example, Susi's answer correctly shows that if you want to correlate 20 electrons, then involving 20 spatial orbitals would be quite difficult. However, with 4 electrons, I found it extremely easy to (for example) do an FCI calculation involving 377 spatial orbitals many years ago.

Perhaps a better "wall" to consider is the wall that you hit when you:

  • try to involve too many Slater determinants.

If you would like to do a calculation involving (22e,22o) and all ~34 billion Slater determinants with double precision (8 bytes per real number), then storing the amplitudes of the determinants alone would require about 8 x 34 billion = 272 GB and storing the bit-strings associated with these determinants would require approximately another (20 x 2 x 34 billion)/8 = 170 GB (20 spatial orbitals x 2 spin orbitals per spatial orbital x 34 billion bit-strings, divided by 8 bits for each byte). In total, storing this data alone would require 442 GB, and:

  • it would be too slow if this were to be stored on a hard drive, and
  • storing this data in the RAM across multiple nodes would significantly complicate things, to the extent that this is not typically done in the high-performance FCI codes available today,

so you would likely need a single node with at least 512GB to do this (20e,20o) calculation. This is not uncommon at the various supercomputing centres that have been discussed on this site, but nodes with this much RAM are still considered a precious commodity and even among the "elite" people who have access to such machines, the vast majority will rarely get an opportunity to run an FCI calculation on such a machine for long enough to get convergence. There is also a way to reduce the RAM required without ignoring any of the Slater determinants, and this is what would allow MRCC to do trillion-determinant FCI calculations with fewer than 8 trillion bytes of RAM.

But not all FCI Slater determinants are needed in order to get the FCI energy correctly to within nano-hartree precision (for example) and real-world applications rarely find more than the micro-hartree digit to be relevant.

The next and probably the most important "wall" that you can hit is the one which you reach when you want to involve too many "important" determinants for your specific application. I think the most I ever used in FCIQMC was 130 billion determinants for a (54e,54o) calculation on FeMoco but I never published it, largely because of the rise of improved methods which can estimate the FCI energy more accurately with smaller resources. The calculation with about 130 billion determinants gave an energy of about -13482.174 Hartree which is 2-3 mH higher than with FCIQMC-PT2 when using only 100 million determinants, and 4 mH higher than obtained with extrapolated SHCI-PT2 calculations, and about 8 mH higher than some of the lowest estimates I found overall.

To conclude, perhaps the largest published serious attempt at an FCI calculation was for the (54e,54o) system in my FeMoco paper, in which the energy we obtained is the correct FCI energy with an error bar of "only" a few kcal/mol (I say "only" because this is a calculation for a system which has about 4 x 10^30 determinants!), but anyone can do a calculation on a slightly larger (55e,55o) system and just not converge the energy to the accuracy that we did, so this is why I said it's perhaps the largest published "serious" attempt at an FCI calculation.

In a comment you mentioned that you're interested in density matrices. Density matrix calculations typically require twice as much RAM and twice as much CPU time compared to basic energy calculations that are calculated while involving nothing more than a wavefunction. Denisty matrices have indeed been calculated for that (54e,54o) system (for example in the above-mentioned FCIQMC-PT2 calculation).

$\endgroup$

You must log in to answer this question.