Is PySCF able to print checkpoint files during a CISD calculation?

Question

The problem

I've described how to restart a CISD calculation in my answer to the question: Restarting an FCI calculation in PySCF, but this requires all iterations of the CISD calculation to finish, so that a civec object is returned, which is the CISD vector, which can be saved by adding np.save('civec.npy', civec) to the end of a PySCF input script. If a CISD calculation with a convergence tolerance of 1e-3 is completed, the resulting civec can be used as a starting point for a CISD calculation with a more stringent tolerance like 1e-5.

However, what if the calculation with a convergence tolerance of 1e-3 crashes before it completes?

Is it possible to print civec after each Davidson iteration, so that if the job crashes during the 5th iteration (for example) without converging to the specified tolerance of 1e-3, we can use the saved civec from the completed 4th iteration to restart the CISD calculation?

What I've found so far

Checkpointing before full convergence seems possible with the MCSCF code:

I looked through all open and closed issues and pull requests on the PySCF Git repository that were suggested when I searched "chk", "checkfile", "chkfile" and "restart", and clicked on several of them that had promising titles, and the only one I think worth mentioning is this one because it showed me that the /mcscf/ folder has a chkfile.py file, but such a file is not included in the /ci/ folder (also note that the keyword dump_chk_ci=True is no longer in the current release PySCF, as I mentioned in a comment on that Pull Request).

By running pyscf/examples/mcscf/13-load_chkfile.py, with:

mci.chkfile = tmpchk.name 

changed to:

mci.chkfile = `mci.chk` 

I do get the chkfile mci.chk printed, and by changing:

mc.max_cycle_macro = 1

to:

mc.max_cycle_macro = 10

I am able to confirm that the .chk file is printed, then more output is produced, then the .chk file is expanded, then more output is produced, etc., suggesting that the checkpoint is being updated after each macro-iteration.

I'm unable to do this with the CISD code:

The pyscf/ci folder doesn't have a chkfile.py, but I did notice that pyscf/ci/cisd.py contains the following:

    def dump_chk(self, ci=None, frozen=None, mo_coeff=None, mo_occ=None):
        if not self.chkfile:
            return self

        if ci is None: ci = self.ci
        if frozen is None: frozen = self.frozen
        # "None" cannot be serialized by the chkfile module
        if frozen is None:
            frozen = 0

        ci_chk = {'e_corr': self.e_corr,
                  'ci': ci,
                  'frozen': frozen}

        if mo_coeff is not None: ci_chk['mo_coeff'] = mo_coeff
        if mo_occ is not None: ci_chk['mo_occ'] = mo_occ
        if self._nmo is not None: ci_chk['_nmo'] = self._nmo
        if self._nocc is not None: ci_chk['_nocc'] = self._nocc

        lib.chkfile.save(self.chkfile, 'cisd', ci_chk)

This dump_chk code doesn't seem to be used though. Also, the following script produces all the correct output, except unlike the mcscf program it does not produce any .chk file:

import pyscf
from pyscf import gto, scf, ao2mo, fci,ci, lib

mol = pyscf.M(atom = 'Ne 0 0 0',basis = 'cc-pvDz',verbose=5,output='out_1e-5_direct.txt')
mhf = scf.RHF(mol).run()
mci = ci.CISD(mhf).set(conv_tol=1e-5,nroots=3)
mci.chkfile = 'cisd.chk'
e, civec = mci.kernel()

Example files don't seem to involve checkpointing for CISD:

I have looked through various files in PySCF's example suite, based on file names in the CI folder (specifically 00-simple_cisd.py, 01-density-matrix, 20-from-fci.py and 32-wfn_overlap.py), and the FCI folder (11-large_ci.py). Unfortunately chk and check and dump don't appear in any of those files! In the MCSCF folder there's the above-mentioned 13-load_chkfile.py and also 13-restart.py, and 14-project_init_guess.py), the former two giving examples of printing check point files for MCSCF, but not for CISD.

Other files associated with checkpointing don't show a clear way to do it for CISD:

The file pyscf/pyscf/lib/chkfile.py has an example in the comments that contains lib.chkfile.save('symm.chk', 'symm', ci), but there's nothing like mci.kernel in that example, for actually running a CISD calculation. The file pyscf/tools/chkfile_util doesn't seem to have anything related to checkpointing CISD.

The question

Is there a native way to dump the CI vector after each Davidson iteration, when doing a CISD calculation in PySCF?

Answer 1

As far as I know, there isn't a native way to do this, e.g. an option that can be passed to CISD. The dump_chk option is never passed to cisd.py, so it clearly was not meant as an option to checkpoint the Davidson iterations.

However, this can be easily implemented by adding a line which saves the most recent input for the Davidson iterator. In file pyscf/lib/linalg_helper.py the iterations begin on line 416. If the following is added on line 417:

numpy.save(f"/path/to/checkpoint/civec_i{icyc}.npy", x0)

before the next line (if fresh_start:) then it will save a checkpoint (labelled with the iteration number) to the specified path.

I should point out that I am working on a merge request for pyscf which will contain (among other things) an option to save checkpoints after each iteration.