Kill or let be: Is this calculation stuck or just very slow?

Question

My question is regarding a medium-to-large DLPNO-UCCSD(T)-F12 calculation.

I have a ~250 electron system, an open-shell molecule inside a water droplet, and I'm trying to get the total energy, first using the default SCF and PNO settings:

!DLPNO-CCSD(T)-F12 cc-pVTZ cc-pVTZ/C cc-pVTZ-F12-CABS RIJCOSX NormalSCF NormalPNO PModel 
%maxcore 14000
%palnprocs 32
end
%base "dlpno"

It am running it on 2 nodes, to get close to the ~500 GB of ram I've seen in used for similar very similar clusters, except those were closed-shell ([arXiv][1], [arxiv][2])

I don't know if the calculation is just stuck and wasting compute resources, or if it is actually still working. I know these calculations can take a while but with NormalSCF/NormalPNO and cc-pVTZ my intuition is that it would have been done by now.

The last thing written on my output is:

3-index local transformation (OBS_F,OBS/AUX) for op=2   ... done (     16.8 sec)3-index local transformation (OBS_F,CABS/AUX)for op=2   ... done (     51.4 sec)------------------------UHF DLPNO F12 CORRECTION------------------------Doing SCF density matrices                              ... done (      0.1 sec)Calculating RI Coulomb operators Retrieving the Coulomb fitting basis                    ... done       .... OBS/OBS done        (      0.9 sec)

This is after --- The CCSD iterations have converged ---STARTING F12 INTEGRAL TRANSFORMATIONS... and then what you see in the block. The thing is, this part was reached yesterday morning so I'm really beginning to give up with this.

Do you have any advice?

Good news is: I have DLPNO-UMP2/def2-QZVPPD energies with TightSCF and TightPNO, but would like to go higher to DLPNO-CCCSD(T). There is no DLPNO-MP2-F12 implemented in ORCA for open-shells though.

As always, thank you very much for any suggestions!

[1]: N. Sylvetsky, J.M.L. Martin "Energetics of (H$_2$O)$_{20}$ Isomers by means of F12 Canonical and Localized Coupled Cluster Methods" (2020): https://arxiv.org/pdf/2002.01749.pdf

[1]: G. Santra, et al, "Is valence CCSD(T) enough for the binding of water clusters? The isomers of (H$_2$O)$_6$ and (H$_2$O)$_{20}$ as a case study" (2023): https://arxiv.org/abs/2308.06120

Answer 1

"I don't know if the calculation is just stuck and wasting compute resources, or if it is actually still working."

When that happens, I log into the compute node and run the top command. On a SLURM system, I would run the command (if my username is epalos):

squeue -u epalos

This will list all of my jobs that are running, and on which compute nodes they are running. If the job corresponding to the calculation that I want to check, is running on the node called monster_023, then I would run the following command:

ssh monster_023

It could ask you for your password now (but might not, because the node to which you're trying to connect is part of the same system), and if you successfully enter the node monster_023, then you just need to run the command: top or top -u epalos.

top will tell you which processes are running on the node, and how much CPU load each process is consuming. If all of the processes that are being run by the user epalos are using a negligible amount of the node's CPU resources, then your job is "struck". If you see any process using 1% or more of the node's CPU resources, then it is doing something. Ideally the CPU usage for each process would be around 100% x N, in which N is the number of cores being used for that process, but it can be significantly less (for example 5%) sometimes (for example if the process is transferring data from the RAM to the disk, which is an extremely slow processes).

It is unlikely that your system does not let you log into compute nodes in the way that I have described above. If the you are not able to log into the compute nodes, then you can ask your system administrator to allow it, because most supercomputing centers do allow you to "monitor your jobs" like this (you can send them this answer along with your email if you think it would help).

If you cannot log into the compute nodes, another way to check if the program is "doing anything", then you can run the following command:

watch ls -l -t

to see if any of the files are changing in size, but this will not work if the program is not writing anything to any of the files for a long time (you also might need to change the number of digits printed in the file sizes, or to change your folder to the "temporary" or "scratch" directory in which the program stores the "junk" files associated with the calculation, in order for this method to work). Some programs can be really doing a lot of number crunching for hours before any new information is written to a file, so attempting to monitor file-size changes is not as effective as simply logging into the compute node and checking the CPU usage of your program's processes.