4
$\begingroup$

My question is regarding a medium-to-large DLPNO-UCCSD(T)-F12 calculation.

I have a ~250 electron system, an open-shell molecule inside a water droplet, and I'm trying to get the total energy, first using the default SCF and PNO settings:

!DLPNO-CCSD(T)-F12 cc-pVTZ cc-pVTZ/C cc-pVTZ-F12-CABS RIJCOSX NormalSCF NormalPNO PModel 
%maxcore 14000
%palnprocs 32
end
%base "dlpno"

It am running it on 2 nodes, to get close to the ~500 GB of ram I've seen in used for similar very similar clusters, except those were closed-shell ([arXiv][1], [arxiv][2])

I don't know if the calculation is just stuck and wasting compute resources, or if it is actually still working. I know these calculations can take a while but with NormalSCF/NormalPNO and cc-pVTZ my intuition is that it would have been done by now.

The last thing written on my output is:

3-index local transformation (OBS_F,OBS/AUX) for op=2   ... done (     16.8 sec)3-index local transformation (OBS_F,CABS/AUX)for op=2   ... done (     51.4 sec)------------------------UHF DLPNO F12 CORRECTION------------------------Doing SCF density matrices                              ... done (      0.1 sec)Calculating RI Coulomb operators Retrieving the Coulomb fitting basis                    ... done       .... OBS/OBS done        (      0.9 sec)

This is after --- The CCSD iterations have converged ---STARTING F12 INTEGRAL TRANSFORMATIONS... and then what you see in the block. The thing is, this part was reached yesterday morning so I'm really beginning to give up with this.

Do you have any advice?

Good news is: I have DLPNO-UMP2/def2-QZVPPD energies with TightSCF and TightPNO, but would like to go higher to DLPNO-CCCSD(T). There is no DLPNO-MP2-F12 implemented in ORCA for open-shells though.

As always, thank you very much for any suggestions!

[1]: N. Sylvetsky, J.M.L. Martin "Energetics of (H$_2$O)$_{20}$ Isomers by means of F12 Canonical and Localized Coupled Cluster Methods" (2020): https://arxiv.org/pdf/2002.01749.pdf

[1]: G. Santra, et al, "Is valence CCSD(T) enough for the binding of water clusters? The isomers of (H$_2$O)$_6$ and (H$_2$O)$_{20}$ as a case study" (2023): https://arxiv.org/abs/2308.06120

$\endgroup$
4
  • $\begingroup$ How long has the calculation taken so far and how long do you think it has been stuck after that particular print? How long did the prior MP2 calculation take and what computational resources were used (if they differ from the current calculation)? $\endgroup$
    – Tyberius
    Sep 24, 2023 at 18:59
  • $\begingroup$ The DLPNO-CCSD(T)-F12 calculation has been running 1.5 days now, and has been stuck on this step for about 1.3 days. Compared to the DLPNO-MP2 calculation, that s.p. took ~6 hours, no F12, and half the memory but 3 x as many cores. For the DLPNO-CCS(T) I opted for reducing cores, getting more memory over 2 nodes and that atleast didn't crash. $\endgroup$
    – epalos
    Sep 24, 2023 at 19:05
  • $\begingroup$ This post had two questions: "Is my calculation stuck?" and "would I be better off using a different method?". In order to comply with this, I removed the second question, which can be asked separately, with a link to this question, if you want! $\endgroup$ Sep 24, 2023 at 20:55
  • $\begingroup$ Side comment: it may be more beneficial to use the dedicated F12 basis sets (cc-pVTZ-F12, for example), which are more diffuse than cc-pVTZ and roughly comparable to aug-cc-pVTZ. The extra diffuse functions help the HF energy to reach the CBS limit, since the correlation cusp is already well taken care of by F12 and the HF error becomes important. $\endgroup$
    – wzkchem5
    Sep 26, 2023 at 7:35

1 Answer 1

6
$\begingroup$

"I don't know if the calculation is just stuck and wasting compute resources, or if it is actually still working."

When that happens, I log into the compute node and run the top command. On a SLURM system, I would run the command (if my username is epalos):

squeue -u epalos

This will list all of my jobs that are running, and on which compute nodes they are running. If the job corresponding to the calculation that I want to check, is running on the node called monster_023, then I would run the following command:

ssh monster_023

It could ask you for your password now (but might not, because the node to which you're trying to connect is part of the same system), and if you successfully enter the node monster_023, then you just need to run the command: top or top -u epalos.

top will tell you which processes are running on the node, and how much CPU load each process is consuming. If all of the processes that are being run by the user epalos are using a negligible amount of the node's CPU resources, then your job is "struck". If you see any process using 1% or more of the node's CPU resources, then it is doing something. Ideally the CPU usage for each process would be around 100% x N, in which N is the number of cores being used for that process, but it can be significantly less (for example 5%) sometimes (for example if the process is transferring data from the RAM to the disk, which is an extremely slow processes).

It is unlikely that your system does not let you log into compute nodes in the way that I have described above. If the you are not able to log into the compute nodes, then you can ask your system administrator to allow it, because most supercomputing centers do allow you to "monitor your jobs" like this (you can send them this answer along with your email if you think it would help).

If you cannot log into the compute nodes, another way to check if the program is "doing anything", then you can run the following command:

watch ls -l -t

to see if any of the files are changing in size, but this will not work if the program is not writing anything to any of the files for a long time (you also might need to change the number of digits printed in the file sizes, or to change your folder to the "temporary" or "scratch" directory in which the program stores the "junk" files associated with the calculation, in order for this method to work). Some programs can be really doing a lot of number crunching for hours before any new information is written to a file, so attempting to monitor file-size changes is not as effective as simply logging into the compute node and checking the CPU usage of your program's processes.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .