I would like to know how to choose the best GPU on the basis of its FP32 or FP64 efficiency, depending on different types of simulations.
In brief I would like to know what types of simulations work in single precision, versus the types that require double-precision.
I wrote a software to "exact quantum dynamics" calculations, meaning that given a density matrix of a system under the influence of noise at one point in time, the program gives you the quantum mechanical time-evolution of the density matrix with no semi-classical, Markovian, weak-coupling, or other physical approximations, as long as you give the program the temperature and the Hamiltonian describing the system and the noise from its environment. The program accomplishes this by calculating the appropriate double Feynman integral numerically, and more details can be found in this answer at Quantum Computing Stack Exchange.
As long as the GPU has enough RAM, the calculation is much faster on a GPU than a CPU, and this difference increases as you try to simulate the system dynamics for more and more picoseconds (all figures below come from my 2013 paper about the software, which is called "FeynDyn" since is uses the Feynman integral to calculate dynamics):
You can see in the figure below that the speed-up gained by using the GPU also increased with the amount of RAM that the calculation required (the RAM required can increase depending on the size of the system, for example) up to about a 20x speed-up, but since the figure below was done prior to May 2012 I couldn't go beyond 4GB on the GPU! I can only imagine what this would look like now in 2022:
The same paper showed that single-precision and double-precision gave almost the same result for every version of the problem except the ones in which the influence of the noise was very strong (rapid loss of quantum coherence in the system of interest):
Therefore, if you want to calculate open quantum system dynamics by numerically calculating the appropriate Feynman integral, the last diagram here shows that single-precision is almost always enough.
When simulating chemicals or materials with quantum mechanical methods, it is common to require at least FP64 precision in order to get reliable results. One reason for this is the requirement that electronic states (or any fermionic states) are strictly orthonormal, which usually involves the calculation and inversion of matrices which can be ill-conditioned. Similarly, the diagonalisation of a Hamiltonian can also be an ill-conditioned problem, for example when states are nearly degenerate.
The term "ill-conditioned" refers to problems where small errors in the input to an operation can give large errors in the output. For example, suppose that we wish to invert the matrix $\mathrm{M}$, where
$$ \mathrm{M}=\left(\begin{array}{cc} 1 & 2\\ \frac{9}{8} & 2 \end{array}\right) $$ First, let's write the matrix in decimal form to four significant figures, so that our matrix is $$ \mathrm{M}_\mathrm{4sf}=\left(\begin{array}{cc} 1.000 & 2.000\\ 1.125 & 2.000 \end{array}\right). $$ In this case, our decimal representation is exact. If we also assume that our inversion operation is perfect, i.e. that it works internally in infinite precision, we will obtain the inverse as $$ \mathrm{M}_\mathrm{4sf}^{-1}=\left(\begin{array}{cc} -8.000 & 8.000\\ 4.500 & -4.000 \end{array}\right), $$ which is easily shown to be the correct answer.
Now let us try the same calculation with only two significant figures of accuracy for $\mathrm{M}$. We have $$ \mathrm{M}_\mathrm{2sf}=\left(\begin{array}{cc} 1.0 & 2.0\\ 1.1 & 2.0 \end{array}\right), $$ which is an exact representation of $\mathrm{M}$ except for $\mathrm{M}_{21}$, which has an error of 0.025 (approximately 2.3%). Giving this matrix to our infinitely-precise inversion algorithm, we obtain $$ \mathrm{M}_\mathrm{2sf}^{-1}=\left(\begin{array}{cc} -10 & 10\\ 5.5 & 5.0 \end{array}\right). $$ Comparing this to $\mathrm{M}_\mathrm{4sf}^{-1}$ we see that our $\sim 2\%$ error in a single element of $\mathrm{M}$ has led to errors of over 20% in every element of $\mathrm{M}_\mathrm{2sf}^{-1}$.
One well-known example in materials modelling is the problem of "charge-sloshing" instabilities in self-consistent field (SCF) methods. In a simple SCF method for a density functional theory calculation, we might start with a guess for the electronic charge density $\rho(r)$, and then:
This simple method fails (diverges) for all but the smallest materials simulations, because it is ill-conditioned. The ill-conditioning arises primarily from the Hartree contribution to the potential, $V_\mathrm{H}[\rho]$. In reciprocal-space, this has the form $$ V_\mathrm{H}(G) = \frac{\rho(G)}{\vert G \vert^2} $$ where $G\neq 0$ is a reciprocal lattice vector, and we've used Hartree atomic units.
During the calculation, the density has some error $\delta\rho$, and so the calculated Hartree potential also has an error $$ \delta V_\mathrm{H}(G) = \frac{\delta\rho(G)}{\vert G \vert^2}. $$ The trouble is, as we increase the simulation size in real-space, we decrease the size of the smallest reciprocal vectors, and so there are some wavevectors for which $|G|^2$ is very small indeed. Any error in the density for these wavevectors will be amplified enormously in the Hartree potential. Thus, even a small error in the density can lead to a large error in the potential.
Since the potential is used to construct the Kohn-Sham equations in the next iteration, the new Kohn-Sham states will also have a large error - and these states are used to construct the new density. In this way, a small error in the density in one iteration leads to a large error in the density for the next iteration - not a good idea for a stable algorithm!
It is common for molecular dynamics programs have the option of using mixed precision approaches on GPUs. This means that the many individual interactions might be calculated in single precision, and then they get added up in (for example) double precision. The result is that these give great performance on less expensive (single-precision) GPU hardware, while still having almost as much accuracy as full double precision.
An older set of benchmarks for Amber provides a metric of "dollars per nanoseconds per day" for a few GPUs (numbers now outdated). If you're shopping, this is the metric you want -- spend the least money to get the most performance. Not surprisingly, gaming-class single precision GPUs are a much better deal for this kind of simulation than full double precision GPUs. Note that more recent information on running Amber on GPUs is available, but they no longer provide numbers in terms of dollar cost.
I focus on Amber here because they have good documentation of these issues; the same principles would apply to other programs that use mixed precision techniques. A few relevant papers on mixed precision calculations:
I have been using the LAMMPS classical molecular dynamics package and several GPU computational codes written by myself for my research activities. While working, I have seen that most calculations (over 95% of the time) have the same result in all precision settings single, mixed and double.
However, in certain systems (mostly inhomogeneous ones), there have been variations between single and double precision results. This has happened to me independent of the pair-potentials being used, and so I would not recommend predicting whether the calculations you plan to do are best off when done with a specific precision setting (since a GPU can last for at least 5-6 years and the calculations being done can change during this time). Because of this, I usually test my calculations in both mixed and double precision modes to ensure the accuracy of the results.
That said, you should consider the requirement of other features such as ECC and memory capacity as well. For example, compare this RTX3080 with the RTX A5000. They are both based on the GA102 die, but the FP64 performance and memory capacity are significantly different (not to mention the price). However, I would expect both to perform similar when using single or mixed precision modes.
Also I should mention that CompuBench gives a pretty good overlook of how each GPU performs based on the kind of calculation being done. However I do not know about the accuracy of the results, but the overall scaling seems reasonable.
Also if you are thinking about the reliability of the consumer GPUs, I have been using an RTX2070 Super ROG Strix for my calculations for well over two years. The card has been used at almost 100% capacity 24x7 in a particularly warm and dusty environment and has experienced multiple sudden shut downs due to power failure (the only time when the GPU stays unused). So far, I have not seen a single problem with the card and I can say for certain that it performs much better than a Quadro RTX A4000 which is twice as expensive.
