17
$\begingroup$

I would like to know how to choose the best GPU on the basis of its FP32 or FP64 efficiency, depending on different types of simulations.

In brief I would like to know what types of simulations work in single precision, versus the types that require double-precision.

$\endgroup$
16
  • 3
    $\begingroup$ +1 but are you asking for us to list every type of calculation and tell you whether it requires single or double precision? $\endgroup$ Mar 2 at 13:05
  • 3
    $\begingroup$ Every GPU will be faster for FP32 than for FP64. You're basically asking which calculations need (or would benefit from) F64? $\endgroup$ Mar 2 at 13:23
  • 8
    $\begingroup$ Ab initio simulations require FP64; classical forcefields can often manage with FP32; machine-learning can sometimes manage with FP16. Is that the sort of thing you wanted? Note that, if you ever want to do ab initio simulations on any CUDA-based GPU port, you will need a server-grade NVIDIA Tesla card, and the fastest is almost certainly an A100; in practice, this is also true for OpenACC-based ports as well. $\endgroup$ Mar 2 at 15:20
  • 3
    $\begingroup$ It might be better to split this question into separate questions for each type of simulation you’re interested in. I can answer for forcefield based MD (tl;dr: mixed precision codes give almost FP64 accuracy at almost FP32 speed, and allows great performance per dollar). OTOH, I don’t personally have the knowledge for ab initio. $\endgroup$
    – dwhswenson
    Mar 2 at 16:17
  • 2
    $\begingroup$ Algorithms can often be made to work in single-precision, but some things won't work. A common thing people need double precision for is if they're having to orthogonalize a basis, e.g. using the Lanczos approach for estimating eigenvalues. Sometimes people need higher precision for applications involving rational functions, such as Pade approximation. This is basically because 1/x can be very large if precision issues make x close to 0, instead of only being 1e-8. $\endgroup$
    – Slenderman
    Mar 2 at 22:06

4 Answers 4

16
$\begingroup$

Numerical Feynman integrals

I wrote a software to "exact quantum dynamics" calculations, meaning that given a density matrix of a system under the influence of noise at one point in time, the program gives you the quantum mechanical time-evolution of the density matrix with no semi-classical, Markovian, weak-coupling, or other physical approximations, as long as you give the program the temperature and the Hamiltonian describing the system and the noise from its environment. The program accomplishes this by calculating the appropriate double Feynman integral numerically, and more details can be found in this answer at Quantum Computing Stack Exchange.

As long as the GPU has enough RAM, the calculation is much faster on a GPU than a CPU, and this difference increases as you try to simulate the system dynamics for more and more picoseconds (all figures below come from my 2013 paper about the software, which is called "FeynDyn" since is uses the Feynman integral to calculate dynamics):

You can see in the figure below that the speed-up gained by using the GPU also increased with the amount of RAM that the calculation required (the RAM required can increase depending on the size of the system, for example) up to about a 20x speed-up, but since the figure below was done prior to May 2012 I couldn't go beyond 4GB on the GPU! I can only imagine what this would look like now in 2022:

enter image description here

The same paper showed that single-precision and double-precision gave almost the same result for every version of the problem except the ones in which the influence of the noise was very strong (rapid loss of quantum coherence in the system of interest):

Therefore, if you want to calculate open quantum system dynamics by numerically calculating the appropriate Feynman integral, the last diagram here shows that single-precision is almost always enough.

$\endgroup$
3
  • 1
    $\begingroup$ It would be interesting to see what can be the speedup now with the recent GPU with 80 GB of memory and improved computational power. $\endgroup$
    – NickZ
    Mar 3 at 5:57
  • $\begingroup$ Could you clarify if this is a single CPU or a single core? $\endgroup$
    – Ian Bush
    Mar 5 at 8:05
  • $\begingroup$ @IanBush it was multi-threaded. $\endgroup$ Mar 5 at 13:45
12
$\begingroup$

First principles simulations

When simulating chemicals or materials with quantum mechanical methods, it is common to require at least FP64 precision in order to get reliable results. One reason for this is the requirement that electronic states (or any fermionic states) are strictly orthonormal, which usually involves the calculation and inversion of matrices which can be ill-conditioned. Similarly, the diagonalisation of a Hamiltonian can also be an ill-conditioned problem, for example when states are nearly degenerate.

Ill-conditioning

The term "ill-conditioned" refers to problems where small errors in the input to an operation can give large errors in the output. For example, suppose that we wish to invert the matrix $\mathrm{M}$, where

$$ \mathrm{M}=\left(\begin{array}{cc} 1 & 2\\ \frac{9}{8} & 2 \end{array}\right) $$ First, let's write the matrix in decimal form to four significant figures, so that our matrix is $$ \mathrm{M}_\mathrm{4sf}=\left(\begin{array}{cc} 1.000 & 2.000\\ 1.125 & 2.000 \end{array}\right). $$ In this case, our decimal representation is exact. If we also assume that our inversion operation is perfect, i.e. that it works internally in infinite precision, we will obtain the inverse as $$ \mathrm{M}_\mathrm{4sf}^{-1}=\left(\begin{array}{cc} -8.000 & 8.000\\ 4.500 & -4.000 \end{array}\right), $$ which is easily shown to be the correct answer.

Now let us try the same calculation with only two significant figures of accuracy for $\mathrm{M}$. We have $$ \mathrm{M}_\mathrm{2sf}=\left(\begin{array}{cc} 1.0 & 2.0\\ 1.1 & 2.0 \end{array}\right), $$ which is an exact representation of $\mathrm{M}$ except for $\mathrm{M}_{21}$, which has an error of 0.025 (approximately 2.3%). Giving this matrix to our infinitely-precise inversion algorithm, we obtain $$ \mathrm{M}_\mathrm{2sf}^{-1}=\left(\begin{array}{cc} -10 & 10\\ 5.5 & 5.0 \end{array}\right). $$ Comparing this to $\mathrm{M}_\mathrm{4sf}^{-1}$ we see that our $\sim 2\%$ error in a single element of $\mathrm{M}$ has led to errors of over 20% in every element of $\mathrm{M}_\mathrm{2sf}^{-1}$.

Sloshing instabilities

One well-known example in materials modelling is the problem of "charge-sloshing" instabilities in self-consistent field (SCF) methods. In a simple SCF method for a density functional theory calculation, we might start with a guess for the electronic charge density $\rho(r)$, and then:

  1. Compute the potential $V[\rho]$
  2. Solve the Kohn-Sham equations with this $V[\rho]$
  3. Use the Kohn-Sham states to construct a new density $\rho_\mathrm{new}(r)$
  4. If $\vert\rho_\mathrm{new}-\rho|$ is small, stop; else set $\rho = \rho_\mathrm{new}$ and go to step 1.

This simple method fails (diverges) for all but the smallest materials simulations, because it is ill-conditioned. The ill-conditioning arises primarily from the Hartree contribution to the potential, $V_\mathrm{H}[\rho]$. In reciprocal-space, this has the form $$ V_\mathrm{H}(G) = \frac{\rho(G)}{\vert G \vert^2} $$ where $G\neq 0$ is a reciprocal lattice vector, and we've used Hartree atomic units.

During the calculation, the density has some error $\delta\rho$, and so the calculated Hartree potential also has an error $$ \delta V_\mathrm{H}(G) = \frac{\delta\rho(G)}{\vert G \vert^2}. $$ The trouble is, as we increase the simulation size in real-space, we decrease the size of the smallest reciprocal vectors, and so there are some wavevectors for which $|G|^2$ is very small indeed. Any error in the density for these wavevectors will be amplified enormously in the Hartree potential. Thus, even a small error in the density can lead to a large error in the potential.

Since the potential is used to construct the Kohn-Sham equations in the next iteration, the new Kohn-Sham states will also have a large error - and these states are used to construct the new density. In this way, a small error in the density in one iteration leads to a large error in the density for the next iteration - not a good idea for a stable algorithm!

$\endgroup$
5
  • $\begingroup$ Interesting facts. Is there any impact of error correction in memory (ECC) on the results? $\endgroup$
    – PBH
    Mar 4 at 0:51
  • 2
    $\begingroup$ Only if there are memory errors! I would not publish any final results from a machine without ECC memory, either a CPU or GPU. $\endgroup$ Mar 4 at 2:42
  • 1
    $\begingroup$ Another reason to use 64 bit, at least in all electron calculations, is that the differences between quantities used to evaluate whether a calculation has converged (e.g. Energy) can easily be in the eighth or latter significant figure, i.e. just zero in IEEE 32 bit. So in 32 bit you could never converge your calculation. Indeed, as Phil mentions, for big systems it's not clear that 64 bit is always enough. $\endgroup$
    – Ian Bush
    Mar 5 at 9:08
  • $\begingroup$ @IanBush I originally had that in my answer, but I thought it was too long! $\endgroup$ Mar 5 at 15:19
  • $\begingroup$ @PhilHasnip I've written some pretty long answers, feel free to make them as long as you want! This particular answer by you has been very well received and it looks like the level of detail in it has made it useful for a lot of people. $\endgroup$ Mar 5 at 15:40
11
$\begingroup$

Forcefield-based Molecular Dynamics (Molecular Mechanics)

It is common for molecular dynamics programs have the option of using mixed precision approaches on GPUs. This means that the many individual interactions might be calculated in single precision, and then they get added up in (for example) double precision. The result is that these give great performance on less expensive (single-precision) GPU hardware, while still having almost as much accuracy as full double precision.

An older set of benchmarks for Amber provides a metric of "dollars per nanoseconds per day" for a few GPUs (numbers now outdated). If you're shopping, this is the metric you want -- spend the least money to get the most performance. Not surprisingly, gaming-class single precision GPUs are a much better deal for this kind of simulation than full double precision GPUs. Note that more recent information on running Amber on GPUs is available, but they no longer provide numbers in terms of dollar cost.

I focus on Amber here because they have good documentation of these issues; the same principles would apply to other programs that use mixed precision techniques. A few relevant papers on mixed precision calculations:

$\endgroup$
2
  • $\begingroup$ Ross Walker from amber is an INVIDIA fellow right now, those folks have put alot of time into GPU research $\endgroup$
    – B. Kelly
    Mar 3 at 14:43
  • $\begingroup$ The GROMACS team has also published a paper benchmarking various CPU-GPU combinations in terms of ns-per-dollar, also comparing consumer-grade vs HPC GPUs. The actual models are long outdated, but the general lessons should still be relevant. $\endgroup$
    – TooTea
    Mar 4 at 11:59
11
$\begingroup$

Classical Molecular Dynamics

I have been using the LAMMPS classical molecular dynamics package and several GPU computational codes written by myself for my research activities. While working, I have seen that most calculations (over 95% of the time) have the same result in all precision settings single, mixed and double.

However, in certain systems (mostly inhomogeneous ones), there have been variations between single and double precision results. This has happened to me independent of the pair-potentials being used, and so I would not recommend predicting whether the calculations you plan to do are best off when done with a specific precision setting (since a GPU can last for at least 5-6 years and the calculations being done can change during this time). Because of this, I usually test my calculations in both mixed and double precision modes to ensure the accuracy of the results.

That said, you should consider the requirement of other features such as ECC and memory capacity as well. For example, compare this RTX3080 with the RTX A5000. They are both based on the GA102 die, but the FP64 performance and memory capacity are significantly different (not to mention the price). However, I would expect both to perform similar when using single or mixed precision modes.

Also I should mention that CompuBench gives a pretty good overlook of how each GPU performs based on the kind of calculation being done. However I do not know about the accuracy of the results, but the overall scaling seems reasonable.

Also if you are thinking about the reliability of the consumer GPUs, I have been using an RTX2070 Super ROG Strix for my calculations for well over two years. The card has been used at almost 100% capacity 24x7 in a particularly warm and dusty environment and has experienced multiple sudden shut downs due to power failure (the only time when the GPU stays unused). So far, I have not seen a single problem with the card and I can say for certain that it performs much better than a Quadro RTX A4000 which is twice as expensive.

$\endgroup$
7
  • $\begingroup$ BTW, sudden shutdown due to power outage is almost completely irrelevant to something like a GPU. The heat generation stops when the fans stop. It only matters for stuff like an SSD which has to keep persistent state across power cycles, but also wants to cache it in volatile DRAM for performance reasons while operating. (Including write buffers and the flash translation table.) Some lower-quality SSDs have had problems with that. Anyway, useful data point, +1. $\endgroup$ Mar 3 at 7:11
  • $\begingroup$ +1 but to follow the format of the other answers, I suppose the title should be "## Classical MD" ? $\endgroup$ Mar 3 at 7:30
  • $\begingroup$ @PeterCordes I just mentioned that as I didn't want to deliberately leave out any information I knew. But yes, there shouldn't be any problem for the GPU due to sudden shut downs. $\endgroup$
    – PBH
    Mar 3 at 7:50
  • $\begingroup$ @NikeDattani, thanks for the advice. I didn't know how the heading was placed. I have edited the answer now. $\endgroup$
    – PBH
    Mar 3 at 7:51
  • 1
    $\begingroup$ Sure, it's not a bad thing to mention things when you aren't sure if they're relevant. Some things that run hot do want to keep their fans spinning as part of a safe shutdown cycle, like for example an overhead project or room heater. The key point with those examples is that their hottest point is not their most temp-sensitive point, e.g. the remaining heat from an incandescent lamp (in an old pre-LED projector) could cook the rest of the insides if it lost power. GPUs can run hot, but the hottest and most temp-sensitive point is the same silicon. Except maybe electrolytic capacitors. $\endgroup$ Mar 3 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.