HF or DFT methods are variational so I can confidently say that any basis set that gives the lowest energy is the best for that system. However, I have learnt that post-HF methods such as MP2, MP3, MP4 and so on, or CCSD, CCSD(T) etc. are not variational, so the lowest energy does not necessarily means that its the best.
So how can I compare the accuracy and efficiency of different basis sets such as def2, cc for correlated methods?
First of all, MP2 (for example) is actually guaranteed to converge from above to the basis set limit even though MP2 in one specific basis set can give an energy lower than the FCI energy in that same basis set. So it's variational character in the basis set sense that matters here.
Furthermore, variational character is not the most pertinent thing to consider when comparing basis sets, since basis sets are not designed to give the lowest total energy. I asked a related question: Which basis set families were optimized to give the lowest possible variational energy for a given number of orbitals?, and still no one seems to know of any basis set families that were optimized to get the lowest variational energy.
When basis sets are designed, other properties are considered more important than lowest variational energy. For example:
If you want to compare the def2 sequence to the cc-pVXZ (Dunning) sequence, then you have to first choose the property in which you are interested: Are you calculating ionization energies? Electron affinities? Atomization energies? Bond lengths? Dipole polarizabilities? Then you can see how well the def2 and Dunning basis sets reproduce known-to-be-accurate benchmark data for the specific property you're trying to calculate, for similar molecules, and use this as a guideline for the molecule you're studying. Lowest total energy is almost never the property at the centre of a scientific project, but all the other properties often are, so it's not always wisest to judge the quality of a basis set (for what you're trying to accomplish) based only on the lowest total energy rather than on the quality of the above things?
There's much more to the trade than just this, and it's not easy to summarize in one answer, because which basis sets I would use will depend on so many other factors. For example in this paper I went all the way up to 8Z (aug-cc-pCV8Z), so I would never have even considered using the def2 series, which does not go beyond 4Z as far as I know. But in this paper we knew it would be hopeless to get anywhere. near the complete basis set limit at the spectroscopic standard, and it was a large molecule so speed was very important and we used a def2 basis set.
Bottom line: for almost every scientifically meaningful study, lowest energy does not mean better basis set, so don't compare basis sets based only on variational characteristics.
In general, when computing any property with different models (e.g. level of theory, basis set, etc), if you don't have some kind of theoretical bound (like the variational principle) to determine what is a better result, you need a reference value to compare against.
One choice for this reference is experimental results. At the end of the day, the goal of computations is to predict real properties, so it makes sense to use the experimentally measured values as a reference. When available, these are arguably the best reference you can have. One potential drawback is the somewhat rare possibility that there are significant errors in the experimental results. Another possible issue would stem from being "right for the wrong reasons". For example, an experimental measured optical rotation might be the result of several conformers of a molecule; if you performed an OR calculation for a single conformer, you may incidentally reproduce the experimental value while doing a poor job of actually simulating the chosen conformer.
Experimental data can often be sparse for systems from theoretical research. After all, if there were a plethora of experimental data in these areas, there wouldn't be as much of a need for simulation. Another common option for benchmarking is to use values from a sufficiently accurate model as you reference. "Sufficiently accurate" is going to be very dependent on the systems/properties you are looking at: a lot of small/medium sized chemical studies consider CCSD(T) with a large basis the "gold-standard" while materials research would probably neccesitate a smaller basis and/or DFT methods. The drawback of using another simulation as a reference is we don't necessarily know that a model with a higher level of theory is more accurate. However, post-HF methods are, at least in principle, systematically improvable, so can develop at least an approximate sense of how converged a property is with respect to the full CI/complete basis set limit.
Well, Nike already answered the point about the variationality: even though methods like MP2, CCSD, and CCSD(T) are non-variational in that they may over- or underestimate the energy of the ground state (or excited states) of the Schrödinger equation, the energy reproduced by any given method typically does behave variationally with respect to the basis set. You can understand this in another way: while CCSD(T) doesn't even have a wave function, you can write a Lagrangian for it, which the solution minimizes. The one-electron basis set just affects the accuracy with which you're minimizing this functional.
Note that this is not restricted to ab initio calculations. DFT is a celebrated class of methods, but the absolute energies reproduced by density functional approximations (DFAs) don't mean anything. DFT is most certainly non-variational with respect to the true solution to the Schrödinger equation; still, there are many atomic-orbital basis sets that are optimized for the minimization of DFT energies.
I would still like to point out that absolute energies are for sure important for doing accurate benchmarks: the whole point of extrapolation schemes is that the error with respect to the basis set is monotonic, and the total energies approach the exact value from above.
If you're studying a new property / a new level of theory, you should always check the convergence with respect to the basis set. If the changes from XZ to (X+1)Z are small, this typically means that you have reached convergence to the basis set limit; your absolute energy should then also be close to the correct value. This does not always happen though: even though you can reach sub-microhartree level accuracy in the total energy for self-consistent field calculations for light atoms in Gaussian basis sets, for heavy atoms the absolute energy is only accurate to millihartree [J. Chem. Phys. 152, 134108 (2020)].
PS. An interesting exception can be found in relativistic methods. The Dirac equation permits two classes of solutions: the electronic solutions with positive energies, and the positronic solutions with negative energies (a.k.a. Fermi sea). In a given AO basis set with K basis functions, you will have K positive-energy solutions, and K negative-energy solutions. The energy is not determined as a minimization of the energy functional with respect to orbital rotations as in non-relativistic theory, but rather as a mini-max procedure where you minimize with respect to electronic rotations and maximize with respect to positronic rotations. Because of this, the variational principle no longer applies. You can see this effect e.g. in calculations with the Douglas-Kroll-Hess (DKH) and exact two-component (X2C) approaches.
