8
$\begingroup$

There are many benchmark sets in different branches of computational chemistry and physics, that are frequently used to assess the "average error" of a method for a certain property, on a certain set of systems. For example, the GMTKN benchmark sets of the Grimme group, the benchmark set compiled by the Head-Gordon group, as well as numerous other benchmark sets, are used for the benchmarking of reaction energies and barriers of molecular reactions, non-covalent interaction energies of molecules, and relative conformational energies etc. Other benchmark sets may deal with excitation energies, geometries, etc., just to name a few, and some may be more focused on periodic systems and surfaces. Usually, one computes the data entries of a benchmark set using several methods (with different functionals, basis sets, etc.), compares them against the reference values, and calculates the mean absolute error and/or RMS error of the methods. The results are then used to assess the relative accuracy of the methods, and can also be used as an estimate of the typical error magnitudes of the methods.

Now, observe that what one really obtains from the benchmarking process, is the average error of a method when the user randomly picks a system from the benchmark set and calculates that system. However, in the real world the user is instead choosing a system based on their research project, and calculating that system. This raises a question: the error estimate from the benchmark study can only make sense, if the benchmark set is a statistically representative sample from all the systems that will be calculated after the benchmark paper is published. But none of the benchmark sets that I'm aware of seem to have taken this into account. Specifically, consider two systems A and B in a benchmark set. I've never seen any benchmark set that assigns appropriate weights to these two systems based on how likely a user will compute a system that is similar to A, compared to computing a system that is similar to B. Most benchmark sets do try to include a sufficiently diverse set of molecules, but even if a benchmark set has exhausted all important classes of systems, this still does not guarantee that it is a statistically representative sample from all the systems that will be studied in the future, unless they are explicitly weighted to account for this. A few benchmark sets (like GMTKN) do assign weights to the different entries of the benchmark set, but the weights only depend on the difficulty of the corresponding calculations; data entries that are hard to compute accurately often receive a smaller weight so that they do not unnecessarily dominate the error function. In particular, a commonly studied type of molecule is not weighted more than a rarely studied, but equally difficult type of molecule.

So my question is: are there any benchmark sets, regardless of what property or what kind of systems they focus on, that explicitly take into account the number of researchers in different subfields, and use this to weight its entries? For example, if a benchmark set contains the SARS-CoV and SARS-CoV-2 spike proteins, and there are 10x more publications on the latter than the former, then the latter receives a weight factor that is 10 times that of the former, even though the two molecules are very similar in terms of prediction difficulty. Only by this way can we point to the benchmark result and say, e.g. "this is the expected error of a random computational prediction of the binding Gibbs free energy of a spike protein with an acceptor/antibody". Of course, what we really want is the expected error of a future research result, not a research result randomly drawn from published data, so an even more rigorous approach is to include a time series prediction that predicts how many publications related to a certain class of molecules will come out in the next few years. But maybe this is too far of a stretch from what has actually been done in existing benchmark sets, so I'm already more than pleased to know if any existing benchmark set has its entries weighted by publication counts, or other metrics that can be used to measure the popularity of different classes of molecules (or materials in general) based on existing publications.

$\endgroup$
9
  • 3
    $\begingroup$ It's an interesting idea, but does anyone know how to find the probability distribution for how likely it is that a given system will be studied in the future? At best it seems you can make a guess for the near future, giving the benchmark a short shelf life. One from 3 yrs ago would be unlikely to include your SARS-CoV-2 example. A more likely approach would be to just use data from historical publications, which offers a retrospective view but doesn't seem like a good predictor for future research... You could probably publish something using machine learning "magic", but would it be useful? $\endgroup$
    – Anyon
    Jan 9 at 18:04
  • $\begingroup$ Indeed one may say that there is no good way of predicting what systems people will study in the future, even the near future (as illustrated by SARS-CoV-2). But the problem is that no one seems to use even a rough guess of this in developing a benchmark set, and some rough guesses should be quite reliable, e.g. that perovskites and graphene will continue to be buzzwords for some time. $\endgroup$
    – wzkchem5
    Jan 9 at 19:00
  • $\begingroup$ I think the question is interesting, but I suspect the answer at the moment is "no, there is no such benchmark set". Would you also accept answers suggested why this isn't common or might not be a useful metric? $\endgroup$
    – Tyberius
    Jan 9 at 21:30
  • 1
    $\begingroup$ @Tyberius I guess I'll wait for a few days, and if really no one comes up with such a benchmark set, I may accept one of the existing answers then $\endgroup$
    – wzkchem5
    Jan 10 at 20:31
  • $\begingroup$ @wzkchem5 sounds good. And no need to formally "accept" an answer with the greencheck mark if its not what you are looking for. I just meant if you would be opposed to seeing frame-challenge answers that argued against the premise of the question if there really is no such set. $\endgroup$
    – Tyberius
    Jan 10 at 20:51

2 Answers 2

9
$\begingroup$

I don't think any studies such as this are currently available, however I think they violate a general feeling I have about "benchmarking". Studies which do general benchmarks are designed more to highlight when a method is good, not which method is good.

If I benchmark pbesol vs pbe and I find that pbesol works well for solids for example, this is useful information that can be used to pre-screen methods. However, if I instead say that 80% of catalysis models do not work with pbesol, this does nobody any good because they do not know which group they are in.

These benchmarking sets are designed to intentionally find corner cases representative of real world research, if you expect a method to have issues with halogen bonding then you include some halogen bonding benchmarks in your dataset.

$\endgroup$
2
  • $\begingroup$ But the performance of PBEsol for solids is also the result of an average over the performance for many solids. How can you say that PBEsol is better for solids and PBE is worse, when you weigh the different solids arbitrarily while calculating the MADs of PBE and PBEsol for solids? If the MDs and MADs are obtained from an unweighted average or an arbitrarily weighted average, then one can only say that, e.g. PBEsol is good for the lattice constant of copper, and also quartz, etc., but never that PBEsol is generally good for solids, unless PBEsol is uniformly good for every solid. $\endgroup$
    – wzkchem5
    Jan 10 at 8:53
  • $\begingroup$ Thats somewhat the point actually, benchmarking is to identify "what" makes something work. Simply knowing things it works on is much less useful. Just because PBEsol is good for most solids for some use doesn't mean it will work for your system. $\endgroup$ Jan 10 at 17:20
6
$\begingroup$

I certainly see the value in having such a benchmark dataset.

If asked why such a dataset doesn't yet exist, I would say that it already takes an enormous amount of work to create things like the GMTKN dataset which you mentioned. Often this is an underappreciated amount of work, such that only the people that made the dataset really have an appreciation for how much work goes into it, and more often than not, people underestimate how much work goes into it. Also, often people simply use the dataset and cite it by name without citing the papers that lead to it, a bit like how people very often name the basis set they use without citing the paper in which its parameters were initially optimized and presented to the world.

What you are proposing, has some enormous advantages over the the databases that currently exist, but would take an extraordinarily larger amount of effort to complete compared to the already enormous work that goes into making datasets like the GMTKN. For example, how exactly are you going to find out how many papers pertain to O3, and how many pertain to CO2?

Peter Bernath hosts the DiRef database, in which you can type the formula for any diatomic molecule and get what was attempted to be every single paper ever published about that diatomic molecule, but this took a huge amount of manual effort from his full-time employee Sean Macleod (not a student, but a senior IT staff who would have likely been paid more than double the average post-doc at the time). The database was probably quite complete in its earlier days, but it went out-of-date extremely quickly. I remember another one of his full-time employees (again not a student) having been tasked with updating it circa 2018 and he was given a long list of journals and was searching in attempt to find every single paper mentioning a molecule in the title and/or abstract, in each of the journals in that list. I don't think this project of updating the database was very successful, and if it was, the database would still only be accurate up to 2018. Now, imagine doing that for triatomics and larger (knowing the vast diversity of molecules that appear in datasets like the GMTKN one)!

For diatomics, I think we could at least come up with a rough estimate of the number of times a paper has been published about that molecule, by counting the query results you get in DiRef for that molecule, and we could scale a functional's error according to the number of times that molecule has been mentioned in the title or abstract of a publication. This would still require a lot of manual work even for diatomics, where DiRef exists, and would require far more work for triatomics and larger moelcules, and even after all this effort, the whole concept will still have its flaws, such as the fact that (for example) H2 and Li2 will appear in that database far more times than many of the other molecules, simply because there's so many "fundamental" studies on these extremely simple benchmark systems, which doesn't necessarily reflect how often you are likely to be using the tested functional on those molecules in a real-world application.

$\endgroup$
8
  • $\begingroup$ I think this is really the core of my argument in a lot of senses, the effort to usefulness ratio here would be incredibly high. If someone did it, it would be great but its hard to believe that would be an optimal use of their time. Interesting to know someone vaguely started to do it with the diatomics though, but that wasn't even a benchmark in the typical manner it seems. $\endgroup$ Jan 10 at 5:48
  • 1
    $\begingroup$ Yea I upvoted your answer. About diatomics, you're right that it's not a benchmark. His group just made a web app that takes as input a diatomic, and outputs all papers about that diatomic (up to the day when the database was last updated for that molecule). This would tell us a rough idea of how often that molecule is studied, which could be used to weight the errors in a benchmark study, but hasn't yet been done. $\endgroup$ Jan 10 at 6:39
  • $\begingroup$ I don't think that creating a benchmark set like what I described is harder than creating a normal benchmark set. The set only needs to be statistically representative, not exhaustive. So one may simply generate a list of all publications that involve computational chemistry, randomly pick 100 publications from the list, manually extract all the molecules calculated in these papers, and randomly pick 100 molecules from the list of molecules. If a molecule was selected N times, assign a weight of N to it. $\endgroup$
    – wzkchem5
    Jan 10 at 8:45
  • 1
    $\begingroup$ Great idea! That sure makes it easier! One can even test how well that works, on the diatomics, using the strategy I described! $\endgroup$ Jan 10 at 12:12
  • $\begingroup$ @wzkchem5 I suspect when that made it to peer review the reviewers would constantly ask for more data to be incorporated, or even specific papers. Its an interesting idea, you definitely could take a monte carlo style approach. I think you would end up with some overlap between similar molecules though that these other benchmarks avoid. With semi-emperical methods however, you can afford to run as much as you want. If you want to do anything post-DFT that might get expensive fast. $\endgroup$ Jan 10 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.