There are many benchmark sets in different branches of computational chemistry and physics, that are frequently used to assess the "average error" of a method for a certain property, on a certain set of systems. For example, the GMTKN benchmark sets of the Grimme group, the benchmark set compiled by the Head-Gordon group, as well as numerous other benchmark sets, are used for the benchmarking of reaction energies and barriers of molecular reactions, non-covalent interaction energies of molecules, and relative conformational energies etc. Other benchmark sets may deal with excitation energies, geometries, etc., just to name a few, and some may be more focused on periodic systems and surfaces. Usually, one computes the data entries of a benchmark set using several methods (with different functionals, basis sets, etc.), compares them against the reference values, and calculates the mean absolute error and/or RMS error of the methods. The results are then used to assess the relative accuracy of the methods, and can also be used as an estimate of the typical error magnitudes of the methods.
Now, observe that what one really obtains from the benchmarking process, is the average error of a method when the user randomly picks a system from the benchmark set and calculates that system. However, in the real world the user is instead choosing a system based on their research project, and calculating that system. This raises a question: the error estimate from the benchmark study can only make sense, if the benchmark set is a statistically representative sample from all the systems that will be calculated after the benchmark paper is published. But none of the benchmark sets that I'm aware of seem to have taken this into account. Specifically, consider two systems A and B in a benchmark set. I've never seen any benchmark set that assigns appropriate weights to these two systems based on how likely a user will compute a system that is similar to A, compared to computing a system that is similar to B. Most benchmark sets do try to include a sufficiently diverse set of molecules, but even if a benchmark set has exhausted all important classes of systems, this still does not guarantee that it is a statistically representative sample from all the systems that will be studied in the future, unless they are explicitly weighted to account for this. A few benchmark sets (like GMTKN) do assign weights to the different entries of the benchmark set, but the weights only depend on the difficulty of the corresponding calculations; data entries that are hard to compute accurately often receive a smaller weight so that they do not unnecessarily dominate the error function. In particular, a commonly studied type of molecule is not weighted more than a rarely studied, but equally difficult type of molecule.
So my question is: are there any benchmark sets, regardless of what property or what kind of systems they focus on, that explicitly take into account the number of researchers in different subfields, and use this to weight its entries? For example, if a benchmark set contains the SARS-CoV and SARS-CoV-2 spike proteins, and there are 10x more publications on the latter than the former, then the latter receives a weight factor that is 10 times that of the former, even though the two molecules are very similar in terms of prediction difficulty. Only by this way can we point to the benchmark result and say, e.g. "this is the expected error of a random computational prediction of the binding Gibbs free energy of a spike protein with an acceptor/antibody". Of course, what we really want is the expected error of a future research result, not a research result randomly drawn from published data, so an even more rigorous approach is to include a time series prediction that predicts how many publications related to a certain class of molecules will come out in the next few years. But maybe this is too far of a stretch from what has actually been done in existing benchmark sets, so I'm already more than pleased to know if any existing benchmark set has its entries weighted by publication counts, or other metrics that can be used to measure the popularity of different classes of molecules (or materials in general) based on existing publications.