12
$\begingroup$

One of the key pillars of scientific method is the reproducibility of scientific results. A 2016 survey by Nature revealed that many fields in science suffer from reproducibility crisis. Lejaeghere et. al$^1$ studied Reproducibility in density functional theory calculations of solids by calculating equation-of-state data for four classes of DFT implementations, totaling 40 methods. They found that most codes agree very well, with pairwise differences that are comparable to those between different high-precision experiments. However the study was limited to the equation of state calculation and a few materials only.

Question: How reproducible are results from materials modelling? Are there any famous results which could not be reproduced?

References

  1. Lejaeghere, Kurt, et al. "Reproducibility in density functional theory calculations of solids." Science 351.6280 (2016): aad3000.
Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I don't know if there's a definitive answer about this, but maybe you could just ask for other papers on the topic of reproducibility? I know one, which is the GW100 paper that compares several different GW codes for the same 100-molecule dataset. $\endgroup$
    – Nike Dattani
    May 16, 2020 at 16:36
  • 1
    $\begingroup$ I have changed the question asking for specific examples for results which could not be reproduced $\endgroup$
    – Thomas ♦
    May 17, 2020 at 3:40
  • $\begingroup$ The notion of "reproducibility" in reproducibility crisis and in the paper you mention means something completely different. I do not think we should conflate the fact that different methods have different biases and errors (what a shock!) with the general concept of reproducibility in research (which is much more about ethics, data management, proper documentation, correctly understanding and applying statistics etc). $\endgroup$
    – Greg
    May 28, 2020 at 18:40

2 Answers 2

Reset to default
9
$\begingroup$

Yes, I can think of a few (non-famous) cases in the literature where results could not be reproduced, but this lack of reproducibility usually comes from omission of details about the computations in the original publications. If you do the same calculation with the same method, you should always get the same results, but if you don't know what exactly was originally calculated, you may get somewhat different results. For instance, for adsorption energies and activation energies of surface caculations, the super cell size matters, but it is not usually reported in the literature for some reason.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Reproducibility is not limited to same calculation with same method. Lejaeghere et. al paper was about reproducibility across all DFT codes and methods. For example if a code predict that a particular material has very special property and if it is experimentally verified, we expect all other methods to predict the same (unless there is drawback for other methods because of choice of approximations etc.). I am asking about reproducibility in this sense $\endgroup$
    – Thomas ♦
    May 17, 2020 at 13:30
  • $\begingroup$ What could be "some reason"? $\endgroup$
    – Camps ♦
    Jun 1, 2020 at 21:56
5
$\begingroup$

While I am not sure how famous this was, there was a paper in Nature Nanotechnology that proposed a design for a charge-driven molecular water pump. Later, though, another article argued that the water flow the first paper reported was actually the result of a numerical artefact.

This is the original paper:

Gong, X., Li, J., Lu, H. et al. A charge-driven molecular water pump. Nature Nanotech 2, 709–712 (2007). https://doi.org/10.1038/nnano.2007.320

The article that challenges the results of the original paper:

Wong-ekkabut, J., Miettinen, M., Dias, C. et al. Static charges cannot drive a continuous flow of water molecules through a carbon nanotube. Nature Nanotech 5, 555–557 (2010). https://doi.org/10.1038/nnano.2010.152

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .