28
$\begingroup$

I know the title sounds opinion-based (and admitedly, part of my question is), but I will try to bring more practical elements to the question:

Here is my problem: I have been testing a force field published in a peer-reviewed paper, and later used by quite a few other authors (I am not sure if I should cite said paper, as I don't want to seem "incriminating")

I was initially able to reproduce quite closely the published results in the original paper while testing it, but they seemed rather unstable (with respect to initial geometry, equilibration time, etc…). To me, this seemed to indicate that the system was not properly equilibrated, and indeed, when equilibrating for a much longer time, I observed the following a behavior (below is the average lattice constant vs number of timesteps for a NPT run at 300K, timestep being 0.5 fs):

enter image description here

Basically, the system relaxes in 2 steps: there’s a first initial “relaxation” to a value that matches well with the paper and experiments (~6.29A), happening over just a few thousand timesteps. But then the system seems to re-adjust on a much slower timescale, and eventually reaches an equilibrium at a lower value (~6.24A).

It is either one fo two things: either I’m doing something wrong, or the original paper published results based on a not properly equilibrated system (which basically makes their force field faulty, and also impacts the credibility of subsequent papers using it)

In favor of the first possibility, I have less experience than the authors, and it seems like something that would be hard to miss for them. However, I am using input files that they were kind enough to provide me themselves, with almost no modification at all. Those files are pretty simple, making the possibility of mistakes rather low, and I’ve been trying to double check again and again.

How likely is it that a force field that has been published in a respectable paper by serious authors and has then been used by many others is faulty and based on non-equilibrated results? It seems like that problem is too basic to go unnoticed. I have tried to contact the author about that issue, providing them a much more detailed account of the problems I found, but they have not reached back.

I can provide more details and explanations if needed. I am currently pretty stuck because of this problem

$\endgroup$
1
  • 1
    $\begingroup$ +10 for a great question!. Posts with more than one question in them get closed as "too broad", so I suggest to stick to the question in this post's title (about whether or not it's possible for a highly-cited published paper to have this type of error), and ask the more science-related question in a separate post. Also, since this has already become a "Hot Network Question", I've simplified the title a bit to make it more accessible to others in the network :) $\endgroup$ Oct 13 at 17:01
24
$\begingroup$

Many published papers, even extremely highly cited ones by what you call "serious" authors, have issues like these.

Consider for example this one which currently has 862 Google Scholar citations, and has been used as a benchmark for the quantum dynamics of the seven density matrix diagonals in a 7-site model of the FMO complex. The authors were (and still are) two of the most respected scientists in their field, and I personally know the author that did the calculations and think extremely highly of him. But in my paper years later, we showed that the several dozen papers that used their supposedly exact dynamics plot as a benchmark to compare the dynamics of a new method, were not actually comparing to exact dynamics after all:

enter image description here

Likewise, this paper currently has 163 Google Scholar citations and they claimed to have factored the number 143 by finding the ground state of a 4-qubit Hamiltonian with an NMR quantum computer. Two years later, with an undergrad-level student, I discovered that the Hamiltonian not only factored 143 but also larger numbers such as 56153. Dozens of articles were published by journalists, about my discovery, and I particularly find amusing the title of this one: "Quantum computing is so powerful it takes two years to understand what happened".

In a different paper currently with 145 Google Scholar citations, I was unable to reproduce a plot in one of the figures when using their parameter values. I kept insisting this, and the author I emailed kept insisting that there was no mistake, and after seven (very long, because there was a lot of detail involved to figure this out) back-and-forth emails, I got this eighth email:

"Hi Nike,

sorry for the late response.

I at least could check for your first question and I have to apologize here. You are right and I simply misprinted the values. We actually used wc = (50 fs)^(-1) as was the values Ishizaki used in their PNAS for figure 2 and 3. They used the other value for figure 4, a calculation which we did not actually repeat.

Sorry for that but thanks for being insitant and pointing my nose to my mistake."

This email from 14 September 2011 came several months after my initial email on 12 April 2011. The paper unfortunately remains with this error and no published Erratum or Comment. I don't know why they didn't publish an Erratum, but I can say that I didn't publish a Comment because I've in general been reluctant to do so (I feel that it's better to talk to the authors, and decide together what the best course of action is, and if they don't suggest publishing an Erratum together, then they probably don't want me publishing a Comment either; but now that I'm more senior I wouldn't be as hesitant to publish a Comment if I'm confident that I'm right).

In another highly-cited paper in PRL, the authors went into great depths to prove that a problem was in the complexity class QMA but a (different) undergrad student of mine found that the solution can actually be found in constant time, meaning that it's not only trivially in QMA but also in NP, P and even smaller complexity classes.

Similarly, there has been a highly-cited and published paper in which the authors used 50 terms to represent a function as a sum of a particular type of basis function, when in fact I was able to get the same quality of fit with only two of the same type of basis function.

I can also provide several examples of errors or impactful typos in spectroscopy papers or quantum chemistry papers related to spectroscopic properties of molecules.

The one example I can think of, where I turned out to be wrong after contacting an author due to not being able to reproduce her result (in this case a mathematical derivation), my first email was an extremely carefully written report authored by myself (PhD student at the time), a post-doc, and our supervisor, and we didn't get any response back at all. It took me approximately two years to figure out (on my own!) why we were wrong and the original paper was right, but I had still never heard from the author of that paper. Several years later I met up with her, and then started having meals with her annually, and while I originally thought that not receiving a response from her was a sign of coldness and not wanting to talk to us, she turned out to be an extremely warm person who just might have had too much going on in her life to reply to the first email, or wasn't able to figure out the reason why my result didn't match hers.


When things like these happen, it's best to compose a careful email to the authors (which you've already done!) and continue to investigate whether or not it's you that's wrong (which you are doing here). So you are going about this in the correct way! Keep in mind the timeline in my above examples (several months before the author in the first example finally agreed that the paper had an error; two years for me to find my mistake in the last example), which is there to show you that figuring out these things can take a very long time.

$\endgroup$
1
  • 2
    $\begingroup$ To be frank, it can be extremly difficult to figure out why one's result may not matching with yours. But you are right, we need more this kind of scrutony in science. $\endgroup$
    – Greg
    Oct 14 at 6:35
15
$\begingroup$

I was actually going to provide this as a comment as this in itself is not a proper answer. But since this got a bit too lengthy for a comment, I am providing this as an answer.

It is possible for a highly cited paper to have non-equilibrated results. However, you should consider the fact that you may be using a different precision for your calculations. Also, if the paper was published a long time back, that could mean that they had significantly slower computing power. Therefore they may never have known that equilibrating the system even further results in a smaller lattice constant (not much guilt on their part then).

Since you are facing doubts, I recommend that you start with an energy minimization, and move to a good NPT equilibration followed by an NVE run. During the NVE run, you can check the system pressure, KE and PE (which should all be constant) to check if the system has actually reached equilibrium. In any case your system (when confirmed to be in equilibrium) has a different lattice constant, you can simply cite the original paper and mention the difference so that the same error does not affect others.

Note: I have seen that LAMMPS installations on different systems had minor differences in results which I think could have been caused by round off errors etc.

$\endgroup$
3
  • $\begingroup$ thanks for your reply. The paper is recent (2017). As I mentionned, I'm mostly using input files they kindly provided, so most of the settings should be the same (I only increased the precision of the pppm scheme, because the slow equilibration would give even stranger results with their initial settings). I have currently been doing minim+NPT, I can try to check further with the NVE as you suggest, thanks for the tip. However, in the case where the force field indeed converges to a wrong value of the lattice constant, I should not use it (I will at least need to re-fit the force field) $\endgroup$ Oct 13 at 8:59
  • 1
    $\begingroup$ @BarbaudJulien I do not know your use case and so cannot give an exact answer. But if you are studying the trends of how a property (such as thermal conductivity) behaves, I don't think that the difference of the lattice constant would be much of an issue (Even though the actual value is different, the trend usually holds). However I am not an expert of the field and so suggest that you either contact someone more experienced, or to justify what you do by clearly mentioning your decisions in your publication. $\endgroup$
    – PBH
    Oct 13 at 9:17
  • 1
    $\begingroup$ Also, if you do refit the force field, do make sure that you validate the results and provide all the details in your publication so that someone else can use your work in the future. Sometimes matching the lattice constant may not be the best thing as other properties could be different. I remember something similar when comparing two force fields where one managed to get the thermal conductivity right, but failed to predict the elastic-plastic transition regime of the material when deformed. So be cautious and validate your results. $\endgroup$
    – PBH
    Oct 13 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.