Can a highly-cited published paper have this type of error?

Question

I know the title sounds opinion-based (and admitedly, part of my question is), but I will try to bring more practical elements to the question:

Here is my problem: I have been testing a force field published in a peer-reviewed paper, and later used by quite a few other authors (I am not sure if I should cite said paper, as I don't want to seem "incriminating")

I was initially able to reproduce quite closely the published results in the original paper while testing it, but they seemed rather unstable (with respect to initial geometry, equilibration time, etc…). To me, this seemed to indicate that the system was not properly equilibrated, and indeed, when equilibrating for a much longer time, I observed the following a behavior (below is the average lattice constant vs number of timesteps for a NPT run at 300K, timestep being 0.5 fs):

Basically, the system relaxes in 2 steps: there’s a first initial “relaxation” to a value that matches well with the paper and experiments (~6.29A), happening over just a few thousand timesteps. But then the system seems to re-adjust on a much slower timescale, and eventually reaches an equilibrium at a lower value (~6.24A).

It is either one fo two things: either I’m doing something wrong, or the original paper published results based on a not properly equilibrated system (which basically makes their force field faulty, and also impacts the credibility of subsequent papers using it)

In favor of the first possibility, I have less experience than the authors, and it seems like something that would be hard to miss for them. However, I am using input files that they were kind enough to provide me themselves, with almost no modification at all. Those files are pretty simple, making the possibility of mistakes rather low, and I’ve been trying to double check again and again.

How likely is it that a force field that has been published in a respectable paper by serious authors and has then been used by many others is faulty and based on non-equilibrated results? It seems like that problem is too basic to go unnoticed. I have tried to contact the author about that issue, providing them a much more detailed account of the problems I found, but they have not reached back.

I can provide more details and explanations if needed. I am currently pretty stuck because of this problem

Answer 1

Many published papers, even extremely highly cited ones by what you call "serious" authors, have issues like these.

Consider for example this one which currently has 862 Google Scholar citations, and has been used as a benchmark for the quantum dynamics of the seven density matrix diagonals in a 7-site model of the FMO complex. The authors were (and still are) two of the most respected scientists in their field, and I personally know the author that did the calculations and think extremely highly of him. But in my paper years later, we showed that the several dozen papers that used their supposedly exact dynamics plot as a benchmark to compare the dynamics of a new method, were not actually comparing to exact dynamics after all:

Likewise, this paper currently has 163 Google Scholar citations and they claimed to have factored the number 143 by finding the ground state of a 4-qubit Hamiltonian with an NMR quantum computer. Two years later, with an undergrad-level student, I discovered that the Hamiltonian not only factored 143 but also larger numbers such as 56153. Dozens of articles were published by journalists, about my discovery, and I particularly find amusing the title of this one: "Quantum computing is so powerful it takes two years to understand what happened".

In a different paper currently with 145 Google Scholar citations, I was unable to reproduce a plot in one of the figures when using their parameter values. I kept insisting this, and the author I emailed kept insisting that there was no mistake, and after seven (very long, because there was a lot of detail involved to figure this out) back-and-forth emails, I got this eighth email:

"Hi Nike,

sorry for the late response.

I at least could check for your first question and I have to apologize here. You are right and I simply misprinted the values. We actually used wc = (50 fs)^(-1) as was the values Ishizaki used in their PNAS for figure 2 and 3. They used the other value for figure 4, a calculation which we did not actually repeat.

Sorry for that but thanks for being insitant and pointing my nose to my mistake."

This email from 14 September 2011 came several months after my initial email on 12 April 2011. The paper unfortunately remains with this error and no published Erratum or Comment. I don't know why they didn't publish an Erratum, but I can say that I didn't publish a Comment because I've in general been reluctant to do so (I feel that it's better to talk to the authors, and decide together what the best course of action is, and if they don't suggest publishing an Erratum together, then they probably don't want me publishing a Comment either; but now that I'm more senior I wouldn't be as hesitant to publish a Comment if I'm confident that I'm right).

In another highly-cited paper in PRL, the authors went into great depths to prove that a problem was in the complexity class QMA but a (different) undergrad student of mine found that the solution can actually be found in constant time, meaning that it's not only trivially in QMA but also in NP, P and even smaller complexity classes.

Similarly, there has been a highly-cited and published paper in which the authors used 50 terms to represent a function as a sum of a particular type of basis function, when in fact I was able to get the same quality of fit with only two of the same type of basis function.

I can also provide several examples of errors or impactful typos in spectroscopy papers or quantum chemistry papers related to spectroscopic properties of molecules.

The one example I can think of, where I turned out to be wrong after contacting an author due to not being able to reproduce her result (in this case a mathematical derivation), my first email was an extremely carefully written report authored by myself (PhD student at the time), a post-doc, and our supervisor, and we didn't get any response back at all. It took me approximately two years to figure out (on my own!) why we were wrong and the original paper was right, but I had still never heard from the author of that paper. Several years later I met up with her, and then started having meals with her annually, and while I originally thought that not receiving a response from her was a sign of coldness and not wanting to talk to us, she turned out to be an extremely warm person who just might have had too much going on in her life to reply to the first email, or wasn't able to figure out the reason why my result didn't match hers.

---

When things like these happen, it's best to compose a careful email to the authors (which you've already done!) and continue to investigate whether or not it's you that's wrong (which you are doing here). So you are going about this in the correct way! Keep in mind the timeline in my above examples (several months before the author in the first example finally agreed that the paper had an error; two years for me to find my mistake in the last example), which is there to show you that figuring out these things can take a very long time.

Answer 2

I was actually going to provide this as a comment as this in itself is not a proper answer. But since this got a bit too lengthy for a comment, I am providing this as an answer.

It is possible for a highly cited paper to have non-equilibrated results. However, you should consider the fact that you may be using a different precision for your calculations. Also, if the paper was published a long time back, that could mean that they had significantly slower computing power. Therefore they may never have known that equilibrating the system even further results in a smaller lattice constant (not much guilt on their part then).

Since you are facing doubts, I recommend that you start with an energy minimization, and move to a good NPT equilibration followed by an NVE run. During the NVE run, you can check the system pressure, KE and PE (which should all be constant) to check if the system has actually reached equilibrium. In any case your system (when confirmed to be in equilibrium) has a different lattice constant, you can simply cite the original paper and mention the difference so that the same error does not affect others.

Note: I have seen that LAMMPS installations on different systems had minor differences in results which I think could have been caused by round off errors etc.