Yesterday, Science published a comment from Gilbert et al on the reproducibility project that received quite a bit of attention last year. Science also included a reply to the comment from the original authors.

There’s a lot to digest across these three documents, but my initial impressions (for what they’re worth) basically boils down to this: Maybe the problem is not quite as big as what the reproducibility project suggested initially, but a problem definitely still exists. There’s a lot of other evidence suggesting that there are problems in the conduct and communication of science, so even if everything Gilbert et al. say is correct, there’s still quite a lot of other evidence that is unrelated to the reproducibility project that suggests the same thing.

The primary point I want to make though, is that despite the fact that this was published all of about 24 hours ago, there are already a few thoughtful analyses and commentaries on it from respected researchers in the field. For instance, Sanjay Srivastava’s, Uri Simonsohn’s, and Gelman’s inevitable take on it along with the associated commentary at Gelman’s blog. There are no doubt others that I’m missing, and some that have yet to be written, but will be in the coming week or so. I get the sense that these individuals had seen the commentary and reply before they were officially published, so were able to be pretty quick about publishing their thoughts.

Now, let’s reflect on this for a minute. There’s a commentary and a reply published in Science, and then a series of other replies. The quality of all of these will vary, and I’d argue that that variability is not particularly well explained by where the commentary is published. Both Srivastava and Simonsohn clearly put a lot of thought into their posts, with Simonsohn even stating that he circulated several drafts before publishing it. Is there a reason why his contribution should be valued less than either of those that will make it into a print journal? It seems a bit ridiculous to me. Especially when, if we wanted to get these reactions in a print journal, we would likely have to wait several months. Why do I need to wait several months to see this discussion unfold?

The world is changing, and academic institutions must work harder to adapt to those changes.

As an aside, I think it’s also worth acknowledging that open access to materials has driven a good part of this important work. A victory for Open Science.