0

A recent question has caused me to think about the idea of reproducible-research and what the concept means. I think that, while it may be fairly common to use this term, that a better term would be preferable, so as to make an important distinction.

The term I would prefer for this would be transparent-research or possibly verifiable-research. These aren't quite the same, actually. Transparent simply means that the researcher has revealed enough that a reader can follow the reasoning completely. Verifiable means that one can demonstrate that the author hasn't lied in presenting conclusion. Both of these are valuable, of course, so my concern is with the naming, not the validity of the underlying idea.

Background:

Currently, it seems that the term is used for a fairly simple idea that is applicable to (I'm afraid) only some fields, such as computer science. The idea is that researchers should publish their code and data so that others can run that code on that data to assure that the original authors aren't misrepresenting their results. Run the same code on the same data - get the same results.

Of course, it has an additional benefit, in that the reader can examine the code to determine whether it is, indeed, the appropriate code to answer the research question. That is definitely a plus, but I'll suggest below that we need more.

The current concept isn't quite so good about the data, I think. Having a fixed data set only partially helps us come to the proper answer for deep questions. You learn more, of course, if you know the standards and techniques by which the data was collected (search terms for some sorts of data) than the specific data itself. If a technique works for only a single set of data it applies to only that set of data, rather than the larger question for which the data was gathered. If using one set of data gives one result, but a different set, gathered to the same standards, gives an opposing result, you have learned almost nothing.

My conclusion here is that the current idea of is useful, it is weak and mis-named. It is weak because of the second idea above (fixed data set), but it is misnamed as it is not adequate to answer a more important question. But transparent-research seems to cover the current concept better.

The More Important Problem:

Recently it has become obvious that quite a lot of published research in the sciences can't be validated, not because the author doesn't publish enough information (which they probably don't), but because taking the same research questions and trying to answer them independently with different models, techniques, and data, leads to contrary results. While this can be due to flaws of statistical design in some sciences it seems to go deeper than that. Unfortunately this sad situation pervades a lot of educational research, which is dear to my heart.

I would prefer that reproducible-research as a term be reserved for questions that relate to this deeper concern. To do this, a researcher would need to reveal more, so that an independent party has enough information to attack the same research question, perhaps in a different way, but get better evidence as to the actual scientific truth, not just the appropriateness of the methodology originally used.

Why It Matters:

In the sciences we seek truth not just results. Often truth is evasive or impossible to achieve definitively, so we often use techniques that give us evidence of the truth, not (as in mathematics) proof of the truth. Statistical techniques in particular work to quantify the potential that our conclusions are wrong. In a study carried out (properly) with 99% confidence, replicating the study (properly) will yield an improper result 1% of the time. But you don't know which time, unless you reproduce the study independently many times. But the replications of the study need to be independent, and so can't use identical methods on identical data.

-- I may need to edit this or append to it. Thinking...thinking

4

I am not exactly sure what you want to achieve here. Tags mainly exist to make questions easier to find or to allow potential experts to subscribe to them. If you want to shape the usage of scientific terminology, tag names on this site are the wrong place to do this.

As it is currently used, the seems to cover all that you describe and is still sufficiently narrow to only contain fifty questions. I see no problems arising from this such as users not being able to find certain questions. It may be a tad more difficult in some cases, but this does not even remotely justify the effort of separating these tags and expecting our users to adhere to the distinction. The only thing we may change is to make the tag wiki a bit broader.

3

I think the more important usage for SE purposes is what the general community has adopted as a meaning for "reproducible research." The general consensus is that it's closer to your first usage, but also much more widespread. It's starting to become a major concern in computational research across all of science and engineering.

The latter concern is also serious, but it's also harder to address. It's also part of reproducible research, but I believe that restricting the use of the tag to that specific issue does a disservice to the site.

  • I raise the issue since the general community (not here) seems to use the broader definition. At least as I read it in NYT and WaPo, for example. – Buffy Nov 22 '18 at 20:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .