A recent question has caused me to think about the idea of reproducible-research and what the concept means. I think that, while it may be fairly common to use this term, that a better term would be preferable, so as to make an important distinction.
The term I would prefer for this would be transparent-research or possibly verifiable-research. These aren't quite the same, actually. Transparent simply means that the researcher has revealed enough that a reader can follow the reasoning completely. Verifiable means that one can demonstrate that the author hasn't lied in presenting conclusion. Both of these are valuable, of course, so my concern is with the naming, not the validity of the underlying idea.
Currently, it seems that the term is used for a fairly simple idea that is applicable to (I'm afraid) only some fields, such as computer science. The idea is that researchers should publish their code and data so that others can run that code on that data to assure that the original authors aren't misrepresenting their results. Run the same code on the same data - get the same results.
Of course, it has an additional benefit, in that the reader can examine the code to determine whether it is, indeed, the appropriate code to answer the research question. That is definitely a plus, but I'll suggest below that we need more.
The current concept isn't quite so good about the data, I think. Having a fixed data set only partially helps us come to the proper answer for deep questions. You learn more, of course, if you know the standards and techniques by which the data was collected (search terms for some sorts of data) than the specific data itself. If a technique works for only a single set of data it applies to only that set of data, rather than the larger question for which the data was gathered. If using one set of data gives one result, but a different set, gathered to the same standards, gives an opposing result, you have learned almost nothing.
My conclusion here is that the current idea of reproducible-research is useful, it is weak and mis-named. It is weak because of the second idea above (fixed data set), but it is misnamed as it is not adequate to answer a more important question. But transparent-research seems to cover the current concept better.
The More Important Problem:
Recently it has become obvious that quite a lot of published research in the sciences can't be validated, not because the author doesn't publish enough information (which they probably don't), but because taking the same research questions and trying to answer them independently with different models, techniques, and data, leads to contrary results. While this can be due to flaws of statistical design in some sciences it seems to go deeper than that. Unfortunately this sad situation pervades a lot of educational research, which is dear to my heart.
I would prefer that reproducible-research as a term be reserved for questions that relate to this deeper concern. To do this, a researcher would need to reveal more, so that an independent party has enough information to attack the same research question, perhaps in a different way, but get better evidence as to the actual scientific truth, not just the appropriateness of the methodology originally used.
Why It Matters:
In the sciences we seek truth not just results. Often truth is evasive or impossible to achieve definitively, so we often use techniques that give us evidence of the truth, not (as in mathematics) proof of the truth. Statistical techniques in particular work to quantify the potential that our conclusions are wrong. In a study carried out (properly) with 99% confidence, replicating the study (properly) will yield an improper result 1% of the time. But you don't know which time, unless you reproduce the study independently many times. But the replications of the study need to be independent, and so can't use identical methods on identical data.
-- I may need to edit this or append to it. Thinking...thinking