The Project

Overview

There is a growing recognition among academic psychologists that conducting reproducible, reliable, and valid psychological science is substantially harder than many researchers used to believe. Meta-analyses (e.g., Hilgard, Engelhardt, & Rouder, 2017), simulations (e.g., Ioannidis, 2005), and large-scale replication projects (e.g., Open Science Collaboration, 2015) suggest that an unknown, but likely non-trivial, fraction of published findings are overstated or simply false.

The authors of a study have privileged access to information on how the study was conducted and how the data were analyzed. Thus, authors might be able to spot substantial theoretical and methodological problems that are not as visible for other researchers. However, except for few notable exceptions (e.g., Dana Carney’s revised position on power pose, Will Gervais’ post-publication peer review of his own study), researchers do not share this type of information: It is anything but common to publicly declare that one has lost confidence in one’s own previous findings.

But science as a whole would profit from such statements. Resources—for both replication studies and new original research building on previous findings—could be better allocated if the scientific community had full access to the information necessary to judge the reproducibility, reliability, and validity of a given study.

The aim of this project is to destigmatize declaring a loss of confidence in one’s own research finding within the field of psychology. We are collecting statements of loss of confidence based on theoretical or methodological problems (see Frequently Asked Questions). Authors can submit a statement via the loss-of-confidence form. Please also read the more detailed inclusion criteria below. We will collect statements until end of June 2018, and then proceed to compile a list of such statements in the form of a journal article.

Authorship policy

Our expectation is that all researchers who submit at least one complete loss-of-confidence form  fulfilling the inclusion criteria will be co-authors on the resulting paper. Order of authorship will be determined at a later time by consensus agreement between all researchers involved in the project. The journal article will of course not only consist of loss-of-confidence statements, but also include an introduction and a discussion of the project. All researchers contributing a loss-of-confidence statement will be invited to contribute to these sections of the article and will have the possibility to withdraw their statement if they do not agree with the resulting article.

Inclusion criteria

Studies are eligible for inclusion in this project if:

  1. The study in question is an empirical report of a novel finding;
  2. The submitting author has lost confidence in the primary/central result of the paper;
  3. The loss of confidence occurred primarily as a result of theoretical or methodological problems with the study design or data analysis;
  4. The submitting author takes responsibility for the errors in question.

We describe each of these criteria in additional detail below.

  1. Empirical report of a novel finding. Only empirical studies are eligible for inclusion in the LCP. Theories, hypotheses, or predictions that an investigator has lost confidence in are not eligible, as there is much less of a communal stigma against researchers changing their minds about theories than about results. The definition of “novel” is open to interpretation to some degree, but at minimum should exclude studies that were primarily presented as direct or conceptual replications of a previously established phenomenon.
  1. Loss of confidence in the primary result. The submitting author must have lost confidence in the primary result reported in the study, and not in a secondary/auxiliary result. For example, a statement to the effect that one still believes in the central result obtained in a study (which may be main effects or interactions), but no longer believes in qualifying higher-order interactions which are not the central result reported, would not be eligible. The primary result does not have to have been produced in a hypothesis-driven way; an unexpected finding identified through data exploration is eligible for inclusion so long as it constitutes the primary finding reported in a study.
    “Loss of confidence” is taken here to imply a qualitative, and not just quantitative, shift in one’s belief about the evidential value of the finding reported in the paper. Generally speaking, the submitting author’s current state of belief should be that the central results reported in the manuscript are invalid. Note that the loss of confidence is assumed to apply only to the particular findings reported in the publication in question; the authors do not have to (and indeed, will be asked not to) pass any judgment on the status of the broader effect or phenomenon. (For example, admitting to a loss of confidence in a particular operationalization of ego depletion does not imply that the author no longer believes that ego depletion exists.)
    1. Loss of confidence is attributable to known errors. The submitting author’s loss of confidence should be primarily attributable to one or more known methodological or conceptual problems with the study as it was originally reported. Examples of valid concerns include, but are not limited to:
      • p-hacking practices (e.g., using optional stopping or outcome switching without accounting for these extra degrees of freedom) and other practices that are considered questionable research practices (QRPs), including the withholding of studies that “did not work” in case these additional studies clearly undermine the evidentiary value of the published claim
      • Strong evidence that the original effect, while perhaps statistically reliable, should be attributed to a cause other than the one originally postulated, to the degree that it no longer makes sense to speak of the original effect existing. Examples of this might include subsequent data obtained in the same lab strongly implicating the presence of demand characteristics; observational studies failing to report an important unmodeled covariate that better explains the effect when included, etc.
      • Inappropriate modeling decisions that invalidate the original conclusions when correctly modeled (e.g., the authors generalized their conclusions beyond the exact stimuli used in the study, but failed to explicitly model the stimuli as a random factor—and when the appropriate model is used, the result no longer supports the conclusions).

      Note that loss of confidence due to factors that were largely beyond one’s control (e.g., failure to replicate the original finding in subsequent studies in the absence of any obvious problems with the original methods) is not a sufficient basis for inclusion of a study in this project. The loss of confidence must be specifically attributable, with a high degree of confidence, to one or more known methodological or conceptual problems with the execution of the study.

    1. Submitting author takes responsibility. The loss-of-confidence admission requires authors to take responsibility for any problems that ultimately led to the loss of confidence. Therefore,  you may only submit studies to this project on which you were either the lead author, or took a very active role as the senior author, and can claim most of the responsibility for any methodological or conceptual problems that have, since publication, caused you to lose confidence in the original report. This means that you cannot express a loss of confidence in a paper you were only a secondary contributor to.