LR ≠1J should exist. But it doesn’t.
Or at least I don’t think LR ≠1J exists! If such a publication has evaded my notice, then my gratitude for having a deficit in my knowledge remedied will more than compensate me for the embarrassment of having the same exposed (happens all the time!). I will be sure to feature it in a follow-up post.
The basic idea (described more fully in the journal’s “mission statement” below) is to promote identification of study designs that scholars who disagree about a proposition would agree would generate evidence relevant to their competing conjectures–regardless of what studies based on such designs actually find. Articles proposing designs of this sort would be selected for publication and only then be carried out, by the proposing researchers with funding from the journal, which would publish the results too.
Now I am aware of a set of real journals that have a similar motivation.
One is the Journal of Articles in Support of the Null Hypothesis, which as its title implies publishes papers reporting studies that fail to “reject” the null. Like JASNH, LR ≠1J would try to offset the “file drawer” bias and like bad consequences associated with the convention of publishing only findings that are “significant at p < 0.05.”
But it would try to do more. By publishing studies that are deemed to have valid designs and that have not actually been performed yet, LR ≠1J would seek to change the odd, sad professional sensibility favoring studies that confirm researchers’ hypotheses (giving a preference to studies that “reject the null” alternative is actually a confirmatory proof strategy–among other bad things). It would also try to neutralize the myriad potential psychological & other biases on the part of reviewers and readers that might impede publication of studies that furnish confirming or disconfirming evidence at odds with propositions that many scholars might have a stake in.
Some additional journals that likewise try (very sensibly) to promote recognition of studies that report unexpected, surprising, or controversial findings include Contradicting Results in Science; Journal of Serendipitous and Unexpected Results; and Journal of Negative Results in Biomedicine. These journals are very worthwhile, too, but still focus on results, not the identification of designs the validity of such would be recognized ex ante by reasonable people who disagree!
I am also aware of the idea to set up registries for designs for studies before they are carried out. See this program, e.g. A great idea, certainly. But it doesn’t seem realistic, since there is little incentive for people to register, even less than that to report “nonfindings,” and no mechanism that steers researchers toward selection of designs that disagreeing scholars would agree in advance will yield knowledge no matter what the resulting studies find.
But if there are additional journals besides theses that have objectives parallel to those of LR ≠1J, please tell me about those too (even if they are not identical to LR ≠1J).
I also want to be sure to add — in case anyone else thinks this is a good idea — that it occurred to me as a result of the work of, and conversations with, Jay Koehler, who I think was the first person to suggest to me that it would be useful to have a “methods sections only” review process, in which referrees reviewed papers based on the methods section without seeing the results. LR ≠1J is like that but says to authors, “Submit before you know the results too.”
Actually, there are journals like this in physics. Papers in theoretical physics often describe why observations of a certain sort would answer or resolve a disputed problem well before there exists the requisite apparatus for making the measurements. My favorite example is Bell’s inequalities– which was readily understood (by those paying attention, anyway!) to describe the guts of an experiment that couldn’t then be carried out but that would settle the issues about the possibility of an as-yet unidentified “hidden variables” alternative to quantum mechanics. A set of increasingly exacting tests started some 15 yrs later–with many, including Bell himself, open to the possibly (maybe even hoping for it!) that they would show Einstein was right to view quantum mechanics as “incomplete” due to its irreducibly probabilistic nature. He wasn’t.
Wouldn’t it be cool if psychology worked this way?
As you can see, LR ≠1J, as I envision it, would supply funding for studies with a likelihood ratio ≠ 1 on some proposition of general interest on which there is a meaningful division of professional opinion. So likely its coming into being — assuming it doesn’t already exist! — would involve obtaining support from an enlightened benefactor. If such a benefactor could be found, though, I have to believe that there would be grateful, public-spirited scholars willing to reciprocate the benefactor’s contribution to this collective good by donating the time & care it would take to edit it properly.
Likelihood Ratio ≠ 1 Journal (LR ≠1J)
The motivation for this journal is to overcome the contribution that a sad and strange collection of psychological dynamics makes to impeding the advancement of knowledge. These dynamics all involve the pressure (usually unconscious) to conform one’s assessment of the validity and evidentiary significance of a study to some stake one has in accepting or rejecting the conclusion.
(1) Confirmation bias is one of these dynamics, certainly (Koehler 1993).
(2) A sort of “exhilaration bias”—one that consists in the (understandable; admirable!) excitement that members of a scholarly enterprise generally experience at discovery of a surprising new result (Wilson 1993)—can distort perceptions of the validity and significance of a study as well.
(3) So can motivated reasoning when the study addresses politically charged topics (Lord, Ross & Lepper 1979).
(4) Self-serving biases could theoretically motivate some journal referees or scholars assessing studies published in peer-reviewed journals to form negative assessments of the validity or significance of studies that challenge positions associated with their own work. Note: We stress theoretically; there are no confirmed instances of such an occurrence. But less informed observers understandably worry about this possibility.
(5) Finally, in an anomalous contradiction of the strictures of valid causal inference (Popper 1959; Wason 1968), the practice of publishing only results that confirm study hypotheses denies researchers and others the opportunity to discount the probability of various plausible conjectures that have not been corroborated by studies that one reasonably would have expected to corroborate them if they were in fact true.
LR ≠1J will solicit submissions that describe proposed studies that (1) have not yet been carried out but that (2) scholars with opposing priors (ones that assign odds of greater than and less 1:1, respectively) on some proposition agree would generate a basis for revising their estimation of the probability that the proposition is true regardless of the result. Such proposals will be reviewed by referees who in fact have opposing priors on the proposition in question. Positive consideration will be given to proposals submitted by collaborating scholars who can demonstrate that they have opposing priors. The authors of selected submissions will thereafter be supplied the funding necessary to carry out the study in exchange for agreeing to publication of the results in LR ≠ 1J. (Papers describing the design and ones reporting the results will be published separately, and in sequence, to promote the success of LR≠1’s sister journal, “Put Your Money Where Your Mouth Is, Mr./Ms. ‘That’s Obvious,’ ” which will conduct on-line predication markets for “experts” & others willing to bet on the outcome of pending LR≠1 studies.)
In cases where submissions are “rejected” because of the failure of reviewers with opposing priors to agree on the validity of the design, LR ≠ 1J will publish the proposed study design along with the referee reports. The rationale for doing so is to assure readers that reviewers’ own priors are not unconsciously biasing them toward anticipatory denial of the validity of designs that they fear (unconsciously, of course) might generate evidence that warrants treating the propositions to which they are pre-committed as less probably true than they or others would take them to be.
For comic relief, LR ≠1J will also run a feature that publishes reviews of articles submitted to other journals that LR≠1J referees agree suggest the potential operation of one of the influences identified above.
References
Koehler, J.J. The Influence of Prior Beliefs on Scientific Judgments of Evidence Quality. Org. Behavior & Human Decision Processes 56, 28-55 (1993).
Lord, C.G., Ross, L. & Lepper, M.R. Biased Assimilation and Attitude Polarization – Effects of Prior Theories on Subsequently Considered Evidence. Journal of Personality and Social Psychology 37, 2098-2109 (1979).
Popper, K.R. The logic of scientific discovery. (Basic Books, New York,; 1959).
Wason, P.C. Reasoning about a rule. Q. J. Exp. Psychol. 20, 273-281 (1968).
Wilson, T.D., DePaulo, B.M., Mook, D.G. & Klaaren, K.J. Scientists’ Evaluations of Research. Psychol Sci 4, 322-325 (1993).
Predictably, many thoughtful replies, including ones that have filled in my own picture of how scholars are addressing the dynamics that I addressed in this post. I discuss them here.