So I started to answer one of the interesting comments in response to the last post & found myself convinced that the issues involved warranted their own post. So this one “supplements” & “adjusts” the last.
And by the way, I anticipate “supplementing” & “adjusting” everything I have ever said and ever will say. If you don’t see why that’s the right attitude to have, then probably you aren’t engaged in the same activity I am (which isn’t to say that I plan to supplement & adjust every blog post w/ another; that’s not the “activity” I mean to be involved in, but rather a symptom of something that perhaps I should worry about, and you too since you seem to be reading this).
Here’s the question (from JB):
I’m puzzled about how the NRC dealt with Figure 2 in this paper, the “Canada graph” of Donohue and Wolfers. This is not multiple regression. (I agree that multiple regression is vastly over-used and that statistical control of the sort it attempts to do is much more difficult, if not impossible in many situations). But this graph settled the issue for me. It is not a regression analysis. . . .
Here’s my answer:
@JB: The answer (to the question, what did NRC say about Fig. 2 in D&W) is . . . nothing virtually nothing!
As you note, this is not the sort of multivariate regression analysis that the NRC’s expert panel on the death penalty had in mind when it “recommend[ed] that these studies not be used to inform deliberations requiring judgments about the effect of death penalty on homicide.”
Your calling attention to this cool Figure furnishes me with an opportunity to supplement my post in a manner that (a) corrects a misimpression that it could easily have invited; and (b) makes a point that is just plain important, one I know you know but I want to be sure others who read my post do too.
The NRC reports are saying that a certain kind of analysis – the one that is afforded the highest level of respect by economists; that’s an issue that they really should talk about—is not valid in this context. In this context – deterrence of homicide by criminal law (whether gun control or capital punishment) — these studies don’t give us any more or less reason to believe one thing or the other.
But that doesn’t mean that it is pointless to think about deterrence, or unjustifiable for us to have positions on it, when we are deliberating about criminal laws, including gun control & capital punishment!
Two points:
First, just because one empirical method turns out to have a likelihood ratio of 1 doesn’t mean all forms of evidence have LR = 1!
You say, “hey, look at this simple comparison: our homicide rate & Candada’s are highly correlated notwithstanding how radically they differ in the use of the death penalty over time. That’s pretty compelling!”
I think you would agree with me that that this evidence doesn’t literally “settle the issue.” We know what people who would stand by their regression analyses (and others who merely wish those sorts of analyses could actually help) would say. Thinks like …
- maybe the use of the death penalty is what kept the homicide rate in the US in “synch” with the Canadian one (i.e., w/o it, the U.S. rate would have accelerated relative to Canada, due to exogenous influences that differ in the 2 nations);
- maybe when the death penalty isn’t or can’t be (b/c of constitutional probhition) used, legislators “make up the difference” by increasing the certainty of other, less severe punishments, and it is still the case that we can deter for “less” by adding capital punishment to the mix (after getting rid of all the cost-inflating, obstructionist litigation, of course);
- maybe the death penalty work as James Fitzjames Stephen imagines – as a preference shaping device – and Canadians, b/c they watch so much U.S. TV are morally moulded by our culture (in effect, they are free riding on all our work to shape preferences through executing our citizens–outrageous);
- variation in US homicide rates in response to the death penalty is too fine-grained to be picked up by these data, which don’t rule out that the U.S. homicide rate would have decelerated in relation to Canada if the US had used capital punishment more frequently after Gregg;
- the Donohue and Wolfers chart excludes hockey-related deaths resulting from player brawls and errant slapshots that careen lethally into the stands, and thus grossly understates the homicide rate in Canada (compare how few players and fans have been killed by baseball since Gregg!);
- etc. etc. etc.
These are perfectly legitimate points, I’d say. But what is the upshot?
They certainly don’t mean that evidence of the sort reflected in Fig. 2 is entitled to no weight – that its “Likelihood Ratio = 1.” If someone thinks that that’s how empirical proof works – that evidence either “proves” something “conclusively,” or “proves nothing, because it hasn’t ruled out all alternative explanations”—is “empirical-science illiterate” (we need a measure for this!).
These points just present us with reasons to understand why the data in Fig. 2 don’t mean LR ≠ ε (if the hypothesis is “death penalty deter”; if hypothesis is “death penalty doesn’t,” then why LR ≠ ∞).
I agree with you that Fig 2 has a pretty healthy LR – say, 0.2, if the hypothesis is “the death penalty deters” – which is to say, that that I believe the correlation between U.S. and Canadian homicide rates is “5 times more consistent with” the alternative hypothesis (“doesn’t deter”).
And , of course, this way of talking is all just a stylized way of representing how to think about this—I’m using the statistical concept of “likelihood ratio” & Bayesianism as a heuristic. I have no idea what the LR really is, and I haven’t just multiplied my “priors” by it.
But I do have an idea (a conviction, in fact) about the sensible way to make sense of empirical evidence. It’s that it should be evaluated not as “proving” things but as supplying more or less reason to believe one thing or another. So when one is presented with empirical evidence, one shouldn’t say either “yes, game over!” or “pfffff … what about this that & the other thing…” but rather should supplement & adjust what one believes, and how confidently, after reflecting on the evidence for a long enough time to truly understand why it supports a particular infernece and how strongly.
Second, even when we recognize that an empirical proposition relevant to a policy matter admits of competing, plausible conjectures (they don’t have to be “equally plausible”; only an idiot says that the “most plausible thing must be true!”), and that it would be really really nice to have more evidence w/ LR ≠ 1, we still have to do something. And we can and should use our best judgment about what the truth is, informed by all the “valid” evidence (LR ≠ 1) we can lay our hands on.
I think people can have justifiable beliefs about the impact (or lack thereof) of gun control laws & the death penalty on homicide rates!
They just shouldn’t abuse reason.
They do that when they insist that bad statistical proofs — simplistic ones ones that just toss out arbitrary bits of raw data; or arbitrarily complex yet grossly undertheorized ones like “y =b1*x1+ b2*x2 +b3*x3 … +b75*x34^3 + …” – “conclusively refute” or “demonstrably establish” blah blah blah.
And they do that and something even worse when they mischaracterize the best scientific evidence we do have.
WAIT …
Not true that National Research Council report “Deterrence and the Death Penalty” says “nothing” about D&W’s cross-country comparison of US & Canada. In fact, it says this:
Lesson: don’t forget to search for “Canadian” & not just “Canada”!
Sorry folks!
Frankly, I think this is a surprisingly weak enagagement with the D&W data. It’s disappointing, even.
Maybe in context it is okay: the NRC is right that a simple correlation like this can only be a starting point (remember my point about controlling for hockey & baseball).
But still, this data analysis, as simple as it is, is probative, especially given that multivariate regression to try to control for all those confounding “factors” wouldn’t be “informative” (to use the NRC panels’ words) due to how arbitrary and fragile the model specification would end up being.
Put it this way: if you just took this snippet of lanaguage out of the report, you might get the impression that the Panel is saying about these data the same thing that it says about multivariate regression studies (including Donohue & Wolfer’s!) on the death penalty: namely, that they should “not be used to inform deliberations requiring judgments about the effect of the death penalty on homicide.”
I don’t see how they could say that. The Panel concludes that MVR is just not a valid method for studying deterrence and the death penalty (also deterrence & gun control), & so necesssarily it is “uninformative.” But thre’s nothing invalid about just looking at the raw data.
The question is just how probative or informative it is. Figuring that out is a matter of judgment — the same way it is w/ any other piece of evidence on pretty much anything of consequence. Like I said, I’d give it a likelhood ratio of 0.2 — that is, 5x more consistent with “not deter” than with “deter.”
And like I said, when I say things like that, I’m not literally doing what I say; I’m conveying to you my attitude about how to think about how to think about empirical evidence. Part of that attitude is that it is wrong wrong wrong to say that just because evidence doesn’t conclusively, demonstrably irrefutably estalish something by ruling out every conceivable alterntive expalanation, etc., one should “ignore” it or assign it an LR = 1! Instead, think for yourself about how the evidence in question supports the inference and how much (which does necessarily involve thinking about alternative explanations for the evidence that would be consistent with the competing hypothesis).