Wisdom from Silver’s Signal & Noise, part 2: Climate change & the political perils of forecasting maturation

This is post 2 in my three part series on Silver’s Signal & Noise, which tied for first (with  Sharon Bertsch McGrayne’s The Theory That Would Not Die) in my “personal book of the year” contest (I’ve already mailed them both the quantity of gold bullion that I always award to the winner—I didn’t even divide it in half; or maybe I did, or possibly I even doubled or tripled it).

It turns out that Silver is not only amazingly good at statistical modeling & pretty decent at story telling. He also happens to be pretty wise (obviously this is a limited sample & I’ll update based on new information etc).

The nugget of wisdom I mined out of the book in the first post had to do with Silver’s idea that we should treat terrorist attacks a bit more like earthquakes.

This time I want to make a report on what Silver had to say about climate-change forecasting. One way to understand his assessment is that the practitioners of it are being punished for their methodological virtue.

Silver essentially structures the book around prototypes. There’s baseball, which is to forecasting what Saudi Arabia is to oil drilling. There are elections, another data-rich field but one that gets screwed up by a combination of bad traits in those who prognosticate (they are full of themselves) and those who are consuming their prognostications (too many of them want to be told only what they want to hear).

And earthquakes—can’t be forecast, but can still yield lots of info.

And economics–a bastion of bad statistics hygiene.

Then there’s meteorology, which is the archetype prototype of forecasting excellence because it is super hard and yet has made measureable progress (that’s much higher praise than “immeasurable,” in this context) due to the purity and discipline of its practitioners. (I’m eager to see who gets to play Richard Loft, the director of Technology Development at NCAR in the upcoming movie adaptation of Signal; I’m guessing Pierce Brosnan, unless he is cast as Silver himself).

In Silver’s account, climate forecasting is traveling the path of meteorology. The problem is that emulating the meteorologists obliges climate forecasters to become unwitting manufacturers of the ammunition being directed against them in the political flack storm surrounding climate change.

One of the things that meteorology forecasters did that makes them the superheroes of Signal was calibration. They not only made prodigious predictions but then revisited and retooled their models in light of how close they came to their targets, thereby progressively improving their aim.

When climate forecasters do this—as they must—they leave themselves wide open to guerilla attack by those seeking to repel the advance of science. The reason is that error is an inevitable and indeed vitally productive element of the Bayesian-evolutionary process that characterizes the maturation of valid forecasting.

Gaps between prediction and reality are not evidence of a deficiency in method. They are just evidence, information that is reprocessed as part of the method of generating increasingly precise and accurate probabilistic estimates.

This is a subtle point to get across even if one is trying to help someone to actually understand how science works. But for those who are trying to confuse, the foreseeable generation of incorrect predictions furnishes a steady supply of resources with which to harass and embarrass and discredit earnest scientists.

Silver recounts this dilemma in explicating the plight of James Hansen, whose forecasts from 30 and 25 years were in many respects impressively good but just as importantly instructively wrong. Ditto for the IPCC’s 1990 predictions.

Another thing that the superhero meteorologists did right was, in effect, theorize. They enriched their data with scientific knowledge that enabled them to do things like create amazing simulations of the dynamics they were trying to make predictive sense of. As a result, they got a lot further than they would have if they had used brute statistical force alone.

Climate forecasters are doing this too, and as a result necessarily enlarging the target that they offer for political sniping.  The reason is that theory-informed modeling of dynamic systems is hard work, the payoffs of which are unlikely to accumulate steadily in a linear fashion but rather to accrue in incremental breakthroughs punctuated by periods of nothing.

Indeed, those who travel this path might well seem to be make slower progress at least temporarily than those who settle for simpler, undertheorized number-crunching strategies, which make fewer assumptions and thus expose themselves to fewer sources of error, which tend to compound within dynamic models. Silver notes, for example, that some of Hansen’s earlier predictions—which were in the nature of simple multivariate regressions—in some respects outperformed some of his subsequent, dynamic-simulation driven ones.

Again, then, the virtuous forecaster will, precisely as a result of being virtuous, find him- or herself vulnerable to opportunistic hectoring, particularly by anti-science, lawyerly critics who will adroitly collect and construct number-crunching models that generated more conservative predictions and thereby outperformed the more theoretically dynamic ones over particular periods of time (including ones defined by happenstance or design to capitalize on inevitable and inevitably noisy short-term fluctuations in things like global temperatures).

Silver mentions the work of Scott Armstrong, a serious forecaster who nevertheless confines himself to simple number-crunching and consciously eschews the sort of theory-driven enrichment that was the signature of meteorology’s advancement. “I actually try not to learn a lot about climate change,” Armstrong, who is famous for his “no change” forecast with respect to global temperatures, boasts. “I am a forecasting guy” (Signal, p. 403).

“This book advises you to be wary of forecasters who say that science is not very important to their jobs,” Silver writes, just as it advises us to be skeptical toward “scientists who say that forecasting is not important to their[s] . . . . What distinguishes science, and what makes a forecast scientific, is that it is concerned with the objective world. What makes forecasts fail is when our concern only extends as far as the method, maxim, or model” (p. 403).

For Silver, the basic reason to “believe” in—and be plenty concerned about—climate change is the basic scientific fact, disputed by no one of any seriousness, that increasing concentrations of atmospheric CO2 (also not doubted by anyone) conduce to increasing global temperatures, which in turn have a significant impact on the environment. Forecasting is less a test of that than a vital tool to help us understand the consequences of this fact, and to gauge the efficacy (including costs and benefits) of potential responses.

Seems right to me. Indeed, seems wise.

* * * *

Okay, here’s something else that I feel I ought to say.

One reason I was actually pretty excited to get to the climate forecasting chapter was to verify an extremely critical review of the book (issued well before the release date of it) by Michael Mann, climate scientist of “hockey stick” fame.

Frankly, I find the gap between Mann’s depiction and the reality of what Silver said disturbing. You’d get the impression from reading Mann’s review that Silver is a “Chicago School” “free market fundamentalist” who dogmatically attacks the assumptions and methods of climate forecasters.

Just not so. I’m mean really really really untrue.

Mann figures very briefly at the end of the chapter, where Silver reports Mann’s reaction to what is in fact the chapter’s central theme—that climate forecasting is exposed to political perils precisely because those engaged in it are taking an uncompromisingly scientific approach.

Mann is obviously—understandably and justifiably!—frustrated and filled with anger.

He describes climate scientists themselves as being involved in a “street fight with these people”—i.e., the professional “skeptics” who hector and harass, distort and mislead (p. 409).

Of course, that’s a response that sees fighting as something climate scientists ought to be doing.

“It would be irresponsible for us as a community to not be speaking out,” Mann explains.

“Where you have to draw the line is to be very clear about where the uncertainties are,” he allows, but it would be a mistake to “have our statements so laden in uncertainty that no one even listens to what we’re saying.”

Silver doesn’t say this—indeed, had no reason to at the time he wrote the book—but I have to wonder whether Mann’s savage reaction to Silver is part of Mann’s “street fighting” posture, which apparently includes attacking even intellectually and emotionally sympathetic commentators whose excessive reflection on climate forecasting “uncertainty”  threatens to prevent the public from even “listen[ing] to what we’re saying.”

Mann is a great climate scientist. He is not a scientist of science communication.

For those who do study and reflect on science communication, whether simplifying things or dispensing with qualifications (not to mention outright effacing of complexity) will promote open-minded public engagement with climate science are matters characterized by uncertainties analogous to the ones that climate change forecasters deal with.

But I think one thing that admits of no uncertainty is that neither climate scientists nor scientists of science communication nor any other scientifically minded person should resort to simplification, effacement of complexity, and disregard for intellectual subtlety in describing the thoughtful reflections of a scholarly minded person who is trying to engage openly and candidly with complicated issues for the benefit of curious people.

That’s a moral issue, not an empirical one, and it goes to the nature of what the enterprise of scholarly discussion is all about.