My paper “Climate Science Communication and the Measurement Problem” features a “climate science literacy” (CSL) test.
I’ve posted bits & pieces of the paper & described some of the data it contains. But I really haven’t discussed in the blog what I regard as most important thing about the CSL results.
This has to do with the relationship between the CSL scores, critical reasoning, and independent or non-conformist thinking. I’ll say something—I doubt the last thing—about that now!
1. The point of the exercise: disentangling knowledge from identity. I’ll start with the basic point of the CSL—or really the basic point of the study that featured it and the Measurement Problem paper.
Obviously (to whom? the 14 billion regular readers of this blog!), I am not persuaded that conflict over culturally disputed risks in general and climate change in particular originates in public misunderstandings of the science or the weight of scientific opinion on those issues.
That gets things completely backwards, in fact: It is precisely because there is cultural conflict that there is so much public confusion about what the best available evidence is on the small (it is small) class of issues that display this weird, pathological profile.
Given the stake they have in protecting their status in these groups, people can be expected to attend to evidence—including evidence about the “weight of scientific opinion” (“scientific consensus”)—in a manner that reliably connects their beliefs to the position that prevails in their identity-defining groups.
But there are two ways (at least) to understand the effect of this sort of identity-protective reasoning. In one, the motivated assimilation of information to the positions that predominate in their affinity groups generates widespread confusion over what “position” is supported by the best available scientific evidence.
Call this the “unitary conception” of the science communication problem.
Under the alternative “dualist conception,” “positions” on societal risk issues become bifurcated. They are known to be both badges of group membership and matters open to scientific investigation.
Applying their reason, individuals will form accurate comprehensions of both positions.
Which they will act on or express, however, depends on what sort of “knowledge transaction” they are in. If individuals are in a transaction where their success depends on forming and acting on the position that accurately expresses who they are, then that “position” is the one that will govern the manner in which they process and use information.
If, in contrast, they are in a “knowledge transaction” where their success depends on forming and acting on the positions that are supported by the best available evidence, then that is the “position” that will orient their reasoning.
For most people, most of the time, getting the “identity-expressive position” right will matter most. Whereas people have a tremendous stake in their standing in cultural affinity groups, their personal behavior has no meaningful impact on the danger that climate change or other societal risks pose to them or others they care about.
But still, every one of them does have an entirely separate understanding of the “best-available-evidence” position. We don’t see that—we see only cultural polarization on an issue like climate change—because politics confronts them with “identity-expressive” knowledge transactions only.
So too do valid methods of public opinion study (observational and experimental) geared to modeling the dynamics of cultural conflict over climate science.
Politics and valid studies both assess citizens’ climate-science knowledge with questions that measure who they are, whose side they are on.
But if we could form a reliable and valid measure that disentangles what people know from who they are, we would then see that these are entirely different things, entirely independent objects of their reasoning.
Or so says the “dualist” view of the science communication probolem.
The aim of the “climate science literacy” or CSL measure that I constructed was to see if it was possible to achieve exactly this kind of disentanglement of knowledge and identity on climate change.
I refer to the CSL measure, in the paper and in this blog, as a “proto-” climate-science literacy instrument. That’s because it’s only a step toward developing a fully satisfactory instrument for measuring what people know about climate science.
Indeed, the idea that there could be an instrument of that sort is absurd. There would have to be a variety, geared to assessing the sort of knowledge that individuals in various settings and roles (“high school student,” “business decisionmaker,” “policymaker,” “citizen” etc.) have to have.
But if the “dualist” conception of the science communication problem is correct, then in any such setting, a CSL, to be valid, would have to be designed to measure what people know and not who they are.
Seeing whether that could be done was the mission of my CSL measure. In that respect, there is nothing “proto-” about it.
2. The strategy
The strategy I followed to construct a CSL of this sort is discussed, of course, in the paper. But that strategy consisted of basically two things.
The first was an effort to create a set of items that would avoid equating “climate science literacy” with an affective orientation toward climate change.
For the most part, that’s what perceptions of societal risks are: feelings with a particular valence and intensity. As such, these affective orientations are more likely to shape understandings of information than be shaped by them.
The affective orientation toward climate change expresses who people are as members of opposing cultural groups engaged in a persistent and ugly form of status competition. If we ask “climate science literacy” questions the answers to which clearly correspond to the ones people use to express their group identities, their answers will tell us that—who they are—and not necessarily what they know.
To avoid this confound, I tried to select a set of items the correct responses to which were balanced with respect to the affective attitudes of “concern” and “skepticism.” Scoring high on the test, then, would be possible only for those whose answers were not “entangled” in the sort of affective reaction that defines who they are, culturally speaking.
Second, I used a semantic device that has proven successful in disentangling identity and knowledge in measuring people’s positions on evolution.
As I’ve discussed in this blog (and as I illustrate with data in the paper), the true-false question “humans evolved from another species of animal” doesn’t measure understanding of evolution or science comprehension generally. Rather it measures a form of identity indicated by religiosity.
But if one simply prefaces the statement “According to the theory of evolution,” the question elicits responses that don’t vary based on respondents’ religiosity. Because it doesn’t force them to renounce who they are, the reworded question makes it possible for religious respondents to indicate what they know about the position of science. (The question is then revealed, too, to be far too easy to tell us anything interesting about how well the person answering it comprehends science.)
I thus used this same device in constructing the CSL items. I either prefaced true-false ones with the phrase “Climate scientists believe . . .” or used some other form of wording that clearly separated “knowledge” from “belief.”
3. The “results”
The results strongly supported the “dualistic” position—i.e., that what people know about climate change is unrelated to their “belief in” human-caused climate change. Their position on that measures who they are in the same manner as items involving their political outlooks generally.
In this way, it becomes possible to see that the cultural polarization that attends climate change is also not a consequence of the effect that cultural cognition has on people’s comprehension of climate science.
It is a consequence of the question that the “climate change” poses to ordinary citizens.
Democratic politics is one of the “knowledge transactions” that measures who one is, whose side one is on, not what one knows about the weight of the best scientific evidence.
People on both sides of the issue, it turns out, don’t know very much at all about climate science.
But if democratic politics were asking them “what they know,” the answer would be a bipartisan chorus of, “We are in deep shit.”
So climate communicators should be working on changing the meaning of the question—on creating conditions that, like the reworded evolution question and related classroom instructional techniques in that setting, make it possible for citizens to express what they know without renouncing who they are.
If you want to see how that’s done, book yourself a flight down to SE Florida. Right now.
4. The “holy shit!” part: the vindication of reason as a source of independent thinking
Now, finally, I get to what for me is the most gratifying part: the vindication of critical reasoning.
The CSL measured featured in the paper is positively correlated with science comprehension in both “liberal Democrats” and “conservative Republicans”!
Why is this so amazing?
As the 14 billion regular readers of this blog know, a signature of the pathology that has infected public discourse on climate change is the impact of science comprehension in magnifying polarization.
The individuals whose science comprehension and critical reasoning dispositions are most acute are the most polarized.
Experiments show that individuals high in the dispositions measured by science literacy batteries, the Cognitive Reflection Test, the Numeracy scale and the like use their reasoning proficiency to selective conform their assessment of evidence to the position that predominates in their group.
Polarization over climate change is not a sign that people in our society lack science comprehension.
It is proof how hostile the putrid spectacle of cultural status competition is to the value our society should be getting form the science intelligence it manages to impart in its citizens.
As the 14 billion regular readers know, too, this doesn’t amuse me. On the contrary, it fills me with despair.
I was heartened in a simple “methods” sense that the CSL had the indicated relationship with science comprehension. That the two rise in tandem helps to validate the CSL as a measure of what people know, and to corroborate the conclusion that “what do you believe about climate change?,” on which polarization increases as people become more science comprehending, measures nothing other than who they are, what side they are on.
But on an emotional level, I was much more than simply heartened.
I was elated to see the vitality of reason and critical thinking as a source of independent thinking and open-mindedness—to be assured that in fact this aspect of our intelligence hadn’t been annihilated by the sickness of cultural status competition, if it ever existed in the first place.
Remember, the CSL was deliberately designed to disentangle knowledge from identity.
One of the central devices used to achieve this effect was to balance the items so that respondents’ affective orientation toward climate change—concern or skepticism—would be uncorrelated with their CSL scores.
Thus, to do well on the CSL, individuals had to answer the questions independently of their affective orientations, and hence with the source of them: their cultural identities.
The people who did that the most successfully were those who scored the highest in science comprehension, a disposition that features critical reasoning skills like cognitive reflection and numeracy, as well as substantive science knowledge.
More later on this, but look: here are your Ludwicks!
This is what happens when one measures what people know.
But this is how it can be, too, in our political life.
If we can just make democratic politics into the sort of “knowledge-assessment transaction” that doesn’t force people to choose between expressing what they know and expressing who they are.