Yesterday, I discussed how evolution “believers” and “nonbelievers” reacted to a cool evolution-science documentary. The data I described came from Study No. 1 of the Annenberg Public Policy Center/CCP “Evidence-based Science Filmmaking Initiative” (ESFI).
That data suggested that “belief” in evolution wasn’t nearly as important to engagement with the documentary (Your Inner Fish, an award-winning film produced by ESFI collaborator Tangled Bank Studios) as was science curiosity.
Today I’ll say a bit more about how we measured science curiosity.
Developing a valid and reliable science curiosity scale was one of the principal aims of Study No. 1. As conceptualized here, science curiosity is not a simple transient state (Loewenstein 1999) but instead a general disposition, variable in intensity across persons, that reflects the motivation to seek out and consume scientific information for personal pleasure.
Obviously, a measure of this disposition would furnish science journalists, science filmmakers, and related science-communication professionals with a useful tool for perfecting the appeal of their work to those individuals who value it the most. But it could also make myriad other contributions to the advancement of knowledge.
A valid science curiosity measure could be used to improve science education, for example, by facilitating investigation of the forms of pedagogy most likely to promote its development and harness it to promote learning (Blalock, Lichtenstin, Owen & Pruski 2008). Those who study the science of science communication (Fischhoff & Scheufele 2014; Kahan 2015) could also use a science curiosity measure to deepen their understanding of how public interest in science shapes the responsiveness of democratically accountable institutions to policy-relevant evidence.
Indeed, the benefits of measuring science curiosity are so numerous and so substantial that it would be natural to assume researchers must have created such a measure long ago. But the simple truth is that they have not.
“Science interest” measures abound. But every serious attempt to assess their performance has concluded that they are psychometrically weak and, more important, not genuinely predictive of what they are supposed to be assessing—namely, the disposition to seek out and consume scientific information for personal satisfaction (Blalock et al 2008; Osborne, Collins & Simons 2003).
ESFI’s “Science Curiosity Scale 1.0” (SCS_1.0) is an initial step toward filling this gap in the study of science communication. The items it comprises, and the process used to select (and combine) them, self-consciously address the defects in existing scales.
One of these is the excessive reliance on self-report measures. Existing scales relentlessly interrogate the respondents on the single topic of their own attraction to or aversion toward information on scientific discovery: “I am curious about the world in which we live,” “I find it boring to hear about new ideas,” “I get bored when watching science programs on TV,” etc. Items like these are well-known to elicit responses that exaggerate respondents’ possession of desirable traits or attributes.
To counteract this dynamic, SCS_1.0 disguises its objectives by presenting itself as a general “marketing” survey.
Individual self-report items relating specifically to science were thus embedded in discrete blocks or modules, each consisting of ten or more items relating to an array of “topics” that “some people are interested in, and some people are not.” Items were presented in random order, each on a separate screen.
There was thus no reason for subjects to suspect that their motivation to learn about science was of particular interest, nor any opportunity for them to adjust the responses across items in a manner that overstated their interest in it. A similar strategy was used to gather information on behavior reflecting such an interest, including visits to science museums, attendance at public science lectures, and the reading of books on scientific discovery.
SCS_1.0 also featured an objective performance measure.
Well into the survey, subjects were advised that we were interested in their reactions to a news story “of interest” to them. In order to assure that the story was one that in fact matched their interests, they were furnished with discrete news story sets, the shared subject matter of which was be identified by a header and reinforced by the individual story headlines and graphics. One set consisted of science stories; the others ones on popular entertainment, on sports, and on financial news.
Subjects, we anticipated, were likely to find the prospect of reading a story and answering questions about it burdensome. Accordingly, the selection of the science set rather than one of the others would be a valid indicator of genuine science interest . Responses to this task were then used to validate the self-reported interest items to help furnish assurance the genuineness of the latter.
When combined, the items displayed the requisite psychometric properties of a valid and reliable scale. Their unidimensional covariance structure warranted the inference that they were measuring the same latent disposition. Formed with item response theory, the composite scale weighted particular items in relation to the level of the disposition that responses to them evinced. The result was an index—SCS_1.0—that reflected a high degree of measurement precision along the entire population distribution of that trait (Embretson & Paul 2000).
Finally and most importantly, SCS_1.0 was behaviorally validated.
As detailed in ESFI Study Report No. 1, subjects were instructed to watch a 10-minute clip from the science documentary Your Inner Fish. SCS_1.0 strongly predicted engagement with the clip as reflected not only in self-reported interest but also in objective measures such as duration of viewing time and subjects’ election (or not) to be furnished free access to the documentary as a whole.
SCS_1.0 is by no means understood to be an ideal science curiosity measure. Additional testing is necessary, both to assure the robustness of the scale and to refine its powers to discern the motivation to seek out and consume science information for pleasure.
Moreover, SCS_1.0 was self-consciously designed to assess this disposition in adult members of the public; variants would be appropriate for specialized populations including elementary or secondary school students.
But what SCS_1.0 does do, we believe, is initiate a process that there’s every reason to believe will generate measures of genuine value to researchers interested in assessing science curiosity in the general public and in specialized subpopulations. The researchers associated with CCP’s ESFI and other evidence-based science communication initiatives are eager to participate in that process. But they are also eager to stimulate others to participate in it either by building on and extending SCS_1.0 or by developing alternatives that genuinely predict behavior that manifests the motivation to seek out and consume scientific information.
Existing “science interest” measures just don’t do that. SCS_1.0 shows that it is possible to do much better.
References
Besley, J.C. The state of public opinion research on attitudes and understanding of science and technology. Bulletin of Science, Technology & Society, 0270467613496723 (2013).
Blalock, C.L., Lichtenstein, M.J., Owen, S., Pruski, L., Marshall, C. & Toepperwein, M. In Pursuit of Validity: A comprehensive review of science attitude instruments 1935–2005. International Journal of Science Education 30, 961-977 (2008).
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, N.J.: L. Erlbaum Associates.
Fischhoff, B. & Scheufele, D.A. The science of science communication. Proceedings of the National Academy of Sciences 110, 14031-14032 (2013).
Kahan, D.M. What is the “science of science communication”? J. Sci. Comm, 14, 1-12 (2015).
Loewenstein, G. The psychology of curiosity: A review and reinterpretation. Psychological Bulletin 116, 75 (1994).
National Science Foundation. Science and Engineering Indicators, 2010 (National Science Foundation, Arlington, Va., 2014).
Osborne, J., Simon, S. & Collins, S. Attitudes towards science: A review of the literature and its implications. International journal of science education 25, 1049-1079 (2003).
Thomas G Reio Jr, Joseph M Petrosko, Albert K Wiswell & Juthamas Thongsukmag, The Measurement and Conceptualization of Curiosity, 167 The Journal of Genetic Psychology 117-135 (2006).
Shtulman, A. Qualitative differences between naïve and scientific theories of evolution. Cognitive Psychology 52, 170-194 (2006).