The psychological  side of

Hempel's paradox of confirmation

CRAIG R. M. McKENZIE and LAURIE A. MIKKELSEN

University of California, San Diego, California

People often test hypotheses about two variables (X and Y), each with two levels (e.g., Xl and X2). When testing "If Xl, then Yl, observing the conjunction of Xl and Y1 is overwhelmingly perceived as more supportive than observing the conjunction of X2 and Y2, although both observations support the hypothesis. Normatively, the X2&Y2 observation provides stronger support than the Xl&Yl observa­ tion if the former is rarer. Because participants in laboratory settings typically test hypotheses they are unfamiliar with, previous research has not examined whether participants are sensitive to the rarity of observations. The experiment reported here showed that participants were sensitive to rarity, even judging a rare X2&Y2 observation more supportive than a common Xl&Yl observation under certain conditions. Furthermore, participants' default strategy ofjudgingXl&Yl observations more informa­ tive might be generally adaptive because hypotheses usually regard rare events.

A fundamental issue in the study of human inference is what, psychologically speaking, constitutes confirma­ tory evidence for a hypothesis. In 1945, philosopher Carl Hempel noted the following paradox regarding confir­ matory evidence. Assume that the hypothesis of interest is "All ravens are black." This statement can be rewritten as "If something is a raven, then it is black," or Raven Black. Clearly, observing a black raven would count as confirming evidence. Similarly,  if the hypothesis  were "If  something  is  not black,  then  it  is not a  raven,"  or

-Black -Raven, observing a nonblack nonraven (e.g., a white shoe or a yellow pencil) would clearly be con­ firming evidence. Because these two hypotheses are log­ ically equivalent (one is the contrapositive of the other), any evidence that confirms one must confirm the other. It follows, then, that observing a nonblack nonraven con­ firms Raven       Black. Thus, one could apparently con­ firm the hypothesis about the color of ravens by sitting in one's office and never even observing a raven. Most peo­ ple find this highly counterintuitive; hence, the paradox. Other philosophers have pointed out that the paradox can be resolved if one conceives of confirmation as a matter of degree rather than all-or-none. Although black

ravens and nonblack nonravens both confirm  Raven Black, they do not do so equally strongly. From a Bayesian perspective, confirming evidence supports a hypothesis to the extent that it is rare, or surprising. Be­ cause nonblack things and nonravens are both common, observing a nonblack nonraven would not be unusual and 

HIV- and psychotic, and (4) those who were HIV- and mentally healthy. Half of these participants were told that the hypothesis to be tested was "If a person is HIV+, then he/she is psychotic," and half were told to test the hypothesis "!fa person is mentally healthy, then he/she is HIV - ." They were then informed that, of the first two people observed, one was HIV+ and psychotic and the other was mentally  healthy  and  HIV -  , and  they were asked to select

which of the two observations provided stronger support for the hy­ pothesis. Before responding, the concrete + statistics group read: "(I) A large majority of the population is HIV - and a small mi­ nority is HIV+. (2) A large majority of the population is mentally healthy and a small minority is psychotic." For the two concrete groups, the HIV+ /psychotic individual is rare, and the mentally healthy/HIV- individual is common. Similarly, the HIV+/psy­ chotic  hypothesis and  the mentally  healthy/HIV-  hypothesis are

the "rare hypothesis" and "common hypothesis," respectively.

On the final page, the participants justified their choice of which observation provided stronger support of the hypothesis. They were instructed not to change the choice they had already made.

To summarize, the experiment was a completely between­ participants 2 (context: abstract vs. concrete) X 2 (statistical infor­ mation: yes vs. no) X 2 (hypothesis tested: common vs. rare) de­ sign. There were between 34 and 36 participants in each condition. They chose either the common or rare observation as more sup­ portive of the hypothesis tested and explained why.

RESULTS

Figure I shows the percentage of participants who se­ lected the rare observation as providing stronger support

as a function of group (abstract, abstract + statistics, concrete, and concrete + statistics) and hypothesis tested (common and rare). The abstract participants (far left) serve as a control group, showing how often the ob­

servation (not) named in the hypothesis is chosen when no information is available regarding the rarity of the ob­ servations. The low value for the common hypothesis re­ flects the fact that few participants in the abstract group selected the rare observation as more supportive when testing the common hypothesis, and the high value for the rare hypothesis shows that many chose the rare ob­ servation when testing the rare hypothesis. These results replicate those of many previous studies and  illustrate the tendency to perceive as more informative the obser­ vation named in the hypothesis.

For the rest of the groups, who were told the relevant statistical information and/or were assumed to have al­ ready known it, the normative response was to choose the rare observation as more supportive, regardless of the hy­ pothesis tested. Thus, to the extent that both columns in Figure l are large for a given group, the better the group's performance. It is easy to see that, across the four groups, there was little variation in selecting the rare observation when the rare hypothesis was tested (hatched columns). All groups tended to select the rare observation when test­ ing the rare hypothesis, though the two concrete groups did so slightly  more often  than the two abstract groups.

Figure 1. Percentage of participants selecting the rare (rather than common) observation as a function of group (abstract, abstract + statistics, concrete, and concrete + statistics) and hypothesis tested (common and rare). Standard error bars are shown.

Choosing the rare observation when testing the rare hypothesis is consistent with both preferring the named observation and preferring the normatively more sup­ portive observation (though see the analysis of the par­ ticipants' reasons for their choices below). More infor­ mative was the participants' behavior when testing the common hypothesis. Figure l reveals that the percentage of rare observation choices when testing the common hy­ pothesis varied considerably across the groups. Relative

to the abstract group, the abstract + statistics group was

more likely to select the rare observation, showing that the statistical information led to more normative re­ sponses. The concrete group also selected the rare ob­ servation more often than the abstract group, which in­ dicates that  the participants'  real-world  knowledge was

helpful. Of interest is that the concrete group performed as well as or better than  the abstract  + statistics group,

demonstrating that implicit real-world  knowledge af­

fected behavior at least as much as the explicit statistical information. The concrete + statistics group was by far the most  likely to select the rare observation  when test­

ing the common hypothesis. Indeed, this group chose the rare observation twice as often as the common observa­ tion  when  testing  the  common  hypothesis.  As can be

seen, the concrete  + statistics group  was about equally

likely to select the rare observation regardless of hy­ pothesis tested.

A 2 (context: abstract vs. concrete) X 2 (statistical in­ formation: yes vs. no) X 2 (hypothesis tested: common vs. rare) X 2 (response: common vs. rare observation) log-linear analysis corroborated the patterns noted above. The participants were more likely to select the rare observation when testing the rare hypothesis than when testing the common hypothesis (p < .00 l ); they were also more likely to select the rare observation when presented with the concrete scenario than when pre­ sented with the abstract scenario (p = .002). In addition, they were marginally significantly more likely to choose the rare observation when presented with statistical in­ formation (p = .056). Furthermore, statistical informa­ tion affected the participants testing the common hy­ pothesis more than those testing the rare hypothesis (p =

.043). Separate analyses confirmed  that the proportion of participants choosing the rare observation when test­ ing the rare versus common hypothesis was different for each of the abstract,  abstract  +  statistics,  and concrete

groups (ps < .02, Fisher's exact tests), but not the con­ crete + statistics group (p  = .43). In other words, only the  concrete  + statistics  participants  were  not signifi­

cantly influenced by the hypothesis tested when choos­ ing an observation.

The participants' reasons for their choices were also analyzed. Reasons were categorized as "correct" if the participant  both  (1)  chose  the  rare  observation and

(2) mentioned the relative commonality of one level of either variable (e.g., being HIV+) or one of the obser­ vations (e.g., being HIV+ and psychotic). Reasons were categorized as "naming" if the  participant  mentioned that the observation named in the hypothesis was more relevant. Any reason not falling under these two cate­ gories was categorized as "other." If more than one rea­ son was given, any mention of rarity combined with choosing the rare observation was categorized as "cor­ rect," and reasons were categorized as "other" only if neither correct nor naming reason(s) were stated. Thus, each participant's reasons fell into exactly one of the three categories.

The results are shown in Table 1 by condition. Not sur­ prisingly, participants in the abstract group never gave a "correct" reason for their choices because they had no information regarding rarity. More interesting is this group's percentage of"naming" reasons,  which serves as a useful benchmark: About three fourths of these partic­ ipants testing either hypothesis justified their choice by stating that the observation named in the hypothesis was more relevant. (The most common "other" reason for both the abstract group and the concrete group was stat­

ing that the unnamed observation must be observed to confirm the hypothesis.) The abstract + statistics group was more likely to provide "correct" reasons when  test­

ing either hypothesis, and more than half did so when testing the rare hypothesis. Thus, though the abstract and abstract  + statistics  groups chose  the  rare observation

about equally often when testing the rare hypothesis (see Figure 1), they apparently did so for different reasons. (The most  frequent "other" reason  for both  the abstract

+ statistics group and the concrete + statistics group was

to mention rarity but select the common observation.) Table 1 also shows that the concrete group stated cor­ rect justifications more often when testing the rare hy­ pothesis than when testing the common hypothesis, but these  responses  constituted  only  a  minority  in  either 

case. Given that this group chose the rare observation at least as often as the abstract + statistics group (see Fig­ ure I), it is interesting that they were less likely to pro­ vide a correct justification. We compared the concrete and abstract + statistics groups in terms of the number of correct versus naming responses (ignoring other re­ sponses). The concrete group was less likely to provide a correct reason when testing both the common hypoth­ esis and the rare hypothesis (ps < .03, Fisher's exact tests). Finally, Table 1 shows that more than half of the participants testing either hypothesis in the concrete + statistics group reported the correct reason for their choices. Thus, regardless of the hypothesis tested, a ma­ jority of this group chose the rare observation and re­ ported the correct reason for doing so.

DISCUSSION

The results of this experiment corroborate earlier find­ ings that observations mentioned in the hypothesis are generally considered more valuable. The results also ex­ tend previous findings by demonstrating that partici­ pants' knowledge regarding the rarity of observations can reduce, perhaps even overcome, this tendency. The rare observation was more often correctly chosen as more supportive when (1) the hypotheses were concrete rather than abstract, which allowed the participants to exploit their real-world knowledge about rarity, and (2) statisti­ cal information about rarity was provided. The combina­ tion of a concrete hypothesis and statistical "reminder" led most participants to select the rare observation even when testing the common hypothesis. The participants in this latter condition were about equally likely to select the rare observation regardless of the hypothesis tested.

It was also found that participants' knowledge-both implicit real-world knowledge and explicit statistical in­ formation provided to them-about rarity led to more correct justifications of choices. Results for the concrete group were most interesting: This group selected the rare observation at least as often as the abstract + statistics group when testing either hypothesis, but they were less likely to provide a correct reason why. It seems reason­ able to assume that the concrete group's choices were af­ fected by real-world knowledge regarding rarity. Thus, the participants in this group could apparently express their real-world knowledge through their choices, but not through verbalization (Nisbett & Wilson, 1977). This is especially intriguing because the concrete scenario rep­ resents the most realistic situation. That is, people typi­ cally make inferences about variables they are familiar with, but they are not usually handed the relevant statis­ tical information. The mechanism(s) governing hypoth­ esis-testing behavior under these circumstances might operate at a level that goes unnoticed by most people. The fact that rarity affected judgments illustrates that using familiar hypotheses enables participants to exploit an aspect of the hypothesis-testing situation that is un­ available  when abstract or  unfamiliar  hypotheses are

used-which is the norm in the laboratory. Klayman and Ha (1987) speculated that participants might prefer cases named in the hypothesis when presented with unfamiliar tasks but use more sophisticated strategies in more fa­ miliar territory. The present study provides empirical support for this notion. Generalizing findings based on unfamiliar hypothesis-testing tasks might mislead us as to people's ability in their natural environment.

Two concerns about our experiment could be raised. First, the hypotheses we asked the participants to test were different from the Raven Black example in that the raven hypothesis is probably more likely to be viewed as deterministic. That is, observing a nonblack raven might be more likely to be seen as conclusively falsifying the raven hypothesis than, for example, observing a men­ tally healthy/HIV+ person would be seen as conclu­ sively falsifying the Mentally Healthy HIV - hypoth­ esis. However, as we mentioned in note 1, the ordering of Xl&YI andX2&Y2 observations in terms of their infor­ mativeness remains the same regardless of whether the hypothesis tested is deterministic or probabilistic.

Second, when analyzing and interpreting our data, we treated "context" as a variable, although there are un­ doubtedly many factors that differ between the abstract and concrete scenarios. For example, the statistical in­ formation presented to the participants might have been interpreted differently in the two scenarios. Nonetheless, our conclusions are similar whether we treat the two con­ text conditions as two levels of a variable or as separate experiments. Relatedly, all participants were told that the four categories of observations were mutually exclusive and exhaustive, but this information might have been re­ inforced by the participants' real-world knowledge in the concrete, but not the abstract, conditions. For instance, it might have been easy to understand that a person must be either HIV - or HIV+, and not both, but  perhaps  this was less clear for the fictitious personality type X versus Y categories. There was, however, no evidence of any such doubt or confusion in the participants' reported rea­ sons for their choices.

Our findings indicate that participants are sensitive to the rarity of data, but there is still the question of why, when presented with unfamiliar hypotheses, they tend to judge the confirming instance named in the hypothesis more informative than the confirming instance not named. Why is this the default strategy? We speculate that it is because hypotheses tend to be formed about rare events (see also Klayman & Ha, 1987). It is rare, not common, events that we seek to explain or predict (Ein­ horn & Hogarth, 1986; Mackie, 1974). Hypotheses are generated about the factors leading to being HIV+, not HIV - , about what causes plane crashes, not normal flights, about what leads to success, not mediocrity, and so on. Rare events are salient and "demand" explanation. If hypotheses  tend to be framed  in terms of rare events, a confirming observation named  in the hypothesis  will be more informative than a confirming observation not named.  In the absence  of information  indicating otherwise 

, behaving as though the named observation is rare might be generally adaptive.

Recent analyses ofWason's (1966, 1968) selection (or four-card) task have led to similar conclusions. In this task, participants are given a rule to test of the form "If  P, then Q" and are presented with four cards: P, -P, Q, and -Q. On the back of the first two cards is either Q or

-Q, and on the back of the last two is either P or -P. For

example, participants might test the rule "If there is a vowel on one side of the card, then there is an even num­ ber on the other side," with the four cards showing A, K, 2, and 7. Each card has a letter on one side and a number on the other. Which cards are necessary to turn over in order to see if the rule is true or false? According  to one

interpretation of the rule, propositional logic dictates that the P and -Q cards should be turned over (A and 7 in the example). However, a common finding is that par­ ticipants want to turn over the cards mentioned in the rule: the P and Q cards (e.g., Wason, 1966, 1968). Oaks­

ford and Chater (1994; see also Nickerson, 1996) have argued that, from a Bayesian perspective, the P and Q cards are the most informative-if one assumes  that P and Qare rare relative to -P and-Q. As predicted by this account, participants presented with a reduced array se­ lection task, where only the Q and -Q cards are present,

were more likely to select the -Q card as the commonal­

ity of Q was increased (Oaksford, Chater, Grainger, & Larkin, 1997). Oaksford and Chater's (1994) account of the selection task meshes well with our account of hy­ pothesis testing: When testing "If P, then Q," partici­ pants generally  behave as  though P and Q are rare, and

information to the contrary changes behavior in the ap­ propriate direction.2

Another related and highly studied task is covariation assessment. In a typical task, there are two variables, ei­ ther present or absent, resulting in four types of obser­ vation. Participants are asked to assess the relation be­ tween the two variables given the frequencies of the four types of observation. Both the joint presence and  the joint absence of the variables are confirming  evidence for a positive relation between the variables, but the for­ mer observation has a much larger influence on judg­ ments (e.g., Kao & Wasserman, 1993; Lipe, 1990; Mc­ Kenzie, 1994; Wasserman, Dorner, & Kao, 1990). Thus, as with hypothesis testing, the confirming observation named in the relation (or hypothesis) to be tested has a bigger impact on behavior than the confirming observa­ tion not named. However, also as with hypothesis test­ ing, covariation tasks tend to use variables that are unfa­ miliar or abstract, conditions that the present study shows lead to a preference for the named observation. Further­ more, the apparent default preference for the named con­ junction might be due to the fact that a variable's pres­ ence is generally rarer than its absence. For example, when assessing the relation between a symptom and a disease, most people will not have the  symptom  and most people will not have the disease, making the prob­ ability of having both much lower than the probability of

having neither. Participants might consider the joint presence of variables to be more informative than their joint absence because, outside the laboratory, the former observation is typically rarer-and, therefore, norma­ tively more informative, at least from a Bayesian per­ spective (see also Anderson,  1990, pp. 149-160).

In sum, participants' tendency to deem as more infor­ mative confirming observations named rather than not named in the hypothesis can be reduced, maybe even eliminated, when participants know that the unnamed observation is rare. The norm in the laboratory, however, is to use hypotheses that participants are unfamiliar with, which precludes their ability to exploit such knowledge. Furthermore, that participants are sensitive to the rarity of data is consistent with our speculation that, outside the laboratory, named confirming observations are typically rarer than unnamed confirming  observations,  making the former generally more informative. Thus, partici­ pants' default hypothesis-testing strategy in the labora­ tory appears generally adaptive as well as adaptable (Klayman & Brown, 1993) in that it changes appropri­ ately when normatively relevant information is available.

McKENZIE, C. R. M. (1998). Taking into account the strength of an al­ ternative hypothesis. Journal of Experimental Psychology: Learning, Memory, & Cognition, 24, 771-792.

NICKERSON, R. S. (1996). Hempel's paradox and Wason's selection task: Logical and psychological puzzles of confirmation. Thinking & Rea­ soning, 2, 1-31.

NISBETT, R. E., & WILSON, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.

OAKSFORD, M., & CHATER, N. (1994). A rational analysis of the selec­ tion task as optimal data selection. Psychological Review, 101, 608- 631.

OAKSFORD, M., CHATER, N., GRAINGER, 8., & LARKIN, J. (1997). Opti­

mal data selection in the reduced array selection task (RAST). Jour­ nal of Experimental Psychology: Learning, Memory, & Cognition, 23, 441-458.

WASON, P. C. (1960). On the failure to eliminate hypotheses in a con­ ceptual task. Quarterly Journal of Experimental Psychology, 12, I 29- 140.

WASON, P. C. (1966). Reasoning. In B. M. Foss (Ed.), New horizons in psychology (pp. 135-161). Harmondsworth, UK.: Penguin.

WASON, P. C. (1968). Reasoning about a rule. Quarter(v Journal of Ex­ perimental  Psychology, 20, 273-281.

WASSERMAN, E. A., DORNER, W. w., & KAO, S.-F. (1990). Contributions

of specific cell information to judgments of interevent contingency. Journal of Experimental Psychology: Learning, Memory, & Cogni­ tion, 16, 509-521.

NOTES

I. From a Bayesian viewpoint, data, D, support a hypothesis, HI, over its alternative, H2, to the extent that the likelihood ratio,p(DIHI )/ p(DIH2), is greater than I. Let HI be Raven Black, and let H2 be that the properties of being a raven and of being black are independent. Assume that the unconditional probability that an object (randomly sampled) is black is .0 I and that the unconditional probability that an object is a raven is also .01. If being black and being a raven are inde­ pendent (H2), then the probability of an object being both black and a raven is .012, or .0001, which isp(black raven I H2). Similarly, given the independence hypothesis, the probability of observing a nonblack non­ raven, p(nonblack  nonraven IH2),  is .992, or  .9801. If  the  Raven Black hypothesis (HI) is true, then the probability of observing a black raven, p(black  raven IHI), is .0I, because all  ravens will  be black. Again, because there are no nonblack ravens, all nonblack things must be nonravens, so p(nonblack nonraven IHI)= .99. Now we can compare likelihood ratios for the black raven observation,

p(black ravenlHl)/p(black ravenjH2) = .01 /.0001 = 100, and the nonblack nonraven observation,

p(nonblack nonraven I HI)/p(nonblack nonraven I  H2)   = .99/.9801

= 1.01.

In this example, the likelihood ratio for the black raven is 99 times greater than that for the nonblack nonraven. Note also that the latter likelihood ratio barely exceeds  I,  showing that the nonblack nonraven is virtually uninformative.

When testing "If XI, then YI," anXl&YI observation is more infor­ mative than an X2&Y2 observation whenever p(XI) <  I  -  p( YI). This is true even when the hypothesized relation between  X and Y is proba-

bilistic, rather than deterministic, as in the above example. To model the probabilistic case, one can use the phi coefficient to represent an imperfect relationship between X and Y (e.g., HI could be$= .5). Fur­ thermore, the alternative hypothesis need not be independence (e.g., H2 could be$ = . I). The result appears to be very general.

2. Our analysis does, however, differ from Oaksford and Chater's (1994) in an interesting way. Consider Raven Black as a rule to be tested in a selection task. The four cards would be raven (R), not a raven (-R), black (B), and not black (-B). Analyses of the selection task­ both  logical  and Bayesian-usually claim that turning over the -R card is uninformative (e.g., Oaksford & Chater, 1994; Wason, 1966, 1968). However, our analysis indicates otherwise: Turning over the -R card could reveal -R&-B, which we showed is informative in note I above. Indeed, observing -R&B is also informative, so we claim that turning over the -R card has informational value, at least when the marginal probabilities, p(R) and p(B}, are not different under the competing hy­ potheses (see also Nickerson, 1996).

Our claims in this note do not depend on R and B being rare, but, for the sake of consistency and concreteness, assume  as  in  note  I  that p(R) = .01,p(-R) = .99,p(B) = .01, andp(-B) = .99. If the  rule  is true (HI), then there should be no instances of R&-B. Let H2 be that the properties R and B are independent.  Because  the marginals are fixed and p(R) = p(B), HI entails not only that there are no instances of R&-B but that there are no instances of -R&B. if all ravens are black, and the number of ravens equals the number of black things, then nothing other than ravens can be black. Observing an instance of -R&B is informa­ tionally equivalent to observing an instance of R&-B. When p(R) = p(B), turning over the -R card is informative: Finding Bon the back of the card falsifies the rule, whereas finding-B provides evidence for the rule (note I).

Now consider the case in which p(R) < p(B). That is, even if all ravens are black, there will still be instances of black nonravens. Nonetheless, these -R&B instances are informative, though less so than R&-B in­ stances. If  p(R) = .0 I,  and p(B) = .05, then the likelihood  ratio for the

-R&B  observation,p(-R&BIH1)/p(-R&BIH2),  equals  .04/.0495. The ratio is less than I, showing that a -R&B observation is evidence against  HI.  The likelihood  ratio for the -R&-B observation  is .95/

.9405, which is greater than l, and shows that it is evidence  for  HI. Thus, turning over the -R card when p(R) < p(B} is also informative: B on the other side provides evidence against the rule, whereas-Bon the other side provides evidence  for the rule.

Finally, consider the situation in which p(R) > p(B). Because  there are more ravens than black things, there must be instances of nonblack ravens. One need not tum  over any cards to know the rule is false.

In short, our analysis indicates that, under circumstances in which at least one card should be turned over, turning over the -R card is infor­ mative if the marginal probabilities of the properties being tested are the same under the competing hypotheses. In terms of our example, this means that whether or not the rule is true does not affect the proportion of things that are ravens and the proportion of things that are black. Oaksford and Chater (1994), on the other hand, fixp(R) but letp(B) de­ pend on the truth of the rule in a way that, by design, renders the -R card uninformative (p. 610). The reasonableness of the assumptions regard­ ing if and how the marginal probabilities vary under the competing hy­ potheses might depend on the specific context.