Astrology believers and their approach to scientific evidence

Bennett Haselton, 2008/08/27

Given the number of people who believe in astrology (in the face of overwhelming scientific evidence, as well as what many people consider "common sense"), it is natural to ask two questions: (1) Do people believe in astrology because they don't consider "scientific" evidence to be more valid than anecdotal evidence? and (2) Would people be dissuaded from believing in astrology in the face of scientific evidence?

The answer to the first question, according to a survey, seems to be that even believers in astrology can distinguish between scientific and unscientific evidence. We surveyed users and asked them to consider two possible methods of testing whether astrology works -- one method clearly scientific and one clearly unscientific. While virtually all of the astrology-non-believers correctly chose the scientific test as the more "fair" one, a clear majority of the astrology believers also chose the scientific test as the more "fair" (even when the order of choices was randomized).

The answer to the second question is more complex. We randomly divided astrology believers into two groups that took two different surveys. In the first group, we described a hypothetical scientific experiment to test the validity of astrology, and asked respondents whether they thought the experiment would produce a positive result or a negative result. We also asked, "If this experiment were to produce a negative result, would your faith in astrology be decreased?" In the second group, we told astrology believers that the experiment actually had been carried out, and produced a negative result (this was true), and asked the believers, "Does the fact of the negative result in this experiment, decrease your belief in astrology?"

Among the respondents in the "hypothetical experiment" group, almost all astrology believers said they would expect the experiment to produce a positive result, and almost all astrology disbelievers said they would expect the experiment to produce a negative result (showing that nearly all subjects are able to translate their beliefs into expected experimental results). However, among astrology believers, the proportion of participants who said that they would be dissuaded by a negative result in a hypothetical experiment, was somewhat higher than the proportion of participants who actually were dissuaded by the news that the experiment had been carried out and produced a negative result. This suggests that among many believers in astrology, there is a capacity for after-the-fact rationalization -- an ability to maintain a belief in scientific inquiry generally, while disregarding results once they are produced.

Surveys

[Note: this page contains links to surveys. You can view the surveys, but please don't fill them out, as that will invalidate the data!]

Participants were recruited using Amazon Mechanical Turk and paid $0.15 apiece to take the surveys.

Participants were randomly shown one of two surveys, here and here. Both surveys asked the subject if they believe in astrology, and then asked them to ask which of two experiments would be a more fair way to test if astrology works -- a "scientific" experiment and an "unscientific" experiment. (The surveys are identical, except in the second question, the order of the responses is changed; this was to ensure no bias based on the ordering of the responses. No such bias was detected, so we consider the responses to the two surveys as a group.)

The proposed "unscientific" test was to poll astrology customers and ask them to rate on a scale of 1 to 10 whether they think their astrologer gives good advice. The proposed "scientific" test was to have astrology customers randomly divided into two groups -- one consulting a real astrologer, and one an actor pretending to be an astrologer -- and a week later, rate whether the predictions were accurate, and see if the real astrologer scored better. (As at least one survey respondent pointed out in the "comments" section, while the second experiment is clearly "more" scientific, it is still flawed, since, for example, a professional astrologer might be better at making vague predictions that turn out to be "true", or reading the customer's desires. A better experiment, carried out in 1985 by Shawn Carlson, is discussed later.)

203 responses were collected. 84 respondents said they believed in astrology while 119 said they did not.

Of the 84 respondents who said they did believe in astrology, 67 picked the scientific experiment as the most fair test, while 17 picked the unscientific experiment.

Of the 119 respondents who said they did not believe in astrology, 114 picked the scientific experiment as the most fair test, while 5 picked the unscientific experiment.

Conclusions: (1) users who disbelieve in astrology are more able to distinguish the scientific experiment as the more fair test of whether astrology works; but (2) even the strong majority of astrology believers were still able to discern that the scientific test was more fair than the unscientific one.

For the next round of surveys, new participants were recruited and randomly redirected to one of two different surveys, here and here. Both surveys discussed an experiment done in 1985 by Shawn Carlson to test the validity of astrology. The first version asked respondents if they believed in astrology, and if so, what would they expect to be the result of the hypothetical experiment. 15 respondents said they believed in astrology, and of those, 14 out of 15 said they expected the experiment to produce a positive result. 25 respondents said they did not believe in astrology, and of those, 21 out of 25 said the experiment would produce a negative result.

The second version of the survey told subjects that the experiment actually had been done, with a negative result. (The descriptions of the experiment in the two versions of the survey, were word-for-word identical, except for the verb tenses.) For our next comparison, we looked only at the responses from subjects in the two groups who said they did believe in astrology.

When comparing the results obtained just from astrology believers:

[These numbers are not large enough to be significant yet. Results are still coming in.]