Bad Sam Harris Brain Scanning Research Paper

This post criticizes The Neural Correlates of Religious and Nonreligious Belief by Sam Harris, Jonas T. Kaplan, Ashley Curiel, Susan Y. Bookheimer, Marco Iacoboni, Mark S. Cohen in 2009. I wrote this as a reply on the Change My View subreddit, and made minor edits so it'd stand alone.

Once we had two groups of subjects (Christians and Nonbelievers)

Specific criteria used are not given, making this research non-reproducible. This especially concerns me because such criteria are controversial and I would expect to disagree with the study authors about some categorizations regarding which persons think about which topics in religious ways (I don't think that religious thinking is all or nothing).

Later, they admit the screening criteria were poor, and make excuses. They later admit, "the failure of our brief screening procedure to accurately assess a person's religious beliefs".

To this end we assessed subjects' general intelligence using the Weschler Abbreviated Scale of Intelligence (WASI)

It's spelled "Wechsler".

IQ tests have many problems. Here is a previous discussion where I pointed out some of the problems. http://curi.us/2056-iq

screened for psychopathology using the Brief Psychiatric Rating Scale (BPRS)

Their non-random screening, including this, dropped 44% of people. That's getting far from a representative sample of the population! And those are only of the people who met the first 5 criteria that already included two related to psychiatric issues.

There are lots of problems with psychiatric screenings. I'm not going to go into it in detail here, but see these books criticizing psychiatry: http://fallibleideas.com/books#szasz

Forty of these participated in the fMRI portion of our study, but ten were later dropped, and their data excluded from subsequent analysis, due to technical difficulties with their scans (2 subjects), or to achieve a gender balance between the two groups (1 subject), or because their responses to our experimental stimuli indicated that they did not actually meet the criteria for inclusion in our study as either nonbelievers or committed Christians (7 subjects).

Dropping those 7 people is a big problem. They were removed because their data didn't fit the expected answer patterns. IMO that should have been a learning opportunity to reconsider mistaken expectations.

---

Here are example stimuli from the experiment. I didn't read them all, but it looked to me like over half the groups of 4 stimuli had a flaw. Also there's a systematic bias: the Christian truths are more often non-hedged statements, while the atheists truths are more often hedged. E.g. in 19 you read "The Bible" in the Christian one and "Most of the Bible" in the atheist one, and 29 has Jesus either "literally" rising from the dead or "probably" not rising from the dead.

The Bible is free from error.

This is categorized as something that all the Christian participants should consider true. But many serious Christians do not believe this.

The Bible is free from significant error.

It's weird that they have two very similar questions.

All books provided perfectly accurate accounts of history.

Grammar error.

The Bible is full of fictional stories and contains historical errors.

This is categorized as something that all the Christian participants should consider false. But many serious Christians do believe this.

People who believe in the biblical God often do so on very good evidence.

This is categorized as something that all the Christian participants should consider true. But many serious Christians do not believe this.

It reasonable to believe in an omniscient God.

Grammar error.

Jesus Christ can’t do anything to help humanity in the 21st century.

This is supposed to be considered true by non-believers, but many non-believers (including me) consider this statement false. (Though it's vague: do they mean Jesus Christ literally and personally can help people today, or his teachings can help? I'd change my answer depending on that. It's his teachings that I think can do "anything" (more than zero) to help.)

In general, they shouldn't have used words like "anything", "all", most", "greatest" because people routinely misread those statements (misreading e.g. "all" as "most", or vice versa). And those kind of statements are so often written incorrectly and carelessly that readers, reasonably, don't expect reliable, literal precision from them.

Jesus was literally born of a virgin.

Lots of Christians don't believe this – possibly because they are more educated about their religion (not less). "Virgin" (in the sense of not having sex) is a mistranslation – he was born of a young women (which, btw, is a typical meaning of "virgin" in English).

The Biblical story of creation is basically true.

Tons of Christians aren't young Earth creationists.

Most of the Bible is inferior to modern thinking on morality and human happiness.

This is supposed to be considered true by atheists, but as an atheist I consider it vague (which modern thinking?). If I try to read it using guesses about what the author of the statement meant, I think I disagree with it. Also if I read it with a "most" before "modern thinking", then I'd judge it false.

The Biblical story of creation is purely a myth.

This is supposed to be an atheist truth, but as an atheist I consider it false (due to "purely", which I mentioned above is the kind of word they shouldn't have used because people vary in how literally they read it). It's also problematic because I think many atheists aren't adequately familiar with the Biblical story of creation.

The Christian doctrine of the Trinity is almost surely fictional.

Many atheists couldn't say what what the Trinity doctrine is. And many atheists, including me, would disagree with this due to the "almost surely fictional". I consider it fictional and would not want to hedge in that way. If the words "almost surely" were deleted then I'd agree with the statement, but I'm not comfortable with this statement as written. There were lots of statements that were supposed to be things I would agree with, but which included hedges I don't believe.

Human beings have complete control over the environment and can grow food anywhere.

This is vague. They consider it false. But we can grow food in airplanes, submarines or spaceships. Where, exactly, can't we grow food? In the middle of active volcanos? In the middle of the sun? Did they expect me to worry about suns or black holes because of taking "anywhere" literally?

The greatest human accomplishments have had nothing to do with God.

This is one of the worst ones. This is meant to be considered true by atheists. But, historically, most human accomplishments (great and small) were accomplished by religious people who often did think God was relevant (or the gods in the case of polytheists like many Greeks). Example: https://en.wikipedia.org/wiki/Religious_views_of_Isaac_Newton

It is wise to create a government that can help protect its citizens from harm.

I'm confused about why this is meant to be false for everyone. Most people agree with this, right?

Also, for 54 and 55 they accidentally swapped the Christian and Atheist truths. Since they have things categorized incorrectly and make grammar and spelling errors in what they published, I'm concerned that these 4 statements were miscategorized in the actual study.

---

I could go on and on. There are tons of stimuli with these kinds of problems. This is not up to the high standards required for scientific progress. And they actually excluded 7 people for not answering the questions reliably enough (over 90%) in the way the study authors expected them to answer based on the poor phone screening. And, overall, it looked to me like a lot of highly religious Christians would agree with well under 90% of the Christian truth stimuli, so I think the experimental design is bad. The researchers seem to think that e.g. if you believe in evolution you aren't a serious, religious Christian, which is incorrect. Note they failed at their own design goal that:

All statements were designed to be judged easily as “true” or “false”

Anyway, I'm not even trying to be comprehensive with the issues. There are just a lot of issues. And cites to a ton more issues, e.g. I could go through "The role of the extrapersonal brain systems in religious activity" and point out flaws with that (it's the cite on some text I particularly disagreed with). For now, I'll continue with some comments on the brain scanning aspect since I didn't get to that yet.

For both groups, and in both categories of stimuli, belief (judgments of “true” vs judgments of “false”) was associated with greater signal in the ventromedial prefrontal cortex

This is like measuring magnitudes of electric signals in different regions of CPUs while running different software. That would be a bad way to understand CPUs or software.

Actually, overall, the brain scanning stuff is hard to criticize due to the lack of substantial claims. They need to conclude something significant for me to point out how the evidence is inadequate for the conclusion. But they didn't. Big picture, the paper says more like "We did something and here's the data we got" which is true as far as it goes. They were looking for correlations and found a couple. Finding correlations is quite different than understanding and making claims about how people think. The world is packed full of non-causal correlations. Due to the lack of major claims about the brain scan correlations meaning anything, I'm done. It has quality issues and doesn't reach important conclusions.