Saturday, December 11, 2010

Perspectives on Psychological Science: Blogs Don't Exist

The Seductive Allure of Neuroscience Explanations


The previous post, Voodoo Correlations: Two Years Later, was a retrospective on the neuroimaging methods paper that was widely discussed in the blogosphere before it was considered "officially" published (Vul et al., 2009). The article, a controversial critique of the statistical analyses used by fMRI investigators in social neuroscience, made its initial appearance on Ed Vul's website once it was accepted by Perspectives in Psychological Sciences. This caused considerable consternation among the criticized authors and the journal editor (Ed Diener).

Now, as part of the November 2010 issue of the journal (Diener's last as editor), six invited articles on Neuroimaging: Voodoo, New Phrenology, or Scientific Breakthrough? appear in a Special Section on fMRI (Diener, 2010). I was pleased to see that one of the articles addressed The Appeal of the Brain in the Popular Press (Beck, 2010), since this has been a major theme of my blog for almost five years. However, I was disappointed that the word "blog" was not mentioned at all in Beck's article.

This should have come as no surprise, given the journal's response to bloggers in May 2009. The Editor's Introduction is worth a mention for the issues it raises about peer review and publication in these modern times.
PREPUBLICATION DISSEMINATION

As soon as I accepted the Vul et al. article, I heard from researchers about it. People around the globe saw the article on the Internet, and replies soon appeared as well. Although my plan was to publish the article with commentary, the appearance of the article on the Internet meant that researchers read the article without the accompanying commentaries and replies that I had planned to publish with it.

In some fields such as economics, it is standard practice to widely disseminate articles before they are published, whereas in much of psychology this has been discouraged. An argument in favor of dissemination is that it speeds scientific communication in a fast-paced world where journal publication is often woefully slow. An argument against dissemination of articles before publication is that readers do not have the opportunity to simultaneously see commentary and replies. ... In the Internet age, the issue of prepublication distribution becomes all the more important because an article can reach thousands of readers in a few hours. Given the ability of the Internet to communicate so broadly and quickly, we need greater discussion of this issue.
In reply, I wrote:
Bloggers have discussed this specific issue months ago. For example, as noted in Mind Hacks,
The paper was accepted by a peer-reviewed journal before it was released to the public. The idea that something actually has to appear in print before anyone is allowed to discuss it seems to be a little outdated (in fact, was this ever the case?).
And The Neurocritic opined that...
[The aggrieved authors] are not keeping up with the way that scientific discourse is evolving. Citing "in press" articles in the normal academic channels is a frequent event; why should bloggers, some of whom are read more widely than the authors' original papers, refrain from such a practice? Is it the "read more widely" part?

-from The paper formerly known as "Voodoo Correlations in Social Neuroscience", by The Neurocritic

Diener originally solicited six commentaries on the Vul et al. paper for the May 2009 issue of the journal. Ironically, authors on two of the papers have serious blogs:

Statistical Modeling, Causal Inference, and Social Science is a blog written by Andrew Gelman, a Professor of Statistics and Political Science at Columbia. He was one of the first to blog about the paper in Suspiciously high correlations in brain imaging studies, with a more detailed post a month later (More on the so-called voodoo correlations in neuroscience). Lindquist and Gelman (2009) applauded the discussion engendered by "pre-publication dissemination":
Their article has in a short time given rise to a spirited debate about key statistical issues at the heart of most functional neuroimaging studies. The debate provides a useful opportunity to discuss core statistical issues in neuroimaging and ultimately provides a chance for the field to grow and move forward.

[citation needed] is the blog kept by Tal Yarkoni, a Post-Doc at the University of Colorado Boulder. He happens to be an expert in statistics for fMRI analysis, and another one of the authors invited to submit a paper for the Vul, Harris, Winkielman, and Pashler festschrift/verdammung (Yarkoni, 2009):
In this article, I argue that Vul et al.'s primary conclusion is correct, but for different reasons than they suggest. I demonstrate that the primary cause of grossly inflated correlations in whole-brain fMRI analyses is not nonindependence, but the pernicious combination of small sample sizes and stringent alpha-correction levels. Far from defusing Vul et al.'s conclusions, the simulations presented suggest that the level of inflation may be even worse than Vul et al.'s empirical analysis would suggest.
His blog started in October 2009, after the commentaries were published.


The Appeal of the Brain in the Popular Press

That brings us back to the article by Diane Beck, an Assistant Professor in Psychology and Cognitive Neuroscience at the University of Illinois Urbana-Champaign. She examines the distorted media coverage of neuroimaging studies, and possible reasons for it (Beck, 2010):
Since the advent of human neuroimaging, and of ... fMRI in particular, the popular press has shown an increasing interest in brain-related findings. In this article, I explore possible reasons behind this interest, including recent data suggesting that people find brain images and neuroscience language more convincing than results that make no reference to the brain (McCabe & Castel, 2008; Weisberg, Keil, Goodstein, Rawson, & Gray, 2008). I suggest that part of the allure of these data are the deceptively simply messages they afford, as well as general, but sometimes misguided, confidence in biological data. In addition to cataloging some misunderstandings by the press and public, I highlight the responsibilities of the research scientist in carefully conveying their work to the general public.
While reading through the examples of poor media coverage, imagine the shock of recognition if you were to realize that you have written several trenchant blog posts criticizing these very articles. Yet all this work (and the writings of many others) is rendered invisible to the mainstream of the Association for Psychological Science.

Why is blogging so non-existent in these circles? There's a large thriving community of science blogs. Go to ResearchBlogging.org and look under Psychology, for starters. An excellent example of a psychology blog written by a senior investigator is BishopBlog by Professor Dorothy Bishop, who studies children's communication impairments at Oxford. Two of my favorite posts are The difference between p < .05 and a screening test and Science journal editors: a taxonomy.

As for Beck's hit list and The Seductive Allure of Neuroscience Explanations (Weisberg et al., 2008), we have:

(1) Men see bikini-clad women as objects, psychologists say
In other words, the choice of tasks or conditions that the researcher will contrast is absolutely critical to obtaining activity, as well as any inference that can be made about this activity. By omitting this fact from a press article, one gets the impression that participants are asked to do a single task in the scanner, such as view pictures of women in bikinis, and voilà, a set of areas light up (Landau, 2009)...
Here's Spanner or Sex Object? by The Neurocritic:
OK, the abstract doesn't specifically mention the tool/bikini experiment, so we have to rely on newspaper articles and quotes from the first author. Judging from the first 4 paragraphs of the Guardian article, Fiske's conclusions rely on the logical fallacy of reverse inference - one cannot directly infer the participants' cognitive or emotional state from the observed pattern of brain activity in neuroimaging experiments. How do we know that the "sex object" neural response was related to tool use? Did the experiment require the subjects to use tools? Did it explicitly ask them to anticipate using tools? How about watching others use tools?

(2) Greene JD, Sommerville RB, Nystrom LE, Darley JM, Cohen JD. (2001). An fMRI investigation of emotional engagement in moral judgment. Science 293:2105-8.
What does it mean to say that moral decisions are associated with activity in regions implicated in emotional processing (Greene et al. 2001)? What exactly is meant by moral decisions or emotional processing? The only way to really understand these statements is to also know what are not moral decisions, and how emotional processing is being defined.
Here's Everybody's a Neurocritic!
In that paper, the authors
reported that the medial frontal gyrus and other brain regions linked to emotion become more active when people contemplate "personal" moral dilemmas--such as shoving the man onto the trolley tracks or removing a man's organs against his will to save five transplant recipients--compared with when they weigh impersonal moral dilemmas--such as flipping a switch to save the workers or declaring bogus business expenses on a tax return.
Besides the medial frontal gyrus [BA 9/10, which did not replicate in Experiement 2], what were these other brain regions linked to emotion? Did they include the insula? No, they did not. They included the posterior cingulate gyrus (which has some grounding in reality) and the L and R angular gyri (which does not).

(3) A Neuroscientific Look at Speaking in Tongues

...In the press article, the author describes a study that used single photon emission computed tomography to compare regional cerebral blood flow in women speaking in tongues to cerebral blood flow in the same women singing gospel music (Newberg, Wintering, Morgan, & Waldman, 2006). Although glossolalia and singing both involve verbal utterances and evoke religious meaning in practioners, the women described a lack of voluntary control over their vocalizations only during glossolalia. The study found decreased activity in the prefrontal cortex during glossolalia, a result that the study’s authors describe as consistent with the women’s descriptions of a lack of intentional control over their utterances. The press article does not explicitly endorse this conclusion but instead chooses to quote Andrew Newberg, the lead author on the study: “The amazing thing was how the images supported people’s interpretation of what was happening. The way they describe it, and what they believe, is that God is talking through them” (Carey, 2006). ... The study’s result, however, is neither amazing nor does it support any particular cause of glossolalia. Decreased activity in the prefrontal cortex could be due to any number of reasons...
Glossolalia from The Neurocritic still gets comments from those who feel they speak in tongues, probably because it appears on the first page of a Google search on the topic.

And The New York Times Is Speaking In Tongues:
As I mentioned, oh, just the other day, (1) the authors did not correct for multiple comparisons, (2) the spatial resolution of SPECT is not that great, and (3) the rCBF reductions in right dorsolateral prefrontal cortex and (especially) the left caudate were their most significant findings.

(4) Different brain responses found in homosexual, heterosexual men [original link from the article was broken]
...the Associated Press (Schmid, 2005) reported on a PET study showing that when homosexual men sniffed a derivative of testosterone, their hypothalamus responded more like that of heterosexual women than heterosexual men (Savic, Berglund, & Lindstrom, 2005). The Associated Press rightly stated that “the findings clearly show a biological involvement in sexual orientation.” However, they then make an erroneous jump from describing homosexuality as being biological to being innate, primarily in the form of a quote from Dr. Sandra Witelson: “It is one more piece of evidence … that is showing that sexual orientation is not all learned” (Schmid, 2005). A difference in the brain in no way indicates that the behavior under study is not learned...
See Sweat, Urine, and Sexual Orientation and The PNAS Word from 2006. [I got tired of waiting for the embargo to lift, so these aren't as critical as I might have liked.]

You get the idea. But in the end, it's all for naught because Diener (2009) has spoken:
I believe that the debate can itself stimulate useful discussions about scientific practices and communication. Further discussion of the issues should now take place in journals that are focused on imaging and neuroscience, so that the readers there can judge and benefit from the ensuing discussions.

References

Beck, D. (2010). The Appeal of the Brain in the Popular Press. Perspectives on Psychological Science, 5 (6), 762-766 DOI: 10.1177/1745691610388779

Diener E (2009). Editor's Introduction to Vul et al. (2009) and Comments. Perspectives on Psychological Science 4:272-273,

Diener E (2010). Neuroimaging: Voodoo, New Phrenology, or Scientific Breakthrough? Introduction to Special Section on fMRI. Perspectives on Psychological Science 5:714-715.

Lindquist MA, Gelman A (2009). Correlations and Multiple Comparisons in Functional Imaging: A Statistical Perspective (Commentary on Vul et al., 2009). Perspectives on Psychological Science 4:310-313.

Vul E, Harris C, Winkielman P, Pashler H (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science 4:274-290.

Weisberg DS, Keil FC, Goodstein J, Rawson E, Gray JR. (2008). The seductive allure of neuroscience explanations. J Cogn Neurosci. 20:470-7.

Yarkoni T. (2009). Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary on Vul et al. Perspectives on Psychological Science 4:294-298.


The Seductive Allure of Neuroscience Blogs

Subscribe to Post Comments [Atom]

11 Comments:

At December 11, 2010 6:25 AM, Blogger Disgruntled PhD said...

That is a disgrace on the part of Perspectives. For my part, i would never have heard about these papers without blogs, and i am grateful for all the good coverage of this important issue.

The solution does not lie in ignoring blogs, rather in speeding up publication times within psychology.

 
At December 11, 2010 7:24 AM, Anonymous scicurious said...

A brilliant post and an excellent point.

 
At December 11, 2010 10:15 AM, Blogger Michelle Greene said...

Excellent post. The other side of this same coin is that as academics, we need to move away from viewing blogging as a time waster. It is an important outreach activity, and can help de-noise the peer review process.

 
At December 11, 2010 11:35 AM, Blogger Unknown said...

That's the last straw! I'm gonna set up my rss feed to auto-email Science and Phys. Review Letter! Lets see how they like them apples

 
At December 11, 2010 1:15 PM, Blogger Unknown said...

I guess a good way to advance on the fight blogs/peer-review would be to make the reviwers' comments public once a paper get published. No?
[I double-posted this comment on the previous post, sorry!]

 
At December 11, 2010 3:25 PM, Anonymous Morton Ann Gernsbacher said...

FWIW, the Vul, Harris, Winkielman, & Pashler "Puzzlingly High Correlations in fMRI Studies ..." article is, according to SAGE, the most cited article in its home journal, Perspectives on Psychological Science.

It's a provocative article, so undoubtedly its inherently high interest value helped drive its citation count. But one can't help wondering if the article's post-acceptance-prior-to-publication blog-enabled publicity, including that provided by The Neurocritic, helped the article, and by extension, the journal (and its parent organization) leap to greater visibility.

(Disclosure: I'm a recent president of APS and an author of an article in this month's issue of the same journal.)

 
At December 11, 2010 7:59 PM, Anonymous Yigal said...

Dear Neurocritic, I understand your frustration. You blog posts, which I follow with much interest, are as good as any peer-reviewed material. Is it possible that Dr. Beck didn't cite your blog because you're using a pseudonym?

 
At December 12, 2010 3:54 AM, Blogger Neuroskeptic said...

I can kind of see why he left out blogs. "The popular press" is generally taken to exclude blogs, because blogs never go to press, they're electronic.

This is a silly distinction because even with paper publications, most people read them online nowadays. And in terms of readers the big blogs easily beat the small and medium "press". However a lot of people still make the distinction.

Still, leaving out blogs misses at least half the story, and he really ought to have at least mentioned them, to explain why he left them out.

 
At December 12, 2010 6:22 AM, Anonymous Anonymous said...

Here is a provocative question: Are all or most "neuro" bloggers essentially disgruntled scientists, sick, for instance, of the old-boy networks that control many scientific publication outlets and grant review panels?
I seriously suggest the formation of academic WikiLeaks sites as a tool to dismantle such networks (more than blogs), because corruption and secrecy is at their core.

 
At December 12, 2010 3:15 PM, Anonymous tal said...

Thanks for the link! I agree it's kind of annoying that blogs don't get more respect, but I think it's simply a matter of time before norms and expectations change. I imagine 10 years from now blogs will be viewed as simply another channel of scientific communication, though I doubt they'll ever replace peer-reviewed journals (nor do I think they should).

As it stands right now, things aren't really that bad. It's true that science bloggers don't seem to reap any official benefits from blogging (e.g., I'm on the job market right now, and I very much doubt committee members are mentally blessing me with an extra couple of papers because of equivalent 'service to the profession'), but I think the informal response is usually very positive. Speaking personally, I know of many scientists who read my blog who probably wouldn't have any clue what I do otherwise. So, in a sense, it's immaterial what people say explicitly about the value of blogs; scientists do read them, and they do exert an influence on the debate in many cases.

 
At December 13, 2010 2:43 PM, Blogger The Neurocritic said...

Thanks to all for your comments. I'd like to see blogs become recognized as useful tools in science publishing, both for promoting one's own work (and that of others), and for providing a post-publication forum for critique and discussion.

This recent post by Rosie Redfield sums up why online commenting systems at journals like PLoS and the Frontiers series haven't worked as well as anticipated:

How to harness distributed discussion of research papers

. . .

"A few forward-thinking online journals (PLoS and BMC groups, I'm talking about you) provide their own Comments thread for each paper, so other researchers can provide informal but public feedback . But the researchers don't use these, saying that they don't feel comfortable doing this publicly, or that they don't like the bother of having to register and log on. I know that's true for me, thought I don't know why - I'll happily blog about a paper I've read, but I almost never post comments on its official Comments page."

One solution she proposes is a link aggregation system:

"Most journals already provide, with each paper they've published, a list of links to the more recent papers that cite it. The suggestion I really liked was that the journals should also aggregate the informal commentary, by providing a separate list of links to ALL the web pages that have link to the paper. Journals could then stop fighting our unwillingness to post comments centrally, and just use our distributed posts to add value to the papers they publish."

----------

Dr. Gernsbacher - Thank you, it was gratifying to hear that someone affiliated with APS (namely, a past president) sees the value in blogs as a way to raise visibility and reach a broader audience.

Yigal - The problem of pseudonymous sources in blogs definitely raises a barrier for those not accustomed to such a thing. How can the reader know the credentials of the writer or trust whether he/she is credibile? Basically, you have to read a number of posts over time to determine the expertise of the author, and many readers don't have the time or inclination. But I think the issue extends to blogs with known authors who have PhDs.

Neuroskeptic - Yes, it's a silly distinction for a completely online publication such as Slate to be considered "legitimate press" while blogs in general are not. I don't know whether all blogs should be considered part of the popular press, however. I'm a scientist, not a journalist, and I typically write for a more specialized audience. Some science blogs are on media networks such as Discover or Wired, and these tend to have a high proportion of science writers, and are aimed at a general audience.

Anonymous - I have to confess that one of the reasons I started blogging was aggravation with anonymous peer review. So why shouldn't I be able to anonymously review other's work (in a format that happens to be online)? When I started, I didn't expect people to read it...

Tal - Thanks for putting things in a more positive perspective... Good luck with the job search, your accomplishments should (hopefully) land you a good one.

 

Post a Comment

<< Home

eXTReMe Tracker