Friday, 14 January 2011

Facts are overrated anyway

This blog was started to share my thoughts on science, rationality and why they are continuously attacked. Today I am convinced intra-personal psychological processes create the caustic responses from humans that are confronted with scientific conclusions which contradict their ideology.

Even before I began writing this blog I was not impressed by what journalists produce. Knowing a little about medicine I was repeatedly surprised by the either outdated, incomplete, or even incorrect articles I read covering medicine. When I became a resident I was taught how to read, and write, articles. In every hospital I worked we would discuss two articles from medical journals -i.e. NEJM, JAMA, Annals of Internal Medicine, BMJ, Intensive Care Medicine, et cetera- on a weekly basis. By talking about the strenghts, and weaknesses, of the article this has increased my critical thinking skills. Undoubtedly one of the reasons this ritual is part of hospital life.

Unfortunately this is not part of journalist school. As such I wonder how this may influence the viability of nonsensical ideas in society. The type of journalism media currently practice does the general public a disservice by making the "fair-and-balanced"-fallacy a popular point of view. This doctrine increases the dumbing down of society, which has given us numerous absurd opinions. Also, the need for celebrities to engage in misinforming the public has devastating effects. Terri Judd, for The Independent, writes about these effects and on how the public sees manufactroversies. To illustrate the spread of the "I-refuse-to-adequately-inform"-virus, within the media, below are stories covering different topics that fall within the "facts-are-to-be-treated-as-opinions"-category.

Regarding vaccines:
Just recently there was a concerted effort to counter the misinformation being spread by the infectious-disease-promotion-movement. It was hugely successful. So much so that today radiostations have been enlisted to help keep us scared of science. One wonders how effective the fearmongerers would be without the complicit media wich refused to point out the numerous, and huge, inaccuracies being presented as facts. A review by David Gorski of:
two — count ‘em, two! — books taking a skeptical, science-based look at vaccines and, in particular, the anti-vaccine movement.
can be found here. Astonishingly, even after Brian Deer in a leading medical journal concluded the father of the current infectious-disease-promotion-movement, Dr. Andrew Wakefield, made up the article which started the recent anti-vaccination scare journalists refuse to discard the "fair-and-balanced"-doctrine. As Cyril Washbrook, for MediaSpy, reminds us:
As Ben Goldacre notes in a well-known critique of the media's reporting on the issue, the so-called "quality" press stood shoulder-to-shoulder alongside trashy tabloids in peddling fears that lacked a credible basis.
What does the Serious Reporter do?:
But even now, purportedly respectable media outlets continue to trot out their post-modernist, we-shall-never-adjudicate routine, showing that the lessons have not been learned. Step forward, CNN.
A now-retracted British study that linked autism to childhood vaccines is an "elaborate fraud," according to a medical journal - a charge the physician behind the study vigorously denies.
Note the immediate establishment of the basic heuristic: one person says the study is a fraud, the other person says it isn't. Unsurprisingly, the entire article proceeds along these very lines. Wakefield says it's a smear campaign; the BMJ says it's a genuine exposé. Wakefield says Deer has been paid off; Deer says he's independent.
There is no sign journalists realise that informing one's audience about the veracity of claims, not merely reporting the claims themselves, is what journalism should be about.

Regarding the placebo-effect:
We all have heard of the placebo-effect. A recent study suggests we underestimate the power of the placebo. According to what has been reported in the media it works even when patients are aware they are taking fake medicine. Predictably, the popular press has not been sceptical enough, as Orac points out.
I don't have a huge problem with the study. After all, it's a pilot study. The biggest problem I have is with how the study is being sold to the press, as though it were evidence that placebo effects can really be triggered without at least some degree of deception. It shows nothing of the sort.
More on this by PalMD and David Gorski.

Evolution is not science:
Once Darwin irked the religious with his theory their response has been: attack, attack, and attack. The reason for this is clearly their fear science might prove The Bible is not The Truth. One can imagine religion being shown to be nothing more than mythology. The horror. There was initially the Scopes trial and recently through slight of hand, introducing the sciency sounding reincarnation of creationism: Intelligent Design. Despite the fact ID has been shown to violate the basics of science they keep claiming this is proof of an Evil Atheist-plot to take over the world. Surely, the media have come to the rescue by promoting, and continuing, the teach-the-controversy-fallacy. Still, I am waiting for journalists to use this argument to report on the Holocaust-controversy, and the Flat-Earth-controversy.

Regarding global warming:
Despite the fact the science is settled ideologues keep telling us AGW is a hoax. An example of inadequate reporting is Climategate. This purportedly showed a conspiracy of scientists to keep The Truth from us. A fact journalists felt compelled to share with us. After it became abundantly clear this was not the case, and the scientific method was exonerated, the media could not be bothered to share that with us. At least, not with the same zeal, and headlines, as the alleged corruption. 

War of Terror:
The breakdown of journalistic standards became painfully apparent when the Bush administration was allowed to make the wildest accusations towards Iraq. Worse, the media themselves were the main purveyors of misinformation. Even today, while those claims have turned out to be unsupported by the then available evidence, the media are incapable of learning from that experience. Like Saddam then both Iran and Julian Assange are now The Biggest Threat In The World. Again, no questions are asked to counter that premise.

Political discourse in the US:
For decades it has been bon ton in the US, and it is spreading beyond its borders, to use the most antagonistic and inflammatory ways of discussing topics. It is called freedom of speech. In short, use ad hominems and never relevant arguments. Politicians, and newsmedia, have engaged in sharp descriptions of individuals. After Obama became President of the US the Republicans, their ministry of truth, and other supporters have used terms like traitors, terrorists, un-american (and much more) to describe members of the Democratic Party. Then, this week, a Democrat was shot. In light of the recent rhetoric by the political Right some have suggested a correlation with the toxic political climate. While it is difficult to prove any causality one has to be blind to ignore the possibility. In the words of Mike the Mad Biologist:
As I've said before, words do have meaning. Words should have meaning: if they don't, then do us all a favor and shut up. I believe Representative Trent Franks. I believe them when Rush Limbaugh and his millions of regular listeners believe we're the problem. And the anti-abortion movement has shown what happens when people post cross-hairs over people's names.
So let's not be so concerned with civility, but instead demand honesty and accuracy. That will serve us far better.
He also discusses a NYT article, by Matt Bai, on the shooting which employs the usual "but-both-sides-do-it"-meme.
........, John Cole succinctly sums it all up:
And then my personal favorite: "He was just crazy!" No shit. You have to be crazy to walk into a crowd of people and start spraying bullets, killing a bunch of elderly people and a little kid. That is crazy.
The point we have been trying to make for the last couple of years is that Republicans need to stop whipping up crazy people with violent political rhetoric. This is really not a hard concept to follow. There are crazy people out there. Stop egging them on.
The problem Bai has is that, if you report the obvious story--Republicans have been engaged in eliminationist and exclusionary rhetoric that has some of the hallmarks of fascism--there's nothing new there. It doesn't establish you as a 'contrary' thinker who comes up with devastating counterintuitive insights. But if you can 'establish' (even though you actually can't) that the Left does it too, then you have something different to say.
Another comment on the incident is made by We Beasties:
There's been plenty of talk about the violent rhetoric that's been spewed for the last 2 years, and many have blamed talk radio, Sarah Palin's map with cross-hairs over congressional districts (including congresswoman Giffords') and the like, and I don't have much to add on that front. There's been no direct connection, and there may never be, but I find it hard to believe that this atmosphere of violent hatred had nothing to do with this gunman's actions.
If you are wondering how this relates to my criticism on journalism I refer you to the fact-free, and at times delusional, opining by Glenn Beck, Rush Limbaugh, Sean Hannity, Bill O'Reilly, and many others which contributed to this climate. In all fairness, yes: they are not journalists but propagandists. Responding to the "the-guy-must-be-crazy"-view, by David Brooks, We Beasties notes:
Vaughan Bell has a devastating critique of this sort of thinking in Slate, noting that the most complete scientific research on the effects of mental illness show very little increase in risk of violent behavior.
More on mental health and violence can be found here.

Wikileaks exposes journalists as not doing their jobs:
As I noted before without the massive failure of journalism we would not have Wikileaks. Their recent disclosures appear to be based on Bradley Manning, who allegedly confessed to a total stranger, Adrian Lamo. His newfound friend then turned informant, and contacted Wired. Note the curious treatment he recieves while he is supposed to be innocent until proven guilty. This tellingly has led the U.N. to start an investigation.

To aid those favouring all sorts of conspiracies Wired refuses to confirm, or deny, the ever changing narrative being told by Adrian Lamo. Kept hidden from us, by Wired, is how he met Manning and what his relationship is with journalist Poulsen, quoth Greenwald:
That's what so much "journalism" now is:  a means of shielding secrets from the public -- usually to protect friends and the agendas of "sources" to ensure further access.  Ironically, it is that very mentality -- the Cult of Secrecy that American journalism has become -- that gave rise to the need for WikiLeaks in the first place.
He concludes:
The chat logs that Wired has but is withholding -- and about which they are refusing to comment -- are newsworthy in the extreme.  They cannot but shed substantial light on what really happened here, on the bizarre series of events and claims for which there is little evidence and much cause for doubt.  I expect government officials to shield the truth from the public and to conceal key evidence and facts.  But those who claim to be journalists should not be aiding in that effort.  Wired is doing exactly that.
Totally unsuspected the media in general also fail to tell about Wikileaks without incorporating copious amounts of inaccurate statements. How journalists, in this story, have become the voice of the Obama administration's PR-department is described by Greenwald. Of course, Wikileaks does things real journalists never do, they endanger lifes, or this is what we are being told. Therefore we should not be afraid of governments limiting free speech for the real media, who never publish secret information. Strangely enough The NYT itself is now endangering National Security with a new article. In the words of Greenwald:
In The New York Times today, Mark Mazzetti and Dexter Filkins expose very sensitive classified government secrets -- and not just routine secrets, but high-level, imminent planning for American covert military action in a foreign country.
He rhetorically asks:
The question that emerges from all of this is obvious, but also critical for those who believe Wikileaks and Julian Assange should be prosecuted for the classified information they have published: should the NYT editors and reporters who just spilled America's secrets to the world be criminally prosecuted as well?  After all, WikiLeaks has only exposed past conduct, and never -- like the NYT just did -- published imminent covert military plans.  Moreover, WikiLeaks has never published "top secret" material, unlike what the NYT has done many times in the past (the NSA program, the SWIFT banking program) and what they quite possibly did here as well.
Just to remind us of what we have learned because of Dah Evil Wikileaks, which Real Journalists failed to uncover, read this, this, this, and this.

With the above in mind, combined with the numerous examples I left out, what I notice about journalism today is that it does not matter what you are writing about, to be seen as a Serious Reporter the following characteristics are mandatory:
  1. There are always two sides to a story, 
  2. In the absence of any dispute pretend there is one, and present any discredited view as if the topic is still debated: failing to point out opposing opinions is a tell-tale sign of bias, hence point 1,
  3. Never point out any incorrect statements, or factual inaccuracies, by the parties involved: that would be taking sides, 
  4. Never reveal anything that damages those in power, i.e. see the hounding of anybody even remotely linked to Wikileaks. Limit your reporting to nefarious pawns and your career is a guaranteed success. 
  5. Never explicitly admit error on your part,
  6. A source is anyone who shares information with you, regardles of its veracity or factual accuracy,
  7. Expertise, or lack thereof, should not influence your decision to use a source, i.e. expert opinion is equivalent to that from laymen, and celebrities, which have studied at the University of Google,
  8. A priori you are required to keep the identity of a source, and any possible conflict of interest on their and your part, from the public. Especially when it turns out your source willfully lied to you in order to advance a political agenda,  
  9. Should you feel overly generous you might include ad hominems, straw men, and other invaluable arguments to get rid of those annoying people trying to steer the article/interview into a more rational position.
Concluding, the abysmal state of reporting is not limited to scientific manufactroversies. As long as journalism school teaches the above characteristics there will be the need for organisations that understand the adagium Serious Reporters find tedious and outdated:
is this a reasonable representation of the facts (NB: opinions do not equal facts) involved, and am I merely reporting a story, or is my reporting the story? In other words, is this a realistic portrayal of the facts involved and can it be supported by independent reliable sources?
Are the media to blame for all that is evil? No, but their habit of having propaganda pose as news is certainly not helping us in making informed decisions, i.e. should I vaccinate, or who do I vote for, does The Law look backward? Those on Planet Reality need to keep pointing out that they sure act like willful footsoldiers in the War on Reason and Sanity.

Update: Nice reading tip on "making mistakes," by We Beasties, for non-journalists too. Also, amended post slightly.

Wednesday, 12 January 2011

Norwegian Christmas 2010

Like last year I spent the holiday season in Telemark. Despite the chaos in Europe I was able to fly to Gardermoen, take the train, and continue by bus. Apparently snow is not a problem in Norway.

In Oslo I went out and tried tapas at Delicatessen, which has been on my wishlist for years. The only thing wrong with the place is that it is unbearingly crowded. They could be more frugal with the number of tables they squeeze in, and refrain from putting four people at a table for two. Having eaten here next on the to-do list is Sult, but it apparently no longer exists. Will find out next time I am here. Afterwards I had homemade beer in the local brewery: SchousKjelleren. The next day I took the Haukeliekspressen to Telemark. When I got there the thermometer claimed it was -22 °C. Brr, luckily I came prepared: warm clothes! Curiously enough I managed to travel while first evading the weather chaos, and then apparently I missed the sillyness at Gardermoen.

Christmas involved lutefisk, rakfisk with homemade flatbrød, pinnekjøtt, aquavit, risengrynsgrøt, rådyr, juleribbe. The cullinary events force me to consider fasting the entire next month. Apparently all my clothes have been washed too hot. Even those I did not bring with me. Guess what my New Year's resolution is.

Like last year I visited some friends to try the Norwegian shrimps. The next morning I took the train to Oslo to prepare for the flight home. The last day in Norway I used to do some shopping. But first I had some excellent coffee at Stockfleth. Oddly enough they forgot to install toilets. Then to see if there are any new DVD's, trousers and other important things: shopping centre. While I was waiting for the train to Gardermoen I ate a burger at Fiasco. Should you have some time and don't want to go shopping around Oslo S, please try their Fiasco burger.

This ends the festive season for this year.

Thursday, 6 January 2011

BMJ: Wakefield is a fraud

In a stunning article in the British Medical Journal we are told that Andrew Wakefield, possibly the principal cause of the recent vaccines-are-evil-hype, is even less trustworthy than we discovered before.

Of course, who but Orac is on the prowl:
The discrepancies between the case reports as described in Wakefield's Lancet paper and the actual medical records are anything but random; all are in the direction of suggesting a link between the MMR and Wakefield's as yet unverified syndrome of regressive autism and enterocolitis. The cases that were selected appear not to have been random, sequential patients but were rather recruited specifically through anti-vaccine activists and trial lawyers.
There is no innocent explanation possible for the systematic and numerous discrepancies between the medical record and Wakefield's paper, as the editors of the BMJ point out in their accompanying editorial:
The Office of Research Integrity in the United States defines fraud as fabrication, falsification, or plagiarism. Deer unearthed clear evidence of falsification. He found that not one of the 12 cases reported in the 1998 Lancet paper was free of misrepresentation or undisclosed alteration, and that in no single case could the medical records be fully reconciled with the descriptions, diagnoses, or histories published in the journal. 
Who perpetrated this fraud? There is no doubt that it was Wakefield. Is it possible that he was wrong, but not dishonest: that he was so incompetent that he was unable to fairly describe the project, or to report even one of the 12 children's cases accurately? No. A great deal of thought and effort must have gone into drafting the paper to achieve the results he wanted: the discrepancies all led in one direction; misreporting was gross.
Wakefield continues to deny that he has done anything at all wrong and blames the criticisms leveled against him on conspiracies. In reality, given the way the anti-vaccine movement has begun to circle the wagons to defend Wakefield yet again, it's tempting to claim that this is a conspiracy.
In his analysis Brian Deer likens this fraud in scope to Piltdown Man. The next commentator is Phil Plait who writes:
Brian Deer, an investigative journalist, has written a multi-part series on the BMJ site which slams Wakefield. Fiona Godlee, BMJ’s editor-in-chief, also writes about this… and just to be clear, she uses the word "fraud" nine times in her editorial. Not surprisingly, it’s been picked up by several news outlets like CNN, MSNBC, and ABC.
His conclusion:
Andrew Wakefield may not have started the antivax movement, but he certainly egged it on very strongly, along with such mouthpieces as Jenny McCarthy, and Meryl Dorey and the AVN in Australia. If the charges of fraud can be made to stick, then we might be able to make some progress toward reality once again, and lower the rate of outbreaks of measles, pertussis, and polio… and save a lot of lives in the process.
This is exactly why I am not opposed to accountability for willfully endangering other people by invoking free speech. More on the BMJ story by both Pharyngula, Deltoid, and Jeffrey H. Toney.

In the past I have wondered why people adhere to a worldview that has been thoroughly discredited. My unscientific opinion was that it must be some form of delusional disorder. Later, I noted the cognitive dissonance which has been shown to explain such behaviour. In light of what the BMJ has just made public one would hope the infectious-disease-promotion-movement will lose members. Unfortunately, being a cynic, I doubt that will happen.

Update: Be sure to read this too.

Update II: A roundup of responses to the BMJ article is provided by Liz Ditz. The Autism Blog discusses the alleged replication of Wakefield's results: it does not exist. In Scientific American David Ropeik explains that there is a discrepancy between the perceived and factual risk:
Sometimes we’re more afraid than the facts say we need to be (vaccines). With many of the bigger threats, we’re not afraid enough (infectious disease). The gap between our fears and the facts can be dangerous all by itself. Just ask the parents of the thousands of kids worldwide now getting, or dying of diseases that vaccines had pretty much controlled.
The harm he [Wakefield] and others have done will persist for a long time…and will continue to serve as a reminder of the risk we face if we don’t recognize that the way we perceive risk can be a huge risk in and of itself.
The danger of the infectious-disease-promotion-movement is shown by Maryn McKenna who contracted whooping cough in India and wrote an article in Wired about the rise in cases. Take home message:
The worst news in this upsetting trend is this: We’re doing it to ourselves. As far as anyone can tell, the rise in pertussis is not due to any change in the organism, or to any mysterious error among the manufacturers who make pertussis vaccines. It’s due to vaccine refusal, to parents turning away from vaccines because they think the vaccines are more harmful than the diseases they prevent — or, more selfishly, because they think the wall of immunity created by other vaccinated children will protect their unimmunized ones.
Grant Jacobs, for Code for Life, made an overview, and Skepacabra did the same. Nice review of communication pitfalls by The Thoughtful Animal:
Giving us incidence and death rates and other such statistics doesn't really get the job done. It doesn't communicate what they want it to. Nor will glossy pamphlets (like the one they gave me) featuring Mia Hamm telling us to get vaccinated. What will get the job done is story-telling, appealing to emotion, and utilizing accessible analogies. Instead of telling us how many gazillions died last year, tell us how many airplanes full of people, or how many football stadiums full of people died last year.
Update III: Luckily Jenny McCarthy comes to our recsue and rehashes long ago refuted non-arguments. Apparently, true or false is determined by the number of times you make a claim.

The Lehrer effect

Ever since Jonah Lehrer identified the supposed flawed nature of the scientific method, and named it the decline effect, the Intertoobz has responded by either advancing the notion this proves science is just as unreliable as other manufactroversies, or by pointing out that what Lehrer sees as problematic actually is why science is reliable: it eventually filters out human bias.

Eventhough I was somewhat late in noticing the current kerfuffle others are slightly faster to comment on the notion that science does not work. Some observations I initially missed are by Kent Anderson, responding to "a study that scientists thought proved that female barn swallows preferred mating with males who had long, symmetrical feathers:"
In this case, the raw observations about feathers are generalized first to one species, then attempts are made to generalize them to other species, and soon the attempts start to fail. This isn’t an erosion of truth. It’s a failure of theorizing. The theory was derived after the fact, and to no useful end other than to publish more papers. It wasn’t a hypothesis that was tested, but a data set shoehorned into a post hoc theory.
The scientific method itself is revealing the limitations of initial findings. It’s working. But we’re so geared to create “headline science,” and so wrapped up in ego and pride that we’ve forgotten the humility we need to exhibit before the facts. But most importantly, we may have forgotten that something is even more important than facts — and that is theory.
Writing for Big Think Matthew C. Nisbet notes that:
the reaction that the article has stirred in some cases does not match the nuance of Lehrer's arguments. The article has been unfairly critiqued by some for giving ammunition to those already committed to extreme doubt about subjects such as climate change or evolution.  As Lehrer notes at his blog, he's also been accused of being a post-modernist, arguing that there is no such thing as truth or reality.
Criticism he does not share. In his view the article, and the ensuing debate, are an opportunity to bolster science education. The added benefit is that they:
are wonderful teaching tools for science students.
A column by John Allen Paulos, for ABC News, reviews the article and elaborates on the explanations for why the "decline effect" might occur:
A greater realization of these effects by journalists, scientists, and everyone else will lead to more caution in reporting results, more realistic expectations, and, I would guess, a decline in the decline affect (more accurately, the stat-psych effect).
The criticism his article generated has led Lehrer to write a response. Nonetheless, it appears he did not grasp what was said, or as Orac puts it:
Actually, what Lehrer's critics have been doing is anything but reassuring ourselves with platitudes about the rigors of replication. Indeed, all of us who bothered to write about Lehrer's article spent considerable time pointing out how regression to the mean, publication bias, and a variety of other factors that could explain much of the decline effect. We spent a lot of effort trying to explain how it is unsurprising that initial promising results often appear less so as more and more scientists investigate a question, developing along the way better techniques and approaches to investigating the question and approaching it from different angles. We spent a lot of verbiage describing how it is not at all unsurprising that new drugs, which seem to work so well in early clinical trials, appear to lose efficacy as their indication is broadened beyond the homogeneous initial small groups of subjects to more patients whose characteristics are less tightly controlled. Indeed, one of the letter writers pointed this very fact out to Lehrer, but he chose not to address this point directly.
The decline effect is something any physician who does clinical research knows from experience (although he may not call it that) because he sees it so often. To Lehrer it seemed to be some sort of shocking revelation in clinical research. The expectation that randomized clinical trials can overestimate the efficacy of new drugs is the very reason why, after drugs are released, physicians sometimes carry out what are known as "pragmatic trials," which are designed to find out how effective a treatment is in everyday, real-world practice, where the conditions are not nearly as controlled and the patient populations not nearly as homogeneous as they are in randomized clinical trials. Efficacy results determined in pragmatic trials are virtually always less robust than what was measured in the original randomized clinical trials. Not that any of this stops Lehrer from simply repeating the same stuff about big pharma having incentives to shape the results of its science and clinical trials. We get it; we get it. Science is done by humans, and sometimes human biases and motivations other than scientific discovery influence thee humans who do science.
He continues with:
More importantly, after discussing the decline effect and impugning the reliability of science, Lehrer still can't seem to give a coherent explanation as to why AGW and evolution are such reliable, well-founded scientific theories compared to what he seems to perceive as the unreliability of the rest of science. Worse, he hasn't addressed many of the more cogent criticisms of his work, in particular the numerous attempts to explain to him why it is not at all remarkable that second generation antipsychotics have not proven to be as effective as initial results suggested or why it is not particularly surprising or disturbing that fluctuating asymmetry never panned out. Lehrer had a great opportunity to explain why making scientific conclusions is so difficult and why all scientific knowledge is provisional. Those points are in his articles on the decline effect, but they're buried in the surrounding implication that the decline effect is mysterious. Then in the last paragraph of his response to critics Lehrer has the chutzpah to declare that "there is nothing inherently mysterious about why the scientific process occasionally fails or the decline effect occurs."
Someone else not overly impressed by Lehrer is David Weisman who strongly opines:
the 'decline effect' is bullshit. Science also ignores things for long periods of time, then recognizes their importance, at which point the theory grows, strengthens, branches off into new fields, and improves. Germ theory is hardly in decline, it grows stronger with every pneumonia. DNA as genetic instruction is also a theory, one that is hardly in decline. Ditto evolution. Ditto greenhouse gas. Ditto cells, tectonic plates, gravity, sodium channels, and atoms. None seem in danger of decline.
Lehrer wrote a faulty article. It got published. Initially it must have seemed almost reasonable. But on second look, it has fatal flaws. In fact, it appears to me that he cherry picked examples to support his own pet theory, a classic fallacy of the highly biased. Now it doesn't look very reasonable at all. It looks like small minded pseudoscience.
Another take on the phenomenon is given by Mike the Mad Biologist by referring to Andrew Gelman:
Gelman (and he has some good slides over at his post) is claiming, correctly, that if the effect is weak and you don't have enough samples (e.g., subjects enrolled in the study), any statistically significant result will be so much greater than what the biology would provide that it's probably spurious. You might get lucky and have a spurious result that points in the same direction as the real phenomenon, but that's just luck.
Then he points out:
So what someone will do is report the statistically significant result (since we tend to not report the insignificant ones). But further experiments, which often aren't well designed either, fail to pick up an effect. The ones that are well designed and have a large sample size will either identify a very weak real effect, leading to a consensus in the field of "Meh", or correctly fail to find a non-existent effect.
Sounds like the Decline Effect to me.
By claiming science is unable to provide any definite answers, i.e. The Truth, Lehrer apperently agrees with the anti-science movement which claims science is merely another opinion. Once everything is opinion how could any reasonable person (you know, the "fair and balanced"-type) object to dissenting views such as: evolution is not true, global warming does not exist, vaccines are evil, HIV is harmless, et cetera? This inadvertent support of denialism, through the law of unintended consequences, I propose we call the Lehrer effect. In other words, the Lehrer effect stands for the proposition that the orchestrated efforts of misinformation-central, in time inevitably will contaminate and debilitate even the protectors of reason, i.e. they too eventually come to believe that the scientific method is no more reliable than pseudoscience and denialism.

Update: Amended definition last sentence.

Monday, 3 January 2011

Why the "decline effect" shows the scientific method works

Part of the struggle between science and ideology is the recurring theme that science too can be wrong. Note the number of times we have seen scientists retreating from their initial position to a more nuanced one. This transition to less pronounced statements causes the anti-science crowd to claim this proves science is not as reliable as we think. This phenomenon is discussed by David Gorski while responding to:
an article in The New Yorker by Jonah Lehrer entitled The Truth Wears Off: Is There Something Wrong With the Scientific Method?
The above concept is called the "decline effect" which, according to Gorski, stands for:
a phenomenon in which initial results from experiments or studies of a scientific question are highly impressive, but, over time, become less so as the same investigators and other investigators try to replicate the results, usually as a means of building on them. In fact, Googling “the decline effect” brought up an entry from The Skeptic’s Dictionary, in which the decline effect is described thusly:
The decline effect is the notion that psychics lose their powers under continued investigation. This idea is based on the observation that subjects who do significantly better than chance in early trials tend to do worse in later trials.
To him this observation is is neither shocking nor new:
In medicine, in particular, early reports tend to be smaller trials and experiments that, because of their size, tend to be more prone to false positive results. Such false positive results (or, perhaps, exaggerated results that appear more positive than they really are) generate enthusiasm, and more investigators pile on. There’s often a tendency to want to publish confirmatory papers early on (the “bandwagon effect”), which might further skew the literature too far towards the positive. Ultimately, larger, more rigorous studies are done, and these studies result in a “regression to the mean” of sorts, in which the newer studies fail to replicate the large effects seen in earlier results. This is nothing more than what we’ve been writing right here on SBM ever since its inception, namely that the normal course of clinical research is to start out with observations from smaller studies, which are inherently less reliable because they are small and thus more prone to false positives or exaggerated effect sizes
His conclusion:
Although Lehrer makes some good points, where he stumbles, from my perspective, is when he appears to conflate “truth” with science or, more properly, accept the idea that there are scientific “truths,” even going so far as to use the word in the title of his article. That is a profound misrepresentation of the nature of science, in which all “truths” are provisional and all “truths” are subject to revision based on evidence and experimentation. The decline effect–or, as Lehrer describes it the title of his article, the “truth wearing off”–is nothing more than science doing what science does: Correcting itself.
Commenting on this article Steven Novella writes that the term "decline effect:"
was first applied to the parapsychological literature, and was in fact proposed as a real phenomena of ESP – that ESP effects literally decline over time. Skeptics have criticized this view as magical thinking and hopelessly naive – Occam’s razor favors the conclusion that it is the flawed measurement of ESP, not ESP itself, that is declining over time.  Lehrer, however, applies this idea to all of science, not just parapsychology.
Just like Gorski he notes:
Lehrer is ultimately referring to aspects of science that skeptics have been pointing out for years (as a way of discerning science from pseudoscience), but Lehrer takes it to the nihilistic conclusion that it is difficult to prove anything, and that ultimately “we still have to choose what to believe.” Bollocks!
His explanation for this phenomenon is the same as Gorski's. The article was also noticed by Pharyngula who stated:
I read it. I was unimpressed with the overselling of the flaws in the science, but actually quite impressed with the article as an example of psychological manipulation.
And he then remarked:
Early in any scientific career, one should learn a couple of general rules: science is never about absolute certainty, and the absence of black & white binary results is not evidence against it; you don't get to choose what you want to believe, but instead only accept provisionally a result; and when you've got a positive result, the proper response is not to claim that you've proved something, but instead to focus more tightly, scrutinize more strictly, and test, test, test ever more deeply. It's unfortunate that Lehrer has tainted his story with all that unwarranted breast-beating, because as a summary of why science can be hard to do, and of the institutional flaws in doing science, it's quite good.
In short, the article erroneously points out the inherent characteristics of the scientific method as "proof" of why science is just another opinion. While provocative it identifies weakness but fails to recognise this actually is science's strongpoint.

As an side, this flowchart on how to debate the anti-science brigade (creationists actually) is brilliant:
(h/t Pharyngula)

Update: Following the "decline effect" there now is the Lehrer effect.