PEER REVIEW AND ITS DISCONTENTS

...Take the case of a journal much in the news lately, the American Journal of Hypertension...

...Follow the likely chain of evidence and you’ll see the problem...

...The press traditionally sees its role as questioning authority, but in this circumstance the press is vesting authority...

...What a journalist needs in these anxious moments is protective cover, and he finds it in these magic words: “peer review...

...Thus is a prestigious publication born...

...Fortunately, JAMA’s incumbent editor is a man of distinguished background and, by all accounts, high character...

...Most journalists (including those of us who produce science television) face such moments armed only with a semester or two of hard science...

...The anxiety attack is quick and sure...

...We have a right to expect transparency—the traditional scientific approach in which the researcher’s work is open to inspection, the reviewers stand by their critiques, the debate is joined, and the community works toward consensus...

...Just as peer review tends to take idiosyncratic shape in the eye of the beholder, so do the adjectives velcroed to medical journals...

...The reporter’s experience has long been that, in most circumstances, a source willing to go on the record is more credible than a source who requests anonymity...

...Which begs this question: Just how reliable is the peer-review process upon which this extraordinary authority is based...

...The public has placed great trust in our scientific institutions—in many cases, the trust to make life-and-death decisions for the rest of us...

...It is not generally considered a first-tier journal, but it is one of the industry’s most energetic practitioners of science by press release, in which a study’s findings are summarized, and hyped, for media distribution...

...What we have, in the case of JAMA, is a peerreviewed journal that publishes articles selected by an editor after he has consulted with unidentified reviewers who may or may not have deemed the article worthy of publication...

...The author of the study introduces his thesis and the group then has at it...

...Jones, says, roughly, that the sky is just fine...

...Indeed, there is no JAMA rule that a study must command even a majority of reviewers...

...The researcher, an expert in a narrow specialty, reports to the editor, who is perforce a generalist...

...When JAMA comes out with a big story, it jumps the editorial queue and appears immaculately on the evening news...

...In other words, peer review does not mean peer approval...

...A curious response, at least to the journalistic ear...

...What to do, as the clock on the wall ticks toward deadline...

...Tom Knisely assisted in the research for this article...

...For starters, there are times when only three, two, or even one reviewer actually participates—for all kinds of reasons...

...Here’s how JAMA subjects an article submitted for publication to peer review...

...The agency invites top experts in the field—FDA staff, academics, practicing physicians...

...But if JAMA is published with integrity, it’s because of Lundberg’s sterling conduct: There is not much of a system, a process, on which other, lesser beings can rely...

...George Lundberg is himself a medical doctor and an editor with 14 years’ experience at the magazine...

...It’s time for the peer-reviewed journals to undergo some rigorous peer review...

...How many times have you heard Tom Brokaw or Peter Jennings speak the ominous words, “The prestigious Journal of the American Medical Association reported today that . . . ” Note the verb: Leading medical journals report...

...For eight hours or more...

...This never happens...

...JAMA is practicing “trust-me” journalism, the scientific equivalent of Bob Woodward’s deathbed interviews...

...And in some cases, the images are reasonably close to reality...

...The history of the human race is clear on this point: Free societies do better when power is restrained by rules and conventions rather than by the unfettered judgments of individual men, however high-minded...

...The producer then explains the story to the broadcast’s managing editor/anchorman, whose first question is, “What’s the headline here...

...They sit around a horseshoe table in the middle of which stands a bare microphone...

...The president of the United States is lucky if he announces...

...3) Search for an authoritative source, a credible third party on which to hang the story...

...JAMA does not feel obliged to inform its readers when reviewers recommend against publication...

...But there’s peer review, and then there’s peer review...

...The editor then explains the study to his publicist, who then explains the story to a network-news producer...

...This happens only occasionally, but still too often: Journalistic careers are not built, after all, on thick files of “the sky is just fine” stories...

...This is the option of choice—and the source of serious problems now emerging in the scientific community...

...The dean of Yale doesn’t report...

...When dealing with peer-reviewed science—the point at which even skeptics and ideological opponents must be prepared to say as a society that the jury is now in and the truth is now out—laymen have a right to expect solid fact and sound process...

...Report is a heavy word, and one is hard pressed to think of another institution routinely entitled to use it...

...The other study, by Ph.D...

...More often these days, he charges, or if he’s having a bad verb day, he denies or shrugs off...

...It’s a marvelously reassuring phrase, summoning images of avuncular mentors looking over the shoulders of researchers at the bench, double-checking data, approving methodology...

...Even Colin Powell doesn’t report...

...First, the study is sent out to as many as 10 peers with credentials in the field...

...The distinguished British journal Lancet uses only two reviewers, but will not publish if both of them review negatively...

...It’s rare,” says editor Jerome Kassirer, “but we do it...

...The atmosphere is polite but tightly structured...

...Take the case of magazines invariably referred to as “leading medical journals”—the Journal of the American Medical Association (JAMA) and the New England Journal of Medicine, to name the two most prestigious...

...The publicist wants to be cooperative...

...And there is no rule that reviewers must recommend unanimously that a study be published...

...Another scientist, generally an admirer of Lundberg’s, told me that all editors play favorites and researchers must game the system to avoid reviews “by people who hate me...

...JAMA explains that anonymity secures a higher level of objectivity while insuring reviewers against professional reprisal...

...Peer review at this level is full-contact intellectual roller derby, and only the rigorous survive...

...It wouldn’t be right,” says staff editor Clair Thompson...

...Hypertension is the beneficiary of a happy confluence of motives between journal staffers and the news outlets they flak: The staffers wish to elevate themselves to “prestige” status for all the obvious reasons, and the media types seek to hang their sky-is-falling stories on “prestige” sources...

...And of course it’s the rare lawyer who assumes anonymous allegations to be intrinsically more credible than charges made in open court...

...2) Yield to the natural instinct and fire off a slightly hedged version of “the sky is falling...

...In fact, the sky may be rising...

...Woodward gets away with it because he’s Woodward, and Lundberg gets away with it because he’s running JAMA, which appears to speak for most American doctors...

...The poor speaker of the House claims, which is the journalistic equivalent of an FDA warning label...

...Reporters who wouldn’t think of taking handouts from, say, the Pentagon, snatch them from obscure medical journals...

...Neal B. Freeman is a Peabody Award-winning television producer for The Blackwell Corporation...

...The producer’s natural query is, “How far can we go...

...For the editor, this comes pretty close to absolute power, and we all know what absolute power can do...

...In Hypertension’s case, one of its own officials complained in an internal document that a recent press release was “as inflammatory a statement as can be imagined...

...The pope doesn’t report...

...The New England Journal of Medicine, which also uses two reviewers, will print studies panned by both reviewers...

...So far, so good—but then the exceptions begin to creep in...

...And then there’s the policy that reviewers are never identified with a particular study...

...One scientist who publishes frequently (but not in JAMA) treasures a note from a journal editor clipped to a file of negative reviews: “F— it, I like it and we’re going to publish...

...The fact that they are peer-reviewed casts the same tranquilizing spell over journalists as does an open FDA hearing...

...In the audience are professional rivals, investigators from related studies, corporate reps, Naderites, kibitzers of all sorts...

...Smith, says, roughly, that the sky is falling...

...They all seem to be “prestigious” or “authoritative...

...The practical options reduce to these: 1) Read the studies and try to make sense of them...

...Anybody who played Telephone in second grade will be able to calculate the odds that the researcher’s nuance survived this journey...

...It is a relatively new publication, eight years old, and serves a readership of less than 4,000...

...These studies can be about the air we breathe or the water we drink or the food we eat or the drugs we take...

...That inflammatory statement, of course, made the network news intact...

...The anchorman then takes 22 seconds to explain it to the rest of us...

...If you want to see your tax dollars effectively at work, check out a peer-review hearing at the Food and Drug Administration...