Blog

How Peer Review at Top-tier Journals is Flawed

[del.icio.us] [Digg] [Facebook] [Mixx] [Reddit] [StumbleUpon] [Twitter] [Email]

October 13th, 2011

How peer review at top-tier journals is flawed.

The long-respected system of peer review is increasingly under attack. According to many scientists and academics, peer review is ineffective and time consuming at best and a tempting venue for favoritism and plagiarism at worst. The main problems with peer review stem from two basic issues: inconsistency in the reviewing process itself and too many submissions to top-tier journals. According to Drummond Rennie, executive editor of the Journal of the American Medical Association, “If peer review was a drug, it would never be allowed on the market.”
 

Problems with the Reviewing Process

Peer review is a hazy concept. Every journal and grant-giving foundation has a different definition of what constitutes peer review but most require it before an article is even considered for publication. Most publishers agree that peer review has something to do with a third party reviewing a manuscript or grant proposal, but after that the details are often fuzzy. Major points of variation in peer review include how many peers review an article, who those peers are and what the conditions of the review are.

Articles are usually reviewed by between two and four people who are chosen by an editor to recommend whether an article is fit for publication. These peers are generally—but not always—experts on the article’s subject and are asked to read for consistency, mistakes, misinformation and evidence of fraud. Some journals prefer for these peers to be anonymous and some have a fully transparent review process but either way, the review process usually does not accomplish its goals. 

Peers miss an amazing amount of mistakes, inconsistencies and outright fraud. Generally unpaid, peers have little motivation to give articles the thorough dissection they need. Peer reviewers are often asked to review far more articles than their schedules allow, a situation documented by Daniel Myers in a recent essay about his experiences as a peer reviewer. As a result, peers tend to breeze through an article, decide whether it sort of makes sense or not and pass it along to the editor with a recommendation based more on what they had for breakfast than the integrity and quality of the research. 

The recently discredited work of Woo-Suk Hwang at Seoul National University is an excellent example. Woo-Suk Hwang’s work went through a rigorous peer review process that entirely failed to catch the researcher’s massive amounts of fraud. Peers caught up in today’s publish-or-perish academic environment are often too overwhelmed with their own department’s writing demands to devote the correct amount of time and attention to the articles of their fellow scientists.

Submission Increases and Stylistic Changes

Authors working in academia are currently under intense pressure to publish in top-tier journals. In the past, researchers were free to submit articles to the most suitable journals instead of simply the most famous ones. Less-than-groundbreaking work could find a home in second and third-tier journals and authors had a low rate of rejected articles. Today, editors at top journals are overwhelmed with submissions that are unsuitable for their publications and authors are constantly frustrated by the rejections of well researched and thoughtfully written articles that simply aren’t appropriate for the most widely read publications. Much as not every short story is ideal for Harpers and the New York Times only has room for a certain number of editorials, top-tier journals are similarly limited by space and focus.

The editor-in-chief of Science, Donald Kennedy, states that his publication rejects at least 6,000 papers every year and submissions are increasing steadily. With so many rejections, editorial staff members are required to use more of their already over-stuffed workdays addressing author complaints and sorting through claims of impartiality and plagiarism. 

Authors are going to increasingly desperate measures to get editors to publish their work. In recent years, there has been a serious uptick in the amount of time scientists spend networking with editors in attempt to make a personal connection that will make rejection less likely. Another tactic is to exaggerate the results of a study, creating a flashier article at the expense of good science. Articles that have any link to human diseases are always more popular and writers have been known to stress very tenuous links between their research and human health just to appeal to editors.

Possible Solutions

In theory, peer review is a wonderful idea. The system of allowing an author’s cohort to review her work for flaws is sound and worked reasonably well for a number of years. When universities began foisting ridiculous publishing demands on their researchers, the system began to break down. Academics are now generally required to publish at least five articles per year, one of which is in a major journal. Considering how many more academics than top-tier journals there are, this is obviously unreasonable. Peer reviewers are often these same academics—constantly struggling to find enough time for their own work, let alone a colleagues article. To improve the peer review process, universities first need to relax their researchers’ publishing quotas.

The peer review system has never been standardized and that would be an excellent next step toward saving it. Creating a basic training program for reviewers, a universal form for reviews and a system to deal with accusations of favoritism and plagiarism in the review process would be useful. Deciding whether reviews for individual publications would be anonymous or transparent would also be a smart move as would limiting the number of reviews a peer can do per year.

The peer review system is deeply flawed but salvageable. Authors and publishers are losing confidence in the whole system for good reason, and many authors are now pushing for its abandonment. Peer reviews is an essential control for maintaining the integrity and quality of research and those interested in saving it need to make changes sooner than later to prevent the breakdown it’s headed toward.

 

Leave a Reply