Scholarly Communication Guide

The following resources offer a critical look at traditional measurements of impact:

• Eugene Garfield - The History and Meaning of the Journal Impact Factor. Journal of the American Medical Association. v. 295, no. 1 (2006). (For more on the work and scholarship of Eugene Garfield, see the Garfield Papers Collection at the University of Pennsylvania) 
• Stephen Curry. Sick of Impact Factors. Occam's Typewriter (Blog). 2012. See also the follow-up to the post here. 
• Jason Priem and Bradley Hemminger. Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday. v. 15, no. 10 (2010).
• Bjorn Brems et al. Deep Impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience. (2013).
• San Francisco Declaration on Research Assessment (DORA) - statement crafted by group of editors and publishers of scholarly journals on the need to improve the way research outputs are measured and evaluated. Site includes links to news stories covering the DORA initiative. See Kent Anderson's post in the Scholarly Kitchen blog for a viewpoint from the other side on DORA.
• Adam Eyre-Walker and Nina Stoletzki. The Assessment of Science: The relative merits of post-publication review, the impact factor and the number of citations. PLoS Biology. (2013)
• "ScienceLive: Should We Ditch the Impact Factor?" - recorded webinar (2013)
• Caspar Chorus and Ludo Waltman. A Large-Scale Analysis of Impact Factor Biased Journal Self-Citations. PLoS ONE. (2016).
• Vincent Lariviere et al. A Simple Proposal for the Publication of Journal Citation Distributions. bioARxiv pre-print (2016).

In the past, peer reviewed journals have been the standard for measuring the quality and importance of scholarly literature. However, the growing number of journals and articles being published has placed significant strain on this process. Part of the problem is attributable to a rate of publication that is far outpacing the pool of available referees. With scant incentive available for referees in the form of credit in the tenure and promotion process, this shortage is expected to continue.

Despite its strong track record in filtering research, the peer review system has come to show signs of weakness.  The authors of the altmetrics manifesto point out that “peer review…is slow, encourages conventionality, and fails to hold reviewers accountable…[and] given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.” This prognosis is not good and is unsustainable over the long term. In response, new alternative forms of peer review have begun to develop that take a more open and transparent approach to the traditional peer review model:

Open Peer Review: Open peer review begins with an author or editor posting an unpublished manuscript online in a comment-enabled web environment, and then inviting peers to contribute comments and criticisms. The argument is that open peer review not only allows for transparency in the peer review process, but also enables a wider variety of input, including cross-discipline critique and more technical, "non-scholarly" input.

• An Introduction to Open Peer Review - Scientific American (2014)
• For a wonderful bibliography on alternative forms of peer review, including open peer review, please see the "Transforming Peer Review" bibliography.
• Presentation Slides - Open Peer Review: Bold Steps Toward Change in Scholarly Communication (2012)
• ImpactStory blog post on open peer review - what it is and how to become an effective open peer reviewer
• Open Review: A study of contexts and practices - white paper by Kathleen Fitzpatrick and Avi Santo

Post-Publication Peer Review:

• Altmetrics are a form of post-publication peer review -- see the Citation Metrics and Altmetrics tab on the left for more info
• Faculty 1000 - Identifies and evaluates the most important articles in biology and medical research publications
• PubPeer - tool for commenting on journal articles
• RetractionWatch - blog on retractions of scholarly articles, which is a function of post publication review