This week the Public Library of Science announced, with little fanfare, that it was publishing a new journal without the benefit of peer review. The implications of that decision are extremely important for the future of scientific communication.
To explain why that is so, I'm reprinting an article I wrote in 1998, called Seeing the Light. Tomorrow I'll talk about the advances in the peer review process as it has grown and changed over the years (and how I was wrong about what would be the format of most scholarly research reporting at least so far), and the exciting ways archaeology and the related fields have involved the capabilities of the Internet to create better science communication.
When I hang around the Society for American Archaeology meetings, I generally spend some time talking to several colleagues about the Internet and its potential––one old friend called me an evangelist for the World Wide Web, and I'm afraid that's not far off. What I heard, I've heard before. People––scientists especially––don't use the Internet for several reasons, some of them good, some of them bad.
Some find the web hopelessly trivial. One person told me the problem with the web is nobody ever "reads" a web site; they "check it out." Others speak of how uneasy the ease of access made them; and still others mention the innumerable terrible sites that are "out there." Some even say that the web will not be useful so long as "just anyone" can post there; and that the proliferation of bad web sites makes good web sites pointless.
There simply is no peer review on the web, they say. Unlike in scholarly journals, there is no mechanism to assure that the quality of the material that gets published is acceptable to the scientific community.
Ah, but they are incorrect. There is a way to do peer review on the web––but like the presence of the web is changing our society, putting peer review on the web is going to change the face of peer review drastically, and to my mind, in a manner long overdue.
I've been associated with the editorial side of academic journal publishing off and on since 1980. One thing I do know about academic publishing, is that the peer review system is in serious need of an overhaul. Too many times, the ostensible reasons and the political reality of peer review are excessively disparate. Theoretically, the reason for peer review is to further science, right? But, too often, reviews are cursory or negative or even positive without substance, and directed at the person's body of work (or simply the person's personality) rather than the specific issue at hand. I think the big culprit is blind reviewing. Most of all, the notion of blind reviewing is ludicrous. If a reviewer is cognizant of an issue enough to review a paper, s/he must know who wrote the paper, and alternatively, the author can probably guess the names of the selected reviewers. So the only thing that's truly blind in a "blind review" is the ownership of specific comments made on a specific paper; the comments that are the most cogent, the most interesting, and the single most important factor in determining the publication fate of a document.
Current Anthropology has the right idea. Open reviews, signed, with bibliographic references. Why can't we do that on the web? My idea is called "salon review" or "collaborative peer review," because it combines an open review system with the dynamic capabilities of web design and authorship.
Here's how I'd imagine it would work. Let's say there's a web site at the University of the Ukraine, and it's dedicated to the study of methods of illumination throughout history and prehistory. Call it Seeing the Light. The site is maintained by a couple of graduate students, but many researchers in the field of lighting are aware of the site and regularly contribute links and course syllabi and images and bibliographies. Professor L. from Sydney State University has developed a web site on whale oil lamps; she sends the URL to the web site and requests a review. Seeing the Light puts an index page on its site, and reviewers are requested to investigate Professor L's site and comment on it. The reviewers may give substantive comments, make suggestions, indicate explanatory links, argue with the conclusions, but they must sign the reviews. The reviewers are probably part selected in advance by Professor L and Seeing the Light, and part simply walk-ins from the readership. Professor L may respond to these comments, again signing them. You'd still need an editor to keep everybody honest, of course, and hold the reviews to acceptable levels; the reviews themselves would remain archived on Seeing the Light, and available for viewing by anyone who wants to see them. At the end of the process (maybe during?), Professor L puts a button on her site indicating that there is a "salon review" located at Seeing the Light.
Everybody wins: Professor L gets to improve her web site; the reviewers get to indicate how insightful they are; people who don't know very much about lighting find a reliable source to learn something; and geeks like you and me can read the comments and learn something interesting about the academic process.
And, best of all, the Internet takes one huge step towards becoming the Library at Alexandria.
Middle: Fresnel, taken by cikaga jamie
Bottom: Globe Chandelier (with Monkey), Minneapolis, taken by Marie Richie
To explain why that is so, I'm reprinting an article I wrote in 1998, called Seeing the Light. Tomorrow I'll talk about the advances in the peer review process as it has grown and changed over the years (and how I was wrong about what would be the format of most scholarly research reporting at least so far), and the exciting ways archaeology and the related fields have involved the capabilities of the Internet to create better science communication.
The Hopelessly Trivial Internet
When I hang around the Society for American Archaeology meetings, I generally spend some time talking to several colleagues about the Internet and its potential––one old friend called me an evangelist for the World Wide Web, and I'm afraid that's not far off. What I heard, I've heard before. People––scientists especially––don't use the Internet for several reasons, some of them good, some of them bad. Some find the web hopelessly trivial. One person told me the problem with the web is nobody ever "reads" a web site; they "check it out." Others speak of how uneasy the ease of access made them; and still others mention the innumerable terrible sites that are "out there." Some even say that the web will not be useful so long as "just anyone" can post there; and that the proliferation of bad web sites makes good web sites pointless.
There simply is no peer review on the web, they say. Unlike in scholarly journals, there is no mechanism to assure that the quality of the material that gets published is acceptable to the scientific community.
Ah, but they are incorrect. There is a way to do peer review on the web––but like the presence of the web is changing our society, putting peer review on the web is going to change the face of peer review drastically, and to my mind, in a manner long overdue.
The Problems with Peer Review
I've been associated with the editorial side of academic journal publishing off and on since 1980. One thing I do know about academic publishing, is that the peer review system is in serious need of an overhaul. Too many times, the ostensible reasons and the political reality of peer review are excessively disparate. Theoretically, the reason for peer review is to further science, right? But, too often, reviews are cursory or negative or even positive without substance, and directed at the person's body of work (or simply the person's personality) rather than the specific issue at hand. I think the big culprit is blind reviewing. Most of all, the notion of blind reviewing is ludicrous. If a reviewer is cognizant of an issue enough to review a paper, s/he must know who wrote the paper, and alternatively, the author can probably guess the names of the selected reviewers. So the only thing that's truly blind in a "blind review" is the ownership of specific comments made on a specific paper; the comments that are the most cogent, the most interesting, and the single most important factor in determining the publication fate of a document.Current Anthropology has the right idea. Open reviews, signed, with bibliographic references. Why can't we do that on the web? My idea is called "salon review" or "collaborative peer review," because it combines an open review system with the dynamic capabilities of web design and authorship.
A Model for Internet Peer Review
Here's how I'd imagine it would work. Let's say there's a web site at the University of the Ukraine, and it's dedicated to the study of methods of illumination throughout history and prehistory. Call it Seeing the Light. The site is maintained by a couple of graduate students, but many researchers in the field of lighting are aware of the site and regularly contribute links and course syllabi and images and bibliographies. Professor L. from Sydney State University has developed a web site on whale oil lamps; she sends the URL to the web site and requests a review. Seeing the Light puts an index page on its site, and reviewers are requested to investigate Professor L's site and comment on it. The reviewers may give substantive comments, make suggestions, indicate explanatory links, argue with the conclusions, but they must sign the reviews. The reviewers are probably part selected in advance by Professor L and Seeing the Light, and part simply walk-ins from the readership. Professor L may respond to these comments, again signing them. You'd still need an editor to keep everybody honest, of course, and hold the reviews to acceptable levels; the reviews themselves would remain archived on Seeing the Light, and available for viewing by anyone who wants to see them. At the end of the process (maybe during?), Professor L puts a button on her site indicating that there is a "salon review" located at Seeing the Light.Everybody wins: Professor L gets to improve her web site; the reviewers get to indicate how insightful they are; people who don't know very much about lighting find a reliable source to learn something; and geeks like you and me can read the comments and learn something interesting about the academic process.
And, best of all, the Internet takes one huge step towards becoming the Library at Alexandria.
Photo credits
Top: Street Light, Kings College, Cambridge, England, taken by monkeyleaderMiddle: Fresnel, taken by cikaga jamie
Bottom: Globe Chandelier (with Monkey), Minneapolis, taken by Marie Richie
Comments
The most startling statement I’ve read about peer-review in an archaeological journal was in Cornelius Holtorf’s paper Beyond crusades: how (not) to engage with alternative archaeologies, which cited an article stating that agreement between reviewers is indistinguishable from chance. Obviously that defies common sense, but taking time read what he cites shows that it is likely to be true and looking further it’s not an isolated case. Similar studies in varying fields have shown no, or a poor significance between of agreement between reviewers compared to chance. I don’t know of any such studies in archaeology or history, but to pick a victim at random, is the American Journal of Archaeology a good journal because of peer-review or despite it? Is there no correlation at all? What about less prestigious journals. Would good results from a survey of the AJA be meaningful in assessing other journals? The evidence to answer these questions seems to be lacking,
Worse, there’s circumstantial evidence to suggest that faith in peer-review is stronger than it should be. I would argue that this shown by the way peer-review seems to have been abandoned by some publishers and no-one has noticed. The veracity of that last sentence depends on what you accept as peer-review.
Peer-review in a respectable journal usually takes the form of a double-blind review. The reviewers are selected by the editor rather than the author. If the author selected the reviewers, and supplied these reviews to the editors would this count as peer-review? I was at a seminar for post-graduate training where we were informed of the realities of publishing a monograph. We can expect to produce camera-ready copy, our own reader’s reports and in exchange the book will be sold with a print run of 200 copies at a price in excess of £40-£60. I asked the speaker if I were doing the work of producing the copy, getting the reader reports and so on, why should I want to publish with a traditional publisher who will price my book out of the reach of many potential readers? I could release it as a PoD book and ebook. The answer I was given was that a commercial publisher has a sales force, a reputation and acts as a gatekeeper. To publish in Tauris Academic Studies you need to supply your own publicity text too [PDF].
Whether or not a TAS monograph should be accorded the same status as a peer-reviewed piece of work depends on how effective you think a publisher which relies on the author supplying the reader reports is as a gatekeeper. In the case of Tempus , an imprint of the History Press anyone publishing to benefit from the press’s reputation has been dealt a major blow recently. The lack of many such complaints would suggest that scholars are usually honest and competent when providing their own reports, and that commercial publishers can adequately assess the value of texts. This raises questions about the need for peer-review.
One response would be to abandon peer-review. I think this could be a bad idea. This proposal for open review is much more helpful. For a start it places the work in a context of a discussion, which is what any academic paper aspires to be. Peer-review can have positive aspects, even if it is a case of showing where confusion occurs. A reviewer may request a clarification of a point, but no one reviewer can anticipate where misunderstanding will occur. Open discussion would provide the reader with multiple views of the same problem, which may be more illuminating. In this case an author asking for a couple of comments from invited scholars would not necessarily be a bad thing, given these comments would be open to view and question.
It also places the importance on peer-review. In this case the author and reviewers are clearly peers rather than an imbalanced relationship of supplicant and expert, which as Hirst points out above, is likely even with anonymous review seeing as the reviewer will probably be able to work out who the author is. There are problems if the author and reviewer clearly aren’t peers. For instance someone may hesitate to comment on a prominent professor’s work if they are a graduate student, even if they feel that the professor has made an elementary mistake, like saying north and south are equivalent same directions.
Additionally it removes a way of preserving gender anonymity, which would seem to be a problem. I should make clear I’ve never seen any evidence of anti-female bias when any of my papers have been reviewed by a journal, but if I did my beard would probably fall out with the shock. Nonetheless there is doubt about whether or not review is gender-neutral. I would hope that open review could not eliminate this problem, but will it make it more visible and thus easier to account for, or will misogynists find other ways to hide their behaviour? Evidence can be mustered to show that a double-blind review can improve the chances of a female author being published. In these cases there are costs as well as opportunities to open review and a lack of consensus to the balance means that double-blind review will have a place in academia for a long while.
Some leadership from prominent societies would be welcome, but it’s possible there’s not really any incentive. If you’re at the top then the only direction any change in method could take you is down. Perhaps societies which sponsor two journals could experiment with an open review system and see if it works better.
An interesting article, which seems a bit predictive of PDQ, even down to the note of a review on the site.
I am interested in your opinion about putting the reviews on the site instead of centralizing them.
I would also like your thoughts on how a site, like “seeing the light” would get started, and how it would reach a significant level of popularity/notoriety necessary to become a respected and valued form of review.
Well, my original thought was that the site would grow and develop from a member of the community who is already interested in the topic. In other words, someone already familiar with the topic and working and published in it (or, more likely, a student of hers or his) would start a clearinghouse website for such a project, and notify others in the ‘community’ that it exists. I do mean (relatively) micro-studies, you realize, not broad topics like “archaeology” or “paleontology” but maybe “Middle-Upper Paleolithic Transition” or “Stable Isotope Analysis” or “Neolithic Agriculture”. I could be wrong of course — I don’t really honestly know what level would be useful, but I go to meetings all the time and in a specific session there are always a group of people who are all working on the same topic, know each other and try to keep up if they can. That’s the kind of thing I find interesting, the discussions outside of the published academic journals. And I perceive, anyway, a difficulty in maintaining groupthink (pardon the Orwell reference) except at the meetings.
The ‘popularity’ of a particular website is, to be quite blunt, at the mercy of Google, but if your purpose is to facilitate discussion on a subject, the development of content will put that subject into search engines.
Today (bear in mind the original article is a decade old now) I don’t think the reviews have to be necessarily at the clearinghouse site, as long as they’re linked there.
Do it! I’d surely like to see such a site in action.