Filtering Costs

As I noted earlier, one of the critical functions of the academic publishing system is to filter. Work cannot be cumulative unless authors have some faith that prior literature is accurate. Peer review helps ensure that work meets appropriate standards for publication.

There is a recognized pecking order among journals, with high-quality journals in each discipline having a reputation for being more selective than others. This pecking order helps researchers focus their attention on areas that are thought by their profession to be particularly important.

In the last 25 years many new journals have been introduced, with the majority coming from the private sector. Nowadays almost anything can be published somewhere -the only issue is where. Publication itself conveys little information about quality.

Many new journals are published by for-profit publishers. They make money by selling journal subscriptions, which generally means publishing more articles. But the value of peer review comes in being selective, a value almost diametrically opposed to increasing the output of published articles.

I mentioned above that one of the significant implications of electronic publication was that monitoring costs are much lower. It will be possible to tell with some certainty what is being read. This monitoring will allow for more accurate benefit-cost comparisons with respect to purchase decisions. But perhaps even more significantly, it will allow for better evaluation of the significance of academic research.

Citation counts are often used as a measure of the impact of articles and journals. Studies in economics [Laband and Piette 1994] indicate that most of the citations are to articles published in a few journals. More articles are being published, a smaller fraction of which are read [de Sola Pool 1983]. It is not clear that the filtering function of peer review is working appropriately in the current environment.

Academic hiring and promotion policies contribute an additional complication. Researchers choose narrower specialties, making it more difficult to judge achievement locally. Outside letters of evaluation have become worthless because of the lack of guarantees of privacy. All that is left is the publication record and

the quantity of publication, whose merits are easier to convey to nonexperts than quality of publication.

The result is that young academics are encouraged to publish as much as possible in their first five to six years. Accurate measures of the impact of young researchers' work, such as citation counts, cannot be accumulated in this short a time period. One reform that would probably help matters significantly would be to put an upper limit on the number of papers submitted as part of tenure review. Rather than submitting everything published in the last six years, assistant professors could submit only their five best articles. This reform would, I suggest, lead to higher quality work and higher quality decisions on the part of review boards.

Dimensions of Filtering

If we currently suffer from a glut of information, electronic publication will only make matters worse. Reduced cost of publication and dissemination is likely to make more and more material available. This proliferation isn't necessarily bad; it simply means that the filtering tools will have to be improved.

I would argue that journals filter papers on two dimensions: interest and correctness. The first thing a referee should ask is, "is this interesting?" If the paper is interesting, the next question should be, "is this correct?" Interest is relatively easy to judge; correctness is substantially more difficult. But there isn't much value in determining correctness if interest is lacking.

When publication was a costly activity, it was appropriate to evaluate papers prior to publication. Ideally, only interesting and correct work manuscripts would undergo the expensive transformation of publication. Furthermore, publication is a binary signal: either a manuscript is published or not.

Electronic publication is cheap. Essentially everything should be published, in the sense of being made available for downloading. The filtering process will take place ex post, so as to help users determine which articles are worth downloading and reading. As indicated above, the existing peer review system could simply be translated to this new medium. But the electronic media offer possibilities not easily accomplished in print media. Other models of filtering may be more effective and efficient.

A Model for Electronic Publication

Allow me to sketch one such model for electronic publishing that is based on some of the considerations above. Obviously it is only one model; many models should and will be tried. However, I think that the model I suggest has some interesting features.

First, the journal assembles a board of editors. The function of the board is not just to provide a list of luminaries to grace the front cover of the journal; they will actually have to do some work.

Authors submit (electronic) papers to the journal. These papers have three

parts: a one-paragraph abstract, a five-page summary, and a 20-to 30-page conventional paper. The abstract is a standard part of academic papers and needs no further discussion. The summary is modeled after the Papers and Proceedings issue of the American Economic Review: it should describe what question the author addresses, what methods were used to answer the question, and what the author found. The summary should be aimed at as broad an audience as possible. This summary would then be linked to the supporting evidence: mathematical proofs, econometric analysis, data sets, simulations, and so on. The supporting evidence could be quite technical and would probably end up being similar in structure to current published papers.

Initially, I imagine that authors would write a traditional paper and pull out parts of the introduction and conclusion to construct the summary section. This method would be fine to get started, although I hope that the structure would evolve beyond this.

The submitted materials will be read by two to three members of the editorial board who will rate them with respect to how interesting they are. The editors will be required only to evaluate the five-page summary and will not necessarily be responsible for evaluating the correctness of the entire article. The editors will use a common curve; e.g., no more than 10% of the articles get the highest score. The editorial score will be attached to the paper and be made available on the server. Editors will be anonymous; only the score will be made public.

Note that all papers will be accepted; the current rating system of "publish or not" is replaced by a scale of (say) 1-5. Authors will be notified of the rating they received from the editors, and they can withdraw the paper at this point if they choose to do so. However, once they agree that their paper be posted, it cannot withdrawn (unless it is published elsewhere), although new versions of it can be posted and linked to the old one.

Subscribers to the journal can search all parts of the on-line papers. They can also ask to be notified by e-mail of all papers that receive scores higher than some threshold or that contain certain keywords. When subscribers read a paper, they also score it with respect to its interest, and summary statistics of these scores are also (anonymously) attached to the paper.

Since all evaluations are available on-line, it would be possible to use them in quite creative ways. For example, I might be interested in seeing the ratings of all readers with whom my own judgments are closely correlated (see Konstan et al. [1997] for an elaboration of this scheme). Or I might be interested in seeing all papers that were highly rated by Fellows of the Econometric Society or the Economic History Society.

This sort of "social recommender" system will help people focus their attention on research that their peers-whomever they may be-find interesting. Papers that are deemed interesting can then be evaluated with respect to their correctness.

Authors can submit papers that comment on or extend previous work. When

they do so, they submit a paper in the ordinary way with links to the paper in question as well as to other papers in this general area. This discussion of a topic forms a thread that can be traversed using standard software tools. See Harnad [1995] for more on this topic.

Papers that are widely read and commented on will certainly be evaluated carefully for their correctness. Papers that aren't read may not be correct, but that presumably has low social cost. The length of the thread attached to a paper indicates how many people have (carefully) read it. If many people have read the paper and found it correct, a researcher may have some faith that the results satisfy conventional standards for scientific accuracy.

This model is unlike the conventional publishing model, but it addresses many of the same design considerations. The primary components are as follows:

• Articles have varying depths, which allows them to appeal to a broad audience as well as satisfy specialists.

• Articles are rated first with respect to interest by a board of editors. Articles that are deemed highly interesting are then evaluated with respect to correctness.

• Readers can contribute to the evaluation process.

• The unit of academic discourse becomes a thread of discussion. Interesting articles that are closely read and evaluated can be assumed to be correct and therefore serve as bases for future work.