Measures of Overall Performance

Distinct regulatory philosophies confine and direct the task of setting standards. Overlapping but different decisionmaking rules characterize the private and public sectors. But outcomes are not as easily differentiated as the philosophies and operating procedures that produce them. There are four reasons why it is difficult to draw policy conclusions from regulatory philosophy alone. First, regulatory philosophy con-

strains but does not determine outcomes. It leaves plenty of room for discretionary decisions. While regulatory philosophy keeps UL from addressing the creosote problem, it leaves considerable latitude in setting other requirements in the standard. These discretionary decisions result in a variety of outcomes. UL 1482 is probably too strict on glass doors, but too lax on metal grates. Similarly, the stability requirements in ANSI Z21.11.2 are undoubtedly stringent, while the clothing ignition test is not.

Second, however distinct these philosophies, the universe of potential outcomes overlap. The public and private sectors are capable of producing very similar outcomes. The FAA and NFPA, acting practically in isolation of each other, developed several similar provisions for aviation fire safety. There were pockets of agreement in the other cases as well—areas where both sectors agreed on the content of particular provisions. The CPSC approved the basic technical and performance provisions in UL 1482 and most of those in Z21.11.2. OSHA did not quarrel with the NFPA provisions for fire safety in grain elevators.

Third, even when regulatory philosophies dictate distinct regulatory outcomes, the normative implications are unclear. The same regulatory philosophy can perform well in one circumstance and poorly in another. Paternalism led the CPSC to adopt a seemingly desirable standard for gas space heaters, but an ill-advised one for woodstoves. Similarly, refusing to make standards retroactive practically gutted NFPA 61B, but had little adverse effect on NFPA 408 or UL 1482. The only way to evaluate regulatory philosophies, then, is by examining outcomes and seeking to understand the conditions under which they perform best.

Finally, it is difficult to draw substantive conclusions about regulatory philosophy because it is multifaceted and signals sometimes conflict. Safety standards are usually polycentric in nature, raising an assortment of complex issues. Under these circumstances, the manifestations of regulatory philosophy are varied and sometimes contradictory. The engineering ethic, for example, helps explain UL's reluctance to address the creosote problem in woodstoves, but it also explains its more stringent structural requirements. In sum, regulatory philosophy shapes the standards-setting process in several important ways, but the normative implications of these differences cannot be stated in simple terms that clearly favor one form over the other.

What is needed, then, is a measure of overall performance to facilitate the comparison of public and private outcomes. Whether that is possible strikes at the heart of a long-standing debate about process

versus substance. Dispensing with pesky substantive issues, those disposed to the legal perspective turn exclusively to process (that perspective is examined in chapter 11). Those venturing substantive conclusions usually couch the analysis in terms of either strictness or reasonableness. Both concepts encompass important social considerations. Strictness generally refers to absolute benefits; reasonableness, to the relationship between benefits and costs. These frames of reference differ significantly, creating conflicting impressions of many standards. Public standards are usually considered stricter, but less reasonable, than private ones. That is, they probably generate more absolute benefits than private standards, but at a cost higher than many considered acceptable. By contrast, almost no one argues that private standards impose unreasonable costs. But reasonableness is thought to come at the expense of strictness. Obviously, these substantive measures should be merged in some manner. Standards should be compared through an aggregation of strictness and reasonableness.

In theory, cost-benefit analysis facilitates the task. But there are practical and philosophical objections to even the roughest forms of cost-benefit analysis. The dearth of reliable data, particularly on benefits, dims the potential for such analysis. Part of the problem is forecasting. Much depends on implementation. For example, OSHA's "action level" for grain elevator housekeeping will produce benefits if it is implemented through a reasonable inspection scheme. It will foster unreasonableness if implemented poorly. On a more philosophical level, cost-benefit analysis requires that monetary values be placed on life and limb, something both methodologically difficult and politically explosive. Cost-benefit analysis is out of the question for current evaluative purposes. The data necessary to support it are not available. Nothing remotely resembling cost-benefit analysis was undertaken by any of the private standards-setters. And the economic analyses conducted by OSHA, the CPSC, and the FAA are easily faulted for reasons elaborated in the case studies.

One way to evaluate outcomes directly despite the significant uncertainties about costs and benefits is to indulge in liberal evidentiary presumptions and seek only to separate the obviously bad outcomes from the possibly good ones. In those terms, a standard is within the "zone of reasonableness" if there is credible evidence that benefits are (1) nontrivial and (2) not significantly in excess of costs. This overly inclusive notion of "reasonableness" avoids the almost intractable disputes about the precise magnitude of uncertain benefits and costs. This approach also permits a surprising number of normative conclusions

about the cases. The results, based on the summary evaluation in the case studies, are summarized and explained below:

On the public side, two of the standards were clearly too strict. There was no evidence that the CPSC's woodstove labeling rule would result in any measurable benefits; and fire protection engineers agree that the FAA's standard for aviation fire safety is very unlikely to generate benefits in excess of cost. The other two public standards were well within the zone of reasonableness. The need for grain elevator regulation is supported by the evidence, and the OSHA standard, although it could be improved, is reasonable in several respects. The CPSC's standard for gas space heaters is probably the best of the bunch. It seems to be responsible for the widespread use of ODS technology, an inexpensive and effective method for dealing with a problem of uncertain dimensions. The private sector was also divided. Two of its standards were clearly undesirable, although one was too strict and the other too lax. The NFPA went even further overboard than the FAA on aviation safety. NFPA officials privately admit that these standards are not supportable in economic terms. The NFPA also missed the mark on grain elevators, adopting a standard that is so weak that it barely addresses the most serious problem: grain dust.

Two tentative conclusions can be drawn from this normative evaluation. First, neither sector is clearly better than the other. Both sectors appear as capable of failing as they are of succeeding, although the public sector seems more likely to be overly strict, while the private sector is more likely to be too lax. Nevertheless, private standards should not be rejected solely on the theory that they tend to be underprotective . Sometimes they are not. This analysis also suggests that there is no simple answer to the question whether the public or private approach is generally better. Private sector behavior is too varied. NFPA 408 and 61B are cases in point. Second, the cases suggest that particular issues can foster distinct patterns of regulatory behavior. It is

no coincidence that both the public and private standards for aviation fire safety seems "overly strict."

Both of these observations underscore the importance of understanding the reasons behind the observed results—the subject of the next chapter. Given the unusual nature of the "paired" case studies, however, it is appropriate to consider first whether these observations are intrinsic to overlapping standards. In other words, to what extent is the behavior of either sector affected by the activities of the other?