Chapter Three
Research Design

Our research is guided by two assumptions: impartiality holds the key to distributive justice, and imperfect information, of a particular kind, generates impartiality. Experimental subjects are confronted with a laboratory situation in which they are ignorant of their own self-interest. Under these conditions they, as a group, choose a principle of distributive justice to govern their rewards. To ensure that our experimental design did not rig the results, we checked to see how sensitive the results were to the specific experimental conditions. Moderate changes in the design should not have produced radically different results. To ensure against cultural bias, and hence nongeneralizability of the results, we sampled more than one national and cultural population. The various experimental conditions are displayed in Table 1, which also indicates where the different variants of the experiment were run. These variations are explained in this chapter.

Basic Experimental Design

The first tier of experiments (five varieties) was designed to test choices of principles of distributive justice under approximate conditions of impartiality. Though not perfectly similar to the

constructs in the philosophical discussions, they were designed to help us examine the validity of the empirical assumptions at the core of the normative debate. We attempted to determine which principles of distributive justice, if any, subjects would agree on. To appreciate the content and thrust of the experiments, it is useful to follow the broad outlines of the process

experienced by the subjects. Details of the experiments are discussed later in this chapter.

For each group we solicited, as subjects, five students from university classrooms. We then escorted them to a room where the experiment was to be conducted. When the subjects were seated (each at his or her own desk), we greeted them and presented each of them with a booklet to read. The booklet began with a brief sketch of four specific principles: Rawls's difference principle, characterized as maximizing the floor income; Harsanyi's idea, depicted as maximizing the average income; and two mixed principles that were discussed in the previous chapter—constraining the average income by setting a floor or a range.^[1] Although our immediate goal was to ensure that the subjects understood the principles, it was important that we not bias them by our description of the problem. So we simply sketched the principles as follows:

To illustrate such principles consider the following four notions of justice:

1.MAXIMIZING THE FLOOR INCOME

The most just distribution of income is that which maximizes the floor (or lowest) income in the society .

This principle considers only the welfare of the worst-off individual in society . In judging among income distributions, the distribution which ensures the poorest person the highest income is the most just . No person's income can go up unless it increases the income of the people at the very bottom.

2.MAXIMIZING THE AVERAGE INCOME

The most just distribution of income is that which maximizes the average income in the society .

[1] As indicated in Table 1, we had a variation of the experiment in which the notion of justice was downplayed. In those experiments certain replacements were made in the booklet. These included, for example, substituting "rules for distributing the monetary gains and losses" for "principles of justice." Given the fixed size of the group (five subjects), some principles become redundant. For example, under these circumstances, maximizing the average income is the same as maximizing total income.

For any society maximizing the average income maximizes the total income in the society.

3.MAXIMIZING THE AVERAGE WITH A FLOOR CONSTRAINT OF $___

The most just distribution of income is that which maximizes the average income only after a certain specified minimum income is guaranteed to everyone .

Such a principle ensures that the attempt to maximize the average is constrained so as to ensure that individuals "at the bottom" receive a specified minimum. To choose this principle one must specify the value of the floor (lowest income).

4.MAXIMIZING THE AVERAGE WITH A RANGE CONSTRAINT OF $___

The most just distribution of income is that which attempts to maximize the average income only after guaranteeing that the difference between the poorest and the richest individuals (i.e., the range of income) in the society is not greater than a specified amount .

Such a principle ensures that the attempt to maximize the average does not allow income differences between rich and poor to exceed a specified amount. To choose this principle one must specify the dollar difference between the high and low incomes.

Of course, there are other possible principles, and you may think of some of them.

After this minimal introduction to the principles, we asked subjects to rank the principles and to indicate how secure they felt about that ranking. Then we gave them a bit more detail about the problem they were to face by having them do a sample problem (see Table 2). The problem involved four possible patterns of income distribution, each of which had five income classes. (The average income was not given to the subjects at this point to draw their attention to the need for information on the population distribution across income classes.)

The subjects were then led through the consequences of applying each of the four principles. They were reminded both that their choice would (later) govern their payoffs and that they could not be sure which income class they would actually be in:

In making this choice, recall that your choice will yield you a payoff. How will your choice determine your pay? If, for example, you choose (a) to maximize the floor or low income, you will have picked that distribution (from the four in the sample question) which has the highest floor. (In this case distribution 4.) This ensures that you would get at least $15,000 if you were to be the worst-off individual. This is the most that a member of the low-income class could get from any of these distributions.

Subjects then read more about the principles of justice that we had placed on their (otherwise open) agenda. After all, we could not presume they had a feel for such things as the average income and how that might relate to the "height" of the minimum income. They then took a short test about various aspects of these principles. If they did not pass the test, they were required to retake a second variant of the test. Passing the test was a precondition to continuing in the experiment.^[2] After the test, subjects were again polled to reveal their rankings of the four principles.

After revealing their rankings of the principles, the subjects continued to explore the nature and implications of the principles, this time experientially. In this section of the experiment, they earned money by making choices in four situations like the one presented in Table 2 (with the average incomes specified). Each subject chose from among the four principles, and each of their choices led to the selection of an income distribution. Subjects knew that they would be randomly assigned to an income class within the distribution they had chosen.^[3] These assignments were made by having the subject reach into a bag and pull out a chit (see Table 3), which showed two things: their income-class assignment (and hence their payoff) and also what they would have received under each of the other

[2] Individuals often failed the first time, but subsequent failures were not frequent. We helped those with difficulties by answering questions about the principles they showed evidence of misunderstanding in the test.

[3] In part because of time constraints, in the experiments involving production (explained later in this chapter), this part of the experiment was excised. Also note that in the production experiments income was tied to work and not just assignment to income class.

principles had they chosen it.^[4] Thus they were familiarized not only with the implications of their particular choice but also with the consequences of different choices. Real monetary payoffs were calculated as $1 for every $10,000 of family income shown in the tables. The amount of money subjects were to

[4] The students were not able to discern the distribution of chits in the bags. Each time the subjects kept the chits so that they could reflect on the consequences of their choices. Judging from the discussions later in the experiment, this procedure had considerable pedagogic impact.

receive was entered on their payoff sheet after each selection in order to make the payoff concrete. The idea here was to get them to experience the consequences of their choices of principles. After the four choices were made, subjects were again polled regarding their rankings of the principles.

Subjects then proceeded to the second part of the experiment. It involved a collective, or group, choice of principle. In that part they were given a chance to decide together, after a

discussion, on a principle of justice.^[5] Again (in the nonproduction experiments), the fact that they would be randomly assigned to an income class was emphasized. The subjects were informed that there was a set of payoff vectors, each consisting of five income classes. The vectors would be used to determine the final payoffs to the subjects.^[6] However, they knew neither the number nor the content of these vectors. If they reached unanimous agreement on a principle, the chosen principle would be used to filter the payoff vectors available. For example, if they chose maximum income only the subset of vectors corresponding to that principle would be used for the drawing of a payoff vector (that is, only the payoff vectors with the highest average income would be drawn from). The payoff for each individual would be determined by random assignment to an income class within the vector drawn. If the group did not reach unanimous agreement, the schedule of payments that would be made to them would be chosen randomly from the full set of vectors (again with individual assignment to a class done randomly). As we put it:

Your payoffs in this section of the experiment will conform to the principle which you, as a group, adopt. If you, as a group, do not adopt any principle, then we will select one of the income distributions at random for you as a group. That choice of income distribution will conform to no particular characteristics.

This procedure provided the incentive to reach an agreement. If the subjects wished to ensure that their payoffs had desirable distributional properties, they had to agree unanimously on a single principle of distributive justice. The stakes for agreement were only their being guaranteed a payoff vector that they all agreed was fair.

[5] In the experiments where a principle was "imposed" on the group, the members of the group did not collectively discuss which principle they would prefer.

[6] Payoff vectors were not drawn in the production experiments. There, the subjects were told that the income they earned individually would be (potentially) redistributed in accordance with the principle they chose.

We attempted in this part of the experiment to drive two other points home:

The first of these points was designed to ensure that subjects did not project from the sample distributions in the first part of the experiment to the second part. The second point was designed to emphasize the seriousness of the task in relation to their monetary stakes.

We permitted lengthy and open discussion.^[7] Indeed, we required that unanimous agreement on secret ballots be reached to terminate discussion.^[8] The agenda was open. As we put it: "You are not restricted , in any way, to the four principles of justice mentioned above. Thus, you can discuss (and later adopt) other principles. Any one of you can introduce and begin discussion of any principle." If all subjects agreed verbally that the discussion had run its course and no further discussion promised to be fruitful, a secret ballot was taken on terminating the discussion. If even a single subject felt further discussion was required, then it was resumed. If the ballot showed that all agreed that no more discussion was needed, voting on the principles was undertaken. A set of (pairwise) secret ballots was then held between all the principles on the agenda to see whether there was a unanimous choice. If there was, the players were paid off as discussed above. If no winner emerged, discussion resumed.

After the discussion and choice, subjects were asked to provide demographic, sociological, and psychological data for analysis. During the questioning, they also gave us, one last

[7] A bare minimum of five minutes was required, but much longer discussions always took place. The transcripts of the discussions average more than ten single-spaced typed pages apiece.

[8] This requirement, again, was varied in some of the production experiments where majoritarian procedures were used to see whether they led to differences.

time, their ranked preferences for the principles. In total, throughout the course of the experiment subjects were asked (on four different occasions) to indicate their rankings of the four principles and to indicate the degree of confidence they attached to their rankings.^[9]

A summary outline of the flow of the experiment might be useful here.

In Part I we handed out a booklet that introduced subjects to the content of the experiment—the substance of the choices they were to make. This part consisted of the following steps:

In Part II the subjects participated in a collective decision under conditions of impartiality in an approximation of Rawls's original position. This part consisted of these steps:

[9] They were asked an additional three times in the production experiments.

In cases b and c, payment was made by random selection of a payoff vector and not in conformity with any particular principle.

Part III involved administration of a debriefing questionnaire dealing with the subject's attitudes, background, and political beliefs. It also dealt with the subject's relationship to the experimental structure itself and asked the subjects for a final ranking of the principles.

Variants of the Basic Experimental Design

To explore the robustness of our results, we varied the experimental conditions (see Table 1). We wanted to ensure that our results would not be a function of the peculiarities and idiosyncracies of the experimental design. For example, consider the

[10] This last choice was not a part of the conceived design. However, an error in the translation of the design into Polish led to the imposition of a time constraint in the discussion periods of some of those experiments.

size of the stakes involved. Rawls and Harsanyi were arguing about principles that would govern the distributive choice of lifetime incomes for oneself, one's class, and one's offspring. Although we framed the questions and choices in similar terms, no amount of verbiage could convert the actual payoffs to choices about lifetime wages. But we could alter the size of the stakes, within the range permitted by the budget of the laboratory experiments, to see whether such alteration made any difference. If choices could be shown to vary as we varied the stakes, then the importance of our small stakes would be clear. Thus, if higher stakes changed the choice, and presumably the underlying reasoning, our experiments probably did not constitute a good test of the theories.

To check for this possible objection, we introduced two experimental variables that might be expected to affect subjects' choices: higher variances in potential payoffs (in the choice situations) and prepayment of $40 with subsequent choices leading to reductions from that credit. The results of these variations were contrasted with the results in the original, or baseline, experiments to yield an indication of the robustness of the results within the range of variation available to the experimenters. The way in which these changes in design might be expected to affect subjects' choices can be outlined simply.

Choices with Higher-Variance Payoffs

Individual choices are often a function of the framing of the situation (Tversky and Kahneman, 1981). We wanted to ensure that our results were not determined by irrelevant framing effects. One way to reframe our experiment was to make the stakes appear higher for the players. We did this by altering the income distributions in Part I so that the floors were lower, and nastier, while the ceilings were higher. This alteration raised the variance in the payoffs that subjects received when they selected principles. We conjectured that the higher variances would give them increased reason to focus on the floor as an undesirable outcome and therefore to choose to maximize the floor. If, as Rawls argues, rational individuals consider nothing

but the floor when selecting a principle of justice, this depression of the available floor should have been an additional inducement to choose the highest floor. Of course the absolute value of the payoffs would still have fallen far short of "life chances," but to the extent that subjects considered the possibilities seriously, the reframing might have worked to increase the attractiveness of a principle that maximizes floor income.

Choices Involving Losses

Increasing variances by lowering the floors and raising the ceilings in Part I of our experiment was not the only way to reframe the situation for our subjects. A. Tversky and D. Kahneman (1981) contend that, for individuals, "the response to losses is more extreme than the response to gains" (p. 454). Therefore, we conjectured that if individuals were concerned with a loss, they would focus their attention on the floor. This focus would, presumably, increase the probability of choosing a principle that guaranteed a maximum floor.^[11] To induce this effect we changed the design slightly by giving subjects a $40 credit prior to the second part of the experiment. In this variant they were told that their choice of a redistributive principle would determine the distribution of reductions to that $40 credit. To make the $40 credit credible, the amount was written on each subject's payoff sheet. This sheet was a cashable credit slip and was the record of the individual's earnings. Thus, subjects were choosing how to distribute losses from each of their $40 credits. Presumably, this change raised the stakes in the eyes of the participants. But this raise was purely subjective. Individuals were given reason to believe that they could win $40 in Part II (and in fact they could), but the final returns that were available to them were identical to those available to subjects in the

[11] The reader should note that the results reported for this variation of the experiment are not congruent with this interpretation of Tversky and Kahneman's hypothesis. A second interpretation of their argument would be that giving subjects a credit of $40 should lead them to increase risk taking to avoid a loss in the second part of the experiment. Hence it could favor the choice of maximum income.

other variants of the experiment. This variation was run with both the original and the higher-variance payoffs.

Choices without Reference to Justice

One concern expressed by early critics of our basic research design was that explicit discussions of distributive justice in the research instrument might skew subjects' responses and choices. The argument was advanced that the subjects' desire to please the experimenters might lead them to give undue and unrealistic consideration to matters of fairness when they considered their choices impartially. Thus, their expressed preferences might unduly reflect the normative content of the research instrument. This outcome would be particularly problematic if Harsanyi and Rawls base their arguments on explicit rationality and self-interest, and make no assumption that normative arguments would be employed from an impartial point of view.

To test for this sort of bias in our basic experimental protocol, we ran one set of experiments in which all explicit references to justice were removed. Thus at the outset the subjects were told: "This experiment deals with the question of the distribution of monetary gains or losses among members of committees." Similar changes were made to remove all explicit references to justice throughout the experiment. Where the word principles was used in the basic experiments, the word rules was substituted to deemphasize the normative content. One can get a sense of the contrast from a quick description at the start of Part I of the experiments. Regular version: "This experiment is concerned with the justice of different income distributions. Let us begin by discussing some ways of judging the justice of an income distribution." Nonjustice version: "This experiment is concerned with committee decisions about rules to distribute income among members of a society which includes you . Let us begin by discussing the effect of different rules on what you will get in the experiment."

In all other respects the research instrument was identical to the basic design. These changes allowed us to test the conjecture

that references to justice might bias the outcome and helped ensure that any results obtained were not disproportionately reflective of secondary aspects of the experimental design.

Postchoice Stability of the Chosen Distributive Principle

Recall, we are concerned not only with the choices of principles but also with the stability of the choices after they are implemented. To test for stability, we created an experimental setting in which subjects experienced the consequences of their choice of distributive principle. We believe the stability of a redistributive policy is a function of both its redistributional impact and its effect on productivity. To test this hypothesis we needed to include a provision for production, earning, and redistribution. Furthermore, we hypothesize that the acceptability of a principle is a function of democratic participation in its choice. To test this hypothesis required that we vary the level of participation in the selection of principle.

We ran three additional sets of experiments to deal with the issues of participation and production. Again, we followed the basic research design, but we added one major component. The additional section involved the opportunity for subjects to perform a task, earn income, and then see it redistributed among group members. The research instrument was expanded to deal with taxation policies associated with each principle, so that subjects would appreciate the taxation consequences of their choices. After the principle was established, group members corrected spelling mistakes in texts excerpted from the works of Talcott Parsons. Each text was arduous (as the reader can easily confirm by reading any random selection from Parsons or by examining the sample paragraph from one of our tasks in Table 4) and contained about twenty spelling errors.

Each individual performed the same task, and each received wages for his or her individual production. The marginal pay rate had considerable returns to scale as is apparent in Table 5. Outputs of each individual were checked; and their earnings,

taxes, and take-home pay were calculated and reported to them along with the equivalent yearly income flows implied by the earnings (see the last column in Table 5). Then their posttax payments for that period were calculated and reported to each of them. Taxes needed to raise individual incomes in accordance with the principle were assessed proportionately against the earnings of those who did not need income transfers. Similar tasks were performed three times in each experiment.

Measurements of preferences for principles, satisfaction with the principles, and degree of certainty with the ranking of principles were made at each stage of the experiments.^[12] In effect the subjects were asked at each stage: "How do you feel now about the principle? How do you rank it now? How sure

[12] Satisfaction with the group's choice of principle was measured only after the last production period in the imposed experiments.

are you? How satisfied are you?" Repeating the task and poll allowed us to examine the relationship between attitudes toward the principles and productivity, and to assess changes in attitudes as a function of experience with redistribution.

Performing these tasks and calculating pre- and posttax incomes extended the time required for the experiments. Without abbreviating the original design, the time would have been unacceptably extended. So the first part of the baseline experiment dealing with individual choices of principles and individual payoffs associated with them (see footnote 3) was omitted.

To obtain information on the effect of different levels of democratic participation on attitudes and productivity, the three different treatments in the production experiments used different group-decision rules. In one variant, the choice of principle and tax policy was by unanimous decision; in a second, by majority rule. In the third variation, the group chose neither the principle nor the tax policy. In this case, there was also no group discussion. Instead, the principle that governed the redistribution of their income was imposed by the experimenters. The experiments involving lack of participation could be contrasted with those involving group choices to test for the impact of participation on the acceptability of the principle and its effect on productivity. To control the situation properly, the principle imposed by the experimenters was the one most often chosen by the previous experimental groups.

In all the treatments individuals were familiarized with principles for redistributing income under conditions of impartial reasoning essentially identical to those in the basic experiments. The only difference was that they were told that their future income would depend on how well they did in an unspecified task. Because they did not know whether the task was physical, mathematical, verbal, analytic, or something else, they were not in a position to estimate their future earning potential. Thus they were effectively unable to estimate their likely future productivity and economic status in the production economy.

An outline of the structure of the production experiments and the points at which each of the relevant variables was measured follows.