7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 1/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

7 Policy Impact, Evaluation, and Change

When it is viewed as a cycle or sequential pattern of functional activities, the final phase of

the policy process is evaluation. As much an art as a science, policy evaluation involves the

estimation, appraisal, or assessment of a policy, its content, implementation, goal

attainment, and other effects. The policy evaluator wants to know whether and to what

extent a policy has accomplished its goals or whether it has had other effects, intended or

unintended.

Policy evaluation may also seek to identify factors that contributed to the success or

failure of a policy. This, in turn, may lead to recycling of the policy process (problem

definition, formulation, adoption, and so on) in order to continue, modify, strengthen, or

terminate the policy. Put differently, information gained through evaluation feeds back into

the ongoing policy process.

As a functional activity, evaluation can occur at any point in the policy process, not simply

after some effort has been made to implement a policy. Thus, attempts are made to

determine prospectively—to estimate or predict—the likely effects, or the costs and benefits,

of policy alternatives prior to their adoption. Typically, however, policy evaluation “looks

backward” to what has happened whereas the other stages of the policy process look

forward to attaining a goal.

Before proceeding further in examining evaluation, we need to pause and look at policy

impacts, at the general sorts of effects or consequences that may ensue from policies.

Policy Impact

To begin, we need to distinguish policy outputs and policy outcomes. Policy outputs are the

things actually done by agencies in pursuance of policy decisions and statements. The

concept of outputs focuses attention on such matters as amounts of taxes collected, miles of

highways built, welfare benefits paid, price-fixing agreements prosecuted, traffic fines

collected, or foreign-aid

290

1

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 2/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

projects undertaken. Outputs usually can be readily counted, totaled, and statistically

analyzed. Examining outputs may indicate, or seem to indicate, that a lot is being done to

implement a policy. Such activity, however, sometimes amounts to nothing more than what

Professor William T. Gormley Jr. calls “bean counting.” Agencies, under pressure from

legislators, interest groups, and others to demonstrate results, “may focus on outputs, not

outcomes, in order to generate statistics that create the illusion of progress.” If the

percentage of students graduating from universities in a state increases, does this tell us

anything about the quality of education that they are receiving?

Policy outcomes (sometimes called “results”), in contrast, are the consequences for society,

intended and unintended, that stem from deliberate governmental action or inaction.

Social-welfare policies can be used to illustrate this concept. It is fairly easy to measure

welfare-policy outputs such as amounts of benefits paid, average level of benefits, and

number of people assisted. But what are the outcomes, or societal consequences, of these

actions? Do they increase personal security and contentment? Do they reduce individual

initiative? Did Aid to Families with Dependent Children, now Temporary Assistance to

Needy Families (TANF), have the effect of encouraging promiscuity and illegitimacy, or

teenage pregnancies, as some alleged? Do welfare programs help keep the poor quiescent,

as others contend? Questions such as these, which are tough to answer, direct our attention

to the societal effects of policies. Among other things, policy students should want to know

whether policies are accomplishing their intended purposes, whether society is changing as

a consequence of policy actions and not because of other factors such as private economic

decisions, and whether it is changing as intended or in other ways. Policy impacts are an

amalgam of outputs and outcomes.

The impact of a policy may have several dimensions, all of which should be taken into

account either in the conduct of a formal evaluation or in the course of an informal

appraisal of the policy. They include the following:

1. Policies affect the public problem at which they are directed and the people involved.

The target populations whom the policy is intended to affect must be defined, whether they

are the poor, small-business people, disadvantaged schoolchildren, petroleum producers, or

whoever. The intended effect of the policy must then be determined. If it is an antipoverty

program, is its purpose to raise the income of the poor, to increase their opportunities for

employment, or to change their attitudes and behavior? If some combination of such

purposes is intended, analysis becomes more complicated because priorities should be

assigned to the various intended effects. Typically, policies accomplish at least a portion of

their goals or objectives.

290

291

2

3

4

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 3/87

Further, a policy may produce either intended or unintended (unforeseen or unplanned)

consequences, or even both. Unemployment compensation may improve the economic

situation of the unemployed, as designed by its proponents, but it may also cause them to

delay in finding new jobs. A publichousing program may improve housing conditions for

urban blacks, but it may also be so administered as to contribute to racial segregation in

housing and

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 4/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

neighborhood schools. The agricultural income and price support programs, intended to

enhance farmers’ incomes, also may lead to overproduction of supported commodities (only

a few such as corn, wheat, and cotton are covered), to higher food prices, and to unearned

increases in land values.

The causes of unintended policy consequences are multiple: the complexity of the issue

being addressed; insufficient information on which to ground a decision; pressure to act

quickly on a matter; too little consultation with other legislators or affected persons;

partisan pressures and the politicization of issues; and, finally, human error. Also, statutory

language needs mention. A law may be clumsily drafted, or too detailed, or too vague, and

thus permit unintended interpretations.

A good illustration of a policy with negative unintended consequences was financial

deregulation (or nonregulation). The Financial Services Modernization Act (1999) permitted

banks to combine commercial banking (deposits and loans), investment banking, and

insurance services. Huge banking conglomerates developed, some of which became viewed

as “too big to fail” because of their potential effects on the economy. The Commodity Futures

Modernization Act (2000), which was slipped into a huge omnibus appropriations bill and

enacted with no debate, prohibited regulation of credit default swaps (don’t ask) and other

financial derivatives. These statutes, and decisions by regulatory officials not to use the

authority they did possess, because of a belief in self-regulating markets, contributed greatly

to the near financial collapse of the American economy and the Great Recession.

2. Policies may affect situations or groups other than those at which they are directed.

These are variously called third-party effects, spillover effects, or externalities. The

construction of urban interstate highways is of much benefit to motorists and trucking

companies. However, they also cause inconvenience, disorder, and social disruption for the

neighborhoods through which they run, and they have helped produce urban sprawl. Clear-

cutting in national forests, which is of benefit to timber companies and in line with the

perspective of those who view forests as tree farms, is profoundly disturbing to

environmentalists, nature lovers, and many sportsmen and women because it results in the

destruction of wildlife habitat, the loss of aesthetic value, and the siltation of trout streams.

These two examples portray negative externalities, but externalities may also be positive.

Public-education programs not only educate students but also provide employers with a

more capable workforce and the community with better-informed citizens. A strong line of

research supports a correlation between education and support for democracy. Those who

contend that only those who have children in public schools should contribute toward their

291

292

5

6

7

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 5/87

support ignore such positive externalities. Although pollution-control programs impose

costs on many industries, they are a boon to the manufacturers of pollution-control

equipment. Many of the outcomes of public policies can be most meaningfully understood

as externalities that impose costs or provide benefits for third parties.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 6/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

3. Policies have consequences for future as well as current conditions; for some policies

most of their benefits or some of their costs may occur in the far future. Was the Head Start

program—a preschool education program for the poor—supposed to improve participating

children’s short-term cognitive abilities or their long-range development and earning

capacity? Did regulation of the field price of natural gas, a policy initiated in the 1950s and

extending into the 1980s, really produce a shortage of gas in the 1970s, as some contended

(notably petroleum-industry officials and their supporters, who had long been critics of

price regulation)? Are we better off under deregulation?

The future effects of some policies may be very diffuse or uncertain. Assuming that patent

and copyright policies do indeed stimulate invention and creativity, and that these activities

in turn enhance economic growth and societal development, how does one measure their

benefits, either qualitatively or quantitatively? Again, how does one appraise (with

reasonable objectivity) the effects of the National Foundation on the Arts and the

Humanities’ support for literacy, artistic, and museum activities? Would the elimination of

such policies as these have deleterious consequences for American society?

4. Just as policies have positive effects or benefits, they also entail costs. Economists seem

never to tire of telling us that there is no such thing as a free lunch. Costs come in different

forms. First, there are the direct costs for the governmental implementation of a policy or

program. These are usually fairly easy to calculate, whether stated as the actual amount of

money spent on the program, its share of total governmental expenditures, or the

proportion of the gross domestic product devoted to it. Budgeting documents will yield such

figures. If, however, a governmental expenditure serves multiple purposes, such as

operating the space program and developing new technology, the allocation of costs

becomes more perplexing.

Direct costs also include private expenditures that are necessary in order to comply with

public policies, such as those for industrial health and safety and environmental-pollution

controls. These may be more difficult to discover or calculate. Moreover, it is possible that

some companies would have installed protective devices in the absence of policy, whether

out of a desire to serve the public interest or to protect themselves from lawsuits. Should

their costs then be assigned to the policy? In the absence of governmental subsidies, the

direct costs of complying with regulatory policies initially fall primarily on the regulated,

who have an ideological incentive to inflate claimed costs, deliberately making the policies

appear more burdensome. Ultimately, of course, most compliance costs are likely to be

shifted to consumers in the form of higher prices for goods and services.

292

293

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 7/87

This brings us to indirect costs. Public policies may cause reduced production, higher

prices, or mental anguish or distress. Expenditures to meet coal-mine-safety requirements

may cause a reduction in mine output. People called for jury services typically receive

compensation that does no more than cover commuting costs. The consequence is lost

wages, or lost production, or both. Many public policies have indirect or social costs of these

sorts. Mental

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 8/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

anguish may occur when one’s hometown is flooded by a new impoundment. Financial

compensation may be paid for one’s childhood home, but what of the loss that occurs

because, being under twenty feet of water, it can no longer be visited? The concept of

indirect cost often calls for putting a price tag on intangibles. They tend to defy precise

measurement.

Finally, there are opportunity costs, a concept that rests upon the fact that we cannot do

everything at once and that resources used for one purpose (e.g., flood control) cannot be

used for another purpose (e.g., public housing). “Opportunity cost is a decision making

rather than an accounting concept.” It focuses attention on what one has to give up (or,

alternatively, what one will gain) if resources are used for one purpose rather than another.

Thus, economists argue that the United States’ all-volunteer army will not attract needed

recruits if the opportunity costs of military service are kept below those of civilian life.

When the ranks of the military were largely filled by draftees, less concern had to be given

to opportunity costs and the compensation of service personnel.

5. The effects of policies and programs may be either material (tangible) or symbolic

(intangible). The consequences of symbols are both important and hard to measure.

Symbolic outputs, according to political scientists Gabriel A. Almond and G. Bingham Powell

include “affirmations of values by elites; displays of flags, troops, and military ceremony;

visits by royalty or high officials; and statements of policy or intent by political leaders.”

Consequently, they “are highly dependent on tapping popular beliefs, attitudes, and

aspirations for their effectiveness.”

Symbolic policy outputs produce no readily discernible changes in societal conditions. No

one eats better because of a Memorial Day parade or a stirring speech by a high public

official on the virtues of free enterprise, however ideologically or emotionally satisfying

such actions may be for many people. More to the point, however, policy actions ostensibly

directed toward meeting material wants or needs may turn out in practice to be more

symbolic than material in their effect.

This shift in policy tone is well illustrated by the Fair Housing Act of 1968. Enacted by

Congress in part because of pressure created by the assassination of Dr. Martin Luther King

Jr., the 1968 law prohibited discrimination in the rental or sale of housing because of race,

color, religion, sex, or national origin. However, the Department of Housing and Urban

Development (HUD), which was assigned primary responsibility for its enforcement, was

authorized only to mediate disputes between a person who thought he or she had been

discriminated against and the renter or seller. The Justice Department, in turn, could not act

293

294

8

9

10

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 9/87

unless it found a “pattern or practice” of discrimination. As a consequence of these weak

enforcement provisions, the Fair Housing Act in practice did not live up to its promise. The

act, which one member of Congress called a “toothless tiger,” was of little use in preventing

discrimination in housing.

The Congress in 1988 reached a compromise agreement on legislation to strengthen

enforcement of the Fair Housing Act. Now a person who believes he

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 10/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

or she has been discriminated against can file a complaint with HUD, which, if it cannot

settle the dispute, can issue a charge of housing discrimination. At this point either party to

the dispute can choose to have it decided by either a federal district court or an

administrative hearing. If either party chooses to go to the federal court, that choice

prevails. This procedure was intended to put some real “teeth” into the enforcement of the

act and give it material rather than merely symbolic effect. It has been partially successful,

but complaints have been far fewer than actual incidents of discrimination.

Studies indicate that presidential level support for fair housing has varied from Johnson to

Obama. So have enforcement actions by federal, state, and local agencies. Though some

forms of discrimination have lessened, overall housing discrimination remains a problem in

many cities and suburbs. Housing segregation and discrimination contribute to other

problems—for example, segregated schools, limited job opportunities, and exposure to high

crime rates in minority areas.

Other public policies that appear to promise more symbolically than their implementation

yields in material benefits include antitrust activity (especially as that involves “trust

busting”), public-utility rate regulation, and equal employment opportunity. Even though

the actual effect of a policy may be considerably less than is intended or desired, it

nonetheless may have significant consequences for society. An antipoverty program that

falls short of its mark may provide some assurance to people that the government cares

about poor people and wants to reduce poverty. Legislation on equal employment

opportunity informs people that their government, officially at least, does not condone

discrimination in hiring on the basis of race, sex, or nationality. Apart from the effects such

policies have on societal conditions, they may contribute to the maintenance of social order,

support for government, and personal self-esteem, which are not inconsequential

considerations.

The analysis and evaluation of public policy is usually focused upon what governments

actually do, why, and with what material effects. We should not, however, neglect the

symbolic aspects of government, despite their intangible and nebulous nature. The rhetoric

of government—what governments say, or appear to say—is clearly a necessary and proper

subject for the policy analyst.

Policy Evaluation

Policy evaluation, as a functional activity, is as old as policy itself. Legislators,

administrators, judges, pressure-group officials, media commentators, and citizens have

294

295

11

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 11/87

always made judgments about the worth or effects of particular policies, programs, and

projects.

The Boston Tea Party, for instance, is often portrayed as an unfavorable evaluation of one

of King George’s colonial tax policies. In actuality, those most threatened were colonial

merchants who sold tea smuggled into the country and enjoyed large profits. They were the

organizers of the Tea Party, not consumers.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 12/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

Many policy judgments have been of an impressionistic or intuitive variety, based at best

on anecdotal or fragmentary evidence, and strongly influenced by ideological, partisan, or

idiosyncratic valuational criteria. Thus, staunch conservatives may regard public-housing

programs as socialistic and in need of repeal, regardless of their causes, necessity, or

consequences; or Democrats may support higher taxes on corporations and the rich because

they believe this will enhance their electoral opportunities. Unemployment compensation

may be deemed a “bad” program because the evaluator claims to know “a lot of people”

who improperly receive benefits.

Apocryphal stories about “welfare queens” who drive Cadillacs to collect their welfare

checks are commonplace among welfare critics. Most of us are familiar with this style of

policy evaluation and have perhaps enjoyed doing a bit of it ourselves. Much conflict results

from this sort of evaluation, however, because different evaluators, depending upon their

values or criteria, may reach sharply divergent conclusions on the merits of the same policy.

The food stamp program is a recent example of this.

Another form of policy evaluation centers on process, on the operation or administration

of a policy or program. Although the terms policy and program are often used

interchangeably, a line can be drawn between them. Policy is defined in this book as a

course of action followed by government in dealing with some problem or matter of

concern. A policy may spawn a variety of programs, a program being defined as a set of

rules, routines, and resources intended to carry out a policy or a portion thereof. Programs,

which give “direct, specific, and concrete expression” to the policies on which they are

based, sometimes are marked out in legislation. More likely, however, they will be the

handiwork of administrative officials. Thus, the Antitrust Division has created several

programs to enforce the antitrust laws, such as the competition advocacy and anti-cartel

enforcement programs.

A program, or several programs, conceivably could be eliminated without impairment of

the underlying policy. In 2003, the Bush administration proposed eliminating several fairly

specifically focused education programs because they were deemed unnecessary,

duplicatory, or replaceable. However, the broad, underlying policy of federal support for

public elementary and secondary education remained firmly in place. Most of the targeted

programs were retained by Congress.

Questions asked about how a policy or program is being run may include: What are its

financial costs? Who receives benefits (payments or services) and in what amounts? Is there

unnecessary overlap with or duplication of other programs? Are legislatively prescribed

295

296

12

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 13/87

standards and procedures being followed? Is the program honestly administered? This kind

of evaluation, which may involve much monitoring of agencies and their officials, will tell

us something about whether there is honesty or efficiency in the conduct of a program, but

like the first kind of evaluation, it will probably yield little or nothing in the way of hard

information concerning the societal effects (outcomes) of a program. On the other hand,

process evaluation is often helpful to program

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 14/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

managers wanting to improve the administration of their programs and reduce their

potential for political criticism.

A third, comparatively new type of policy evaluation, which has been getting increasing

attention in the national government in recent decades, is the systematic and intendedly

objective evaluation of programs. This form of evaluation, which I will refer to as systematic

evaluation, employs social science methodology to measure the societal effects of policies or

programs and the extent to which they are achieving their goals or objectives.

The Departments of Labor, Health and Human Services, and Energy, for instance, have

assistant secretaries whose responsibilities include program evaluation. Bureaus within

these and other departments often include policyand program-evaluation staffs. Moreover,

they enter into many contracts with private research organizations, university scholars, and

others for the performance of evaluation studies. More attention appears to be given to

evaluating social programs (welfare, education, health, and nutrition) than most other areas

of governmental activity. This preference probably arises from the proliferation of social

programs in recent decades, their substantial financial costs, the controversies that swirl

around them, and the dislike of “welfare.”

Systematic evaluation seeks information on the impact of a policy or program on the

public need or problem at which it is directed. Utilizing particularly the talents of social

scientists (economists, political scientists, psychologists, and sociologists), it involves the

specification of goals or objectives; the collection of information and data on program

inputs, outputs, and consequences; and their rigorous analysis, preferably through the use

of quantitative or statistical techniques.

Evaluation researchers use a variety of evaluation research designs, which are often

coupled with high-powered statistical and mathematical techniques. The three evaluation

designs discussed here—the experimental design, the quasi-experimental design, and the

before-and-after study—are basic. An examination of them will convey a notion of the

problems involved in setting up evaluation studies.

The experimental design is the classic method for evaluating a policy or program. Two

comparable groups—an experimental, or treatment, group and a control group—are

randomly selected from the target population. The experimental group receives treatment

through a policy or program; the control group does not. Pretests and posttests of the two

groups are used to determine whether changes, such as improved reading scores or lower

incidences of a disease, have occurred in the two groups. If the performance of the

experimental group is significantly better than that of the control group, the program is

296

297

13

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 15/87

held to be effective. A high level of validity and generalizability is accorded the results of

experiments. An example is in order.

Several years ago, the Delaware Department of Labor conducted a field experiment to

assess the effectiveness of various activities in helping “dislocated workers”—“persons who

have lost long-term, stable jobs due to an increased international competition and/or

changing technology.” The goals of

14

15

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 16/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

the program for dislocated workers, which was funded through the Job Partnership

Training Act, were to help increase workers’ earnings and reduce their need for

unemployment compensation benefits. A coterie of 175 workers with comparable

characteristics was identified; 65 were randomly assigned to the treatment group, with the

other 110 becoming the control group. Members of the treatment group were provided

counseling on job-search activities, assistance in locating openings, retraining, and other

services. Both groups were monitored for a year. Comparison of their performance revealed

that the control group did better than the treatment group in meeting the program’s goals. It

was concluded that “the program did not appear to improve participants’ job prospects.”

Use of the experimental design may not be possible because of costs, time, and ethical or

other considerations. The quasi-experiment, then, may be a useful alternative. The process

of random selection is not used. Rather, the treatment group is compared with another

group (a “comparison group”) that is similar in many respects. Consequently, in the quasi-

experiment there is greater likelihood that the performances of the two groups could be

influenced by their internal characteristics rather than the program treatment. Quasi-

experiments nonetheless are still seen as quite useful for many purposes.

A well-known quasi-experiment involves the Connecticut highway speedcontrol

program. Following a record number of highway traffic fatalities, the state initiated a

crackdown on speeding. Initial data indicated that the enforcement program had

significantly reduced fatalities. It was possible, however, that this resulted from other

factors, such as differences in the weather or more safe cars in operation. To control for

such possibilities, the Connecticut highway fatality rate per 100,000 people was compared

with that of neighboring states where there had been no enhanced enforcement program.

This showed that Connecticut’s fatality rate was lower than those of the other states, thus

supporting the inference that the Connecticut crackdown had a positive effect in reducing

traffic deaths.

The before-and-after study of a program compares the results of a program after a period

of implementation with the conditions existing prior to its inception. Thus, one might

compare the quality of water in a river before and after a pollution-control program was

put into effect. Before-and-after studies often have low costs and take less time to conduct. A

major drawback, however, is that the changes that occur are open to rival explanations.

Improved water quality in our imaginary river could be due to increased flow, voluntary

action by polluters, or an economic recession’s having caused reduced industrial activity. On

the other hand, if a before-and-after study finds little change in the desired direction, then it

is likely that the program is not having much effect. Given all of this, it is still possible for

297

298

16

17

18

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 17/87

before-and-after studies to produce useful information about a program that otherwise

would be unavailable.

As this discussion indicates, systematic evaluation draws on experience in assessing the

effects a policy or program has on the public need or problem at which it is directed. It

permits at least tentative, informed responses to such

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 18/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

questions as: Is this policy achieving its stated objectives? What are its costs and benefits?

Who are its beneficiaries? What happened as a consequence of the policy that would not

have happened in its absence? Consequently, systematic evaluation gives policy-makers and

the general public, if they are interested, some notion about the actual impact of policy and

provides discussions of policy with some grounding in reality. Evaluation findings can be

used to modify current policies and programs and to help design others for the future.

Of course, evaluation studies can also be used for less laudable purposes. Professor Carol

Weiss comments, “Program decision-makers may turn to evaluation to delay a decision; to

justify and legitimate a decision already made; to extricate themselves from controversy

about future directions by passing the buck; to vindicate the program in the eyes of its

constituents, its funders, or the public; to satisfy conditions of a government or foundation

grant through the ritual of evaluation.” In short, evaluators may be motivated by self-

service as well as public service and by a desire to use analysis as ammunition for partisan

or personal political purposes. Thus, the staff of the Federal Paperwork Commission was

interested only in evaluations supportive of their goal of eliminating as much governmental

regulation of business as possible, albeit under the guise of paperwork reduction.

We now turn to some of the ways in which policy evaluation is handled in the national

government.

Policy Evaluation Processes

Much policy evaluation is performed by nongovernmental actors. The communications

media; university scholars and research centers; private research organizations such as the

Brookings Institution, the Urban Institute, and the American Enterprise Institute; pressure

groups; and public-interest organizations, such as Common Cause, the Audubon Society, and

Ralph Nader and his “raiders” (a collection of public-interest activists) are examples; all

make evaluations of public policies and programs that have greater or lesser effects on

public officials. They also provide the general public with information, publicize policy

success and failure, sometimes act as advocates for unpopular causes, and occasionally

provide representation for those unrepresented in the policy process, such as the aged

confined to negligently run nursing homes or exploited migratory farm workers.

Collectively, this aggregation of private evaluators of public policies has been designated

by Professor Richard Hofferbert as the “policy evaluation industry.” We will take a look

here at three of its components.

298

299

19

20

21

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 19/87

Many university scholars engage in evaluation on their own initiative, as members of

university research centers, while under contracts with government agencies, or to gain

academic advancement or fame. Many universities have graduate schools of public policy

whose faculty have a more “applied” or practical orientation and are inclined to undertake

policy research. Some of

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 20/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

the research of university scholars is published in such journals as Evaluation and Program

Planning, Evaluation Review, and Journal of Policy Analysis and Management. For the most

part, though, university scholars are more interested in making theoretical contributions to

their academic disciplines than in getting involved in current policy debates.

A number of private research organizations (“think tanks”) devote all or most of their time

to performing evaluation studies. The Manpower Development Research Corporation uses

funds from national, state, and local governments and foundations to research social

welfare and employment programs. The RAND Corporation operates mainly as a contract

researcher producing policy reports for the Department of Defense. Research organizations

of this sort often can act more independently and be more critical of agency actions than

can internal agency personnel. “Of course, contract researchers often face pressures to

follow the agency line, especially if they are highly dependent on a single agency.”

In their reporting of the news when public policies are involved, the media tend to focus

on policies and government actions that go awry. Some sort of policy scandal is even more

“newsworthy.” The operative assumption seems to be that that sort of reporting will attract

more viewers or readers than will information about a successful program or the routine

operations of government. Some years ago, one of the television networks featured an

occasional series titled “The Fleecing of America” as part of its evening news report. The

focus was on governmental blunders and misadventures. Coverage of this nature

contributes to the (incorrect) notion that “the government can’t do anything well.” More

positively though, it may also help to focus public attention on some matters that need

examination and correction. Television and radio programs that recount public-policy

successes are scarce. One does not hear much about soil conservation, brownfield

restoration, railroad safety, or public housing.

Moving on, within the boundaries of the national government, policy evaluation is

performed in numerous ways by officials and organizations. Sometimes these evaluations

are rigorous and systematic; at other times, they are rather haphazard or sporadic. In some

instances policy evaluation has become formalized, a regular component of the policy

process; in others it remains essentially informal and unstructured. In the remainder of this

section, we glance at a few forms of official policy evaluation—congressional oversight,

Government Accountability Office (GAO) studies, the activities of presidential commissions,

and “in house” evaluations handled within agencies.

Congressional Oversight

299

300

22

23

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 21/87

Although it is not specified in the Constitution, one of the primary functions of Congress is

the supervision and evaluation of the administration and execution of laws and policies,

which is commonly referred to as congressional oversight. Some, agreeing with the English

political theorist John Stuart Mill, think that this is the most important function a legislature

performs. Oversight, however, is not a24

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 22/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

separable, distinct activity; rather, it is an integral part of almost everything that members

of Congress do, including gathering information, legislating, authorizing appropriations,

and being of service to constituents. It may be intended either to control the actions of

agencies, as when they sometimes are required to clear actions in advance with particular

committees, or to evaluate agency actions, as when individual members or committees seek

to determine whether administrators are complying with program objectives established by

Congress. It is the evaluative aspect of oversight that is pertinent here.

Oversight may be performed by a number of techniques, including (1) casework, that is,

intercession with agencies as a consequence of constituents’ demands and requests; (2)

committee hearings and investigations; (3) the appropriations process; (4) approval of

presidential appointments; and (5) committee staff studies. In the course of these

activities, members of Congress reach conclusions about the efficiency, effectiveness, and

impact of policies and programs—conclusions that can have profound consequences for the

policy process. Congressional oversight is in essence more fragmented and disjointed than

continuous and systematic. Bits and pieces of information, impressionistic judgments, and

the members’ intuition and values are blended to yield evaluation of policies and those who

administer them. On the whole, however, members of Congress are more likely to be

involved with initiation and adoption of policy than with evaluation, at least in any

systematic sense.

The congressional practice of enacting laws—for example, the Clean Air Act and the Head

Start law—for determinate lengths of time builds evaluation into the legislative process.

Periodically such laws have to be reauthorized by Congress, thereby creating opportunities

to make changes in them on the basis of experience and available information. At the state

level, sunset laws have a similar effect. These laws provide that state agencies will be

terminated unless they are periodically evaluated and reauthorized. The programs of a

terminated (“sunsetted”) agency may be shifted to another agency. That opportunities for

evaluation are created does not, of course, guarantee that they will be well exploited.

Government Accountability Office

The GAO, usually regarded as an arm of Congress, has broad statutory authority to audit the

operations and financial activities of federal agencies, to evaluate their programs, and to

report its findings to Congress. The agency, which has several thousand employees, has

become increasingly involved with the evaluation of programs and now gives only a minor

portion of its attention to financial auditing.

300

301

25

26

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 23/87

The Legislative Reorganization Act of 1970, which revamped the congressional committee

system, also directed the GAO to “review and analyze the results of government programs

and activities carried on under existing law, including the making of cost-benefit studies,”

and to make personnel available to assist congressional committees in handling similar

activities. A subsequent

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 24/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

statute authorized GAO to establish an Office of Program Review and Evaluation. Because of

its expanded evaluation activities, the agency hires many more people trained in the social

sciences than it once did.

Evaluation activities may be undertaken by the GAO on its own initiative, because of

directives in legislation, at the request of congressional committees, or sometimes at the

behest of individual members of Congress. In the course of a year, the GAO produces several

hundred evaluation studies, varying in length from several pages to a few hundred. The

titles of four evaluation studies issued by the GAO in 2012 indicate the diversity and

evaluative thrust of its actions: FDA’s Food Advisory and Recall Process Needs

Strengthening; Border Security: Additional Steps Needed to Ensure that Officers Are Fully

Trained; Nuclear Safety: DOE Needs to Determine the Costs and Benefits of Its Safety Reform

Effort; and School Bullying: Extent of Legal Protections for Vulnerable Groups Needed to Be

More Fully Assessed. GAO officials are also frequently called upon to testify before

congressional committees.

Copies of GAO studies are sent to members of Congress and the affected agencies. The

agencies are required by law to report to Congress and the Office of Management and

Budget (OMB) on actions taken in response to the GAO’s recommendations or on why they

did not act.

CASE STUDY The GAO and Food Safety

A short narrative will help convey an understanding of the GAO evaluation process. Food

safety is an ever-present problem in the United States. The federal Centers for Disease

Control and Prevention estimated in 2011 that about 48 million people in the United States

annually suffer from food-borne illnesses. These result in 120,000 hospitalizations and

3,000 deaths. In recent years, food poisoning outbreaks have been traced to lettuce,

peanuts, hamburger meat, and jalapeno peppers. For example, in 2011, Listeria bacteria on

cantaloupes grown in Colorado caused thirty-three deaths.

Fifteen federal agencies operating under more than thirty laws govern the U.S. food

supply. Primary among them are the Food and Drug Administration (FDA) in the

Department of Health and Human Services; the Food Safety and Inspection Service (FSIS),

the Agricultural Marketing Service (AMS), and the Grain Inspection, Packers and Stockyards

Administration (GIPSA) in the Department of Agriculture; and the National Marine

Fisheries Service (NMFS) in the Department of Commerce. The FDA regulates some 57,000

food-processing establishments and sets standards of quality and identity for food

301

302

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 25/87

products. The FSIS supervises more than 6,000 plants engaged in slaughtering and

processing meat and poultry products. The AMS has jurisdiction over hundreds of egg and

egg-product facilities. The NMFS provides voluntary (fee for service) safety and sanitary

inspections for approximately 2,500 foreign and domestic seafood firms.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 26/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

In 1992, at the request of a House subcommittee chair, the GAO undertook an evaluation

of the consistency, coordination, and effectiveness of the federal food-safety inspection

program. Data and information for the study were acquired by interviewing agency

officials; by analyzing agencies’ inspection policies, procedures, and records; and by

meeting with industry and consumer groups.

The GAO reached several negative conclusions about the food-safety inspection system.

Food products presenting similar health risks were treated differently by the various

agencies, duplicative inspections occurred, and coordination among the agencies was

inadequate. The agencies’ effectiveness was limited by inconsistencies and illogical

differences, based on statutory authority and funding, in their approaches to food safety.

Thus, plants processing meat and poultry were inspected daily by the FSIS, but food-

processing plants under FDA jurisdiction were inspected once every three to five years.

Further, the agencies drew some narrow jurisdictional lines. For one, the FSIS handled

inspection of open-faced meat sandwiches made with one slice of bread whereas the

traditional sandwich made with two slices of bread was the FDA’s responsibility. For

another, pizza with meat topping was inspected by the FSIS, pizza without meat topping by

the FDA.

Finally, meat and poultry plants were required to register for inspection with the FSIS

before they could operate and sell their products. In contrast, food-processing factories

were under no such mandate and had to be tracked down by the FDA.

To correct these and other problems, the GAO recommended the creation of a single food-

safety agency. Because this was not a good political possibility, the GAO said a “more

realistic” alternative would be the appointment of a panel of experts to develop an

inspection model based on public-health risks and strong enforcement authority. This

would provide a rational basis for improving the effectiveness of the food-safety system. No

Congressional action was taken on the GAO’s recommendations.

Since 1992, the GAO has produced dozens of studies regarding the foodsafety system,

calling for reform and the creation of a single, independent food safety agency. Members

of Congress have introduced bills calling for creation of a unified Food Safety

Administration. However, because of resistance from committees with jurisdiction over

various aspects of food safety, reservations about the creation of a consolidated agency, and

lack of strong public concern, the single-agency idea has foundered.

302

303

27

28

29

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 27/87

Some improvements have been made to protect the food supply. In 2011, the Food Safety

and Modernization Act became law. It strengthens the authority of the FDA to regulate the

way foods are grown, harvested, and processed. Inspections of food facilities will become

more frequent and the agency was given authority to require food recalls. Also, the FDA’s

authority to protect against threats to the food supply under the Bioterrorism Act of 2002

was enhanced.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 28/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

Generally, however, the food safety system continues to be characterized by

fragmentation, duplication, and insufficient resources. The GAO’s proposal for a single

agency continues to be unacceptable to the Congress.

Presidential Commissions

Earlier we examined the presidential commissions’ role in formulating policy. Now we will

see that they can also be used as an instrument of policy evaluation. Whether set up

specifically to evaluate policy or governmental management in some area or for other

purposes such as fact finding, making policy recommendations, or simply creating the

appearance of presidential concern, most commissions involve themselves in policy

evaluation to some degree.

In November 1986, it was disclosed that the U.S. government, under the leadership of

National Security Council (NSC) officials, had sold arms to the Iranian government and

diverted some of the profits to the Nicaraguan rebels (the “Contras”). This news touched off

a major political controversy. To inquire into this matter and to make recommendations for

correction, President Reagan appointed the President’s Special Review Board. Better known

as the Tower Commission, it was composed of former Republican senator John Tower (from

Texas) as chair; Edmund Muskie, former Democratic senator from Maine and secretary of

state; and Brent Scowcroft, former national security adviser to President Gerald Ford (and

later to President George Bush).

In its report issued early in 1987, the Tower Commission sharply criticized President

Reagan and his administration for their conduct in the Iran-Contra affair. The NSC was

depicted as carrying on operations outside its advisory realm, deceiving Congress, paying

little heed to the law, and avoiding any effective oversight. The president himself was

viewed as uninformed, detached, and not in control of NSC action, which ran counter to his

administration’s own policy of no arms sales to the Iranians. The commission’s report made

specific recommendations for bringing the NSC system under more effective presidential

control and direction. President Reagan accepted the commission’s report and instituted

some changes designed to prevent recurrence of such problems.

A more traditional presidential commission was the sixteen-member Social Security

Commission established by President George W. Bush to appraise the Social Security

program and propose ways to strengthen it. Much as its Democratic critics contended it

would, given its membership, the commission set forth three options for partial

303

304

30

31

32

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 29/87

privatization of the program in its report. It is not uncommon for commissions to reach

conclusions agreeable to the appointing president. Some do not, however. A commission

appointed by President Richard Nixon to recommend actions on marijuana surprised him

by calling for its legalization. The President refused to even read the report.

It appears that the policy evaluations and recommendations made by presidential

commissions often have little immediate influence on policymaking. For whatever effect

they do have, the important variables are probably not the quality and soundness of their

findings. Charles Jones concludes that an

33

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 30/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

evaluation commission is likely to have the greatest effect when its report coincides with

other supporting events and accords with the president’s policy preferences, when it

includes some members who hold important governmental positions and are committed to

its recommendations, and when commission staff personnel return to governmental

positions in which they can influence acceptance of its recommendations. These

conditions, however, are often not present.

Administrative Agencies

Many evaluations of policies and programs are produced within the administering agencies,

either on their own initiative or at the direction of Congress or the executive. Agencies

usually want to get some notion of how their programs are working and what can be done

to improve them. Educational program evaluations are often labeled either formative or

summative. Formative evaluations (also known as program monitoring) are designed to

assist officials in making mid-course corrections or adjustments in programs to improve

their operation. Summative evaluations are broader and more thorough in scope and are

used to inform upper-level policy-makers of the overall effects of important policies and

programs. They may lead to major program changes. There is not much reason to expect,

however, that such evaluations will cause agency officials to recommend terminating

favored policies or programs.

The Johnson, Nixon, and Carter administrations tried to build policy analysis and

evaluation into the national budgetary process. The Johnson administration instituted the

Planning-Programming-Budgeting System (PPBS), which required agencies to search for the

most effective and efficient (least-cost) means to achieve their goals. (Remnants of PPBS

remain in a few agencies.) The Nixon administration replaced PPBS with Management by

Objectives (MBO), a more modest effort requiring agencies to specify goals and measure

progress toward achieving them. By the time the Carter administration came to town and

installed Zero-Base Budgeting (ZBB), MBO had evaporated. The ZBB required agencies to

specify different levels of funding and accomplishment for programs, including the “zero”

base (defined not as nothing but rather as the funding level below which the program

would have no real worth). By so doing, agencies would assess the worth of programs and

indicate where spending would do the most good. ZBB did not survive the Carter

administration.

These efforts at reform failed for several reasons. They were instituted on a government

wide basis without much planning or testing, they conflicted with existing budgetary

practices and habits, they were difficult and time-consuming to use, they were viewed as

304

305

34

35

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 31/87

efforts to shift power to higher executive levels, and they lacked continuing presidential

interest and support. They were not, however, without effect, for they left behind in the

agencies a residue of interest and support for more systematic analysis and evaluation of

agencies’ activities. It has now become fairly standard practice for Congress to specifically

direct agencies to undertake program and policy evaluations.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 32/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

Agencies are not prone to evaluate themselves. The conduct of their programs and day-to-

day operations take precedence over more “distant” concerns like program evaluations.

Moreover, evaluation would divert scarce funds from current operations—issuing licenses,

making grants, processing applications, inspecting products and facilities, and so on. If

evaluation of program effectiveness were conducted, and it turned out unfavorable from

the agency’s perspective, it could be used as political ammunition by the agency’s critics. In

all, agencies do not have much incentive to evaluate themselves.

The Government Performance and Results Act (GPRA) (1993), supported by the Clinton

administration and enacted by Congress with little controversy or public notice, is intended

to shift the focus of agencies from inputs and processes to outputs and outcomes—or

results. That is, how do agency operations affect such matters as air quality, industrial

safety, the employability of job trainees, or investment opportunities? This is a laudable

aspiration.

To accomplish this purpose, agencies are directed to work with Congress and its

committees in formulating five-year strategic plans. These plans are to include “(1) the

annual performance goals for agencies’ major programs and activities, (2) the measures that

will be used to gauge performance, (3) the strategies and resources required to achieve the

performance goals, and (4) the procedures that will be used to verify and validate

performance information.” In 1999, after trial runs in a few agencies, the GPRA became

effective for all agencies who annually report in March on their progress and results.

Successfully implemented, the GPRA should facilitate program evaluation and provide

useful information for executive and congressional budget decision- makers. Initially,

agencies had problems in putting GPRA requirements in place. Thus, the Department of

Education’s progress in attaining some of its outcome goals was difficult to appraise because

of data shortages. In other instances, it showed little progress in achieving outcome goals.

For the Environmental Protection Agency (EPA), the GAO reported that “the annual

performance measures established under GPRA are often selected on the basis of available

data that focus primarily on outputs rather than environmental results [or outcomes] for

which credible data are often lacking.”

The agencies have continued to wrestle with GPRA mandates. What has been

accomplished?” Some believe that GPRA has followed the path of previous reforms—a big

investment by agencies at the outset leading to the generation of vast amounts of

paperwork that satisfy formal requirements but provide little evidence that the innovations

have made a dent in behavior or decisions.” A less cynical view holds that GPRA has been

305

306

36

37

38

39

40

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 33/87

beneficial. Some congressional appropriations committees, for example, have pressured

agencies to include information on outcomes or results in their budget justifications.

In 2010, in an effort to strengthen the act, Congress enacted the Government Performance

and Results Modernizations Act. Requirements for agencies to consult with Congress were

enhanced. A new framework was specified aimed at getting agencies to take a more

crosscutting and integrated approach to focusing on results and improving government

performance. The GAO is to

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 34/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

evaluate agency accomplishments. Performance evaluation continues to be an appealing

idea, if less than fully successful.

Problems in Policy Evaluation

The most useful form of policy evaluation for policy-makers and administrators, and for

policy critics who want a factual basis for their positions, is a systematic evaluation that

tries to determine cause-and-effect relationships and rigorously measures the results of

policy. It is, of course, often impossible to measure quantitatively the effect of public

policies, especially social policies, with any real precision. In this context, then, to “measure

rigorously” is to seek to assess policy impacts as carefully and objectively as possible, using

the best information available and making careful judgments. There is no reason to assume

that only quantitative data and analysis are meaningful.

Determining whether a policy or program is doing what it is supposed to do, or doing

something else, is not an easy, straightforward task, as some appear to assume. Snap

judgments are easy to make but lack definitiveness. A variety of conditions raise obstacles

or create problems for the effective accomplishment of policy evaluation. These include

uncertainty over policy goals, difficulty in determining causality, diffuse policy impacts, and

others, all of which are reviewed in this section.

Uncertainty over Policy Goals

When the goals of a policy are unclear, diffuse, or diverse, as they frequently are,

determining the extent to which they have been attained becomes a difficult and frustrating

task. This situation is often a product of the policy-adoption process. Because the support

of a majority coalition is needed to secure adoption of a policy, it is usually necessary to

appeal to persons and groups possessing differing interests and diverse values. To win their

votes, commitments to the preferred policy goals of these various groups may be included in

the legislation.

The Model Cities Act, enacted during the Johnson administration, was a laudable attempt

to confront urban problems. Goals of the Model Cities Act included rebuilding slum and

blighted areas; improving housing, income, and cultural opportunities therein; reducing

crime and juvenile delinquency; lessening reliance on welfare programs; and maintaining

historic landmarks. No priorities were assigned to the various goals; nor, indeed, were their

dimensions well-specified. Substantial funding was provided for Model Cities evaluation

research, which was supposed to determine the extent to which the diverse goals of the act

306

30741

42

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 35/87

were being met. Limited in impact at best, the Model Cities program has been relegated to

the dustbin of history.

Determining the real goals of a program can be a difficult or conflictual task. Persons

occupying different positions in the policy process, such as legislators and administrators, or

national and state officials, or possessing differing ideological or philosophical perspectives,

may define goals differently,

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 36/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

act accordingly, and reach differing conclusions about a program’s accomplishments or

success. Moreover, “because ‘what you measure is what you get,’ choosing the right goals to

measure is essential.” Later in this chapter, we will see how the multiple goals of the Head

Start program, and measurement problems, have complicated its evaluation.

Difficulty in Determining Causality

Systematic evaluation requires that societal changes must be demonstrably caused by policy

actions. The mere fact that when action A is taken, condition B develops does not

necessarily mean the presence of a cause-and-effect relationship. Other actions (or

variables) may have been the actual causes of condition B. As we know, many common

colds are “cured,” not by ingesting medicines, applying ointments, or using nasal sprays, but

by the human body’s natural recuperative power.

Consider this example. Many states require periodic automobile safety inspections in an

attempt to reduce highway traffic accidents and fatalities. Research indicates that states

with mandatory inspection laws do tend to have fewer traffic fatalities than do other states.

Other factors, however, such as population density, weather conditions, and percentage of

young drivers might in fact have more power in explaining the difference. Only if such

conditions are controlled in the analysis, and if differences remain between states with and

without inspections, can it be accurately stated that a policy of periodic automobile

inspections reduces traffic deaths. In actuality, such laws do seem to have a modest

beneficial effect.

To further illustrate the problem of determining causality, let us take the case of crime-

control policies. The purpose, or at least one of the purposes, of these policies is deterring

crime. Deterrence may be defined as the prevention of an action that can be said to have

had a “realistic potential of actualization,” that is, one that really could have happened.

(This assumption is required to avoid the kind of analysis that holds, for example, that

consumption of alcoholic beverages prevents stomach worms since no one has ever been

afflicted with them after starting to drink.) The problem here is that not doing something is

a sort of nonevent, or intangible act. Does a person’s not committing burglary mean that he

or she has been effectively deterred by policy from so acting? The answer, of course, first

depends upon whether he or she was inclined to engage in burglary. If so, then was the

person deterred by the possibility of detection and punishment, by other factors such as

family influence, or by lack of opportunity? As this example indicates, the determination of

causality between actions, especially in complex social and economic matters, frequently is

a daunting task.

307

308

43

44

45

46

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 37/87

Diffuse Policy Impacts

Policy actions may affect groups other than those at whom they are specifically directed. A

welfare program may affect not only the poor but also others such as taxpayers, public

officials, and low-income people who are not receiving welfare benefits. The effects on these

groups may be either symbolic or material. Taxpayers may

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 38/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

grumble that their “hard-earned dollars are going to support those too lazy to work.” Some

low-income working people may indeed try to go on welfare rather than continue working

at grubby, unpleasant jobs for low wages. So far as the poor who receive material benefits

are concerned, how do benefits affect their initiative and self-reliance, family solidarity, or

maintenance of social order? We should bear in mind that policies may have unstated

intentions. Thus, an antipoverty program might have been covertly intended to help defuse

the demands of black activists, or a program to control importing of beef may be intended to

appease cattle growers politically but not really do much to limit foreign competition.

The effects of some programs may be very broad and long-range in nature. Antitrust

policy is an example. Since the policy was originally intended to help maintain competition

and prevent monopoly in the economy, how does one now evaluate its effectiveness? We

can look at current enforcement activity and find that some mergers have been prevented

and many price-fixing conspiracies have been prosecuted, but this record will tell us little

about the extent of competition and monopoly in the economy generally. It would be

pleasing to be able to determine that the economy is n percent more competitive than it

would have been without antitrust policy. Because its goals are general and because

measuring competition and monopoly is difficult, this determination just is not possible.

Interestingly, after a century of antitrust action, we are still without agreed-upon definitions

of monopoly and competition to guide policy action and evaluation. No wonder those

assessing the effectiveness of antitrust policy sometimes come to sharply different

conclusions.

Difficulties in Data Acquisition

As implied in some previous comments, a shortage of accurate and relevant statistical data

and other information may handicap the policy evaluator, particularly when one’s concern

is with policy outcomes. Thus, an econometric model may predict how a tax cut will affect

economic activity, but suitable data to indicate its actual impacts on the economy are hard

to come by. Again, think of the problems in securing the data needed to determine the effect

on criminal-law enforcement of a Supreme Court decision such as Miranda v. Arizona,

which held that a confession obtained when a suspect had not been informed of his or her

rights when taken into custody was inherently invalid. The members of the President’s

Crime Commission in 1967 disagreed about its effect, the majority saying it was too early to

determine results. A minority, however, held that, if fully implemented, “it could mean the

virtual elimination of pretrial interrogation of suspects . . . . Few can doubt the adverse

effect of Miranda upon the law enforcement process.” Absence of data does not

necessarily hinder many evaluators.

308

309

47

48

49

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 39/87

The use of “Miranda cards” to inform suspects of their rights now has become standard

police practice. A consensus exists among criminal-justice scholars and law-enforcement

officers to the effect that this reform has had

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 40/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

little adverse effect on law enforcement. Various field and quantitative studies support this

view. Moreover, it is suggested that the Miranda rule has helped improve professionalism

among the police.

For many social and economic programs, a question that typically arises is “Did those who

participated in programs subsequently fare better than comparable persons who did not?”

Providing an answer preferably involves an experimental evaluation design utilizing a

control group. The difficulty in devising a control (or comparison) group for a workforce

program is summed up in this passage:

A strict comparison group in the laboratory sense of the physical sciences is virtually

impossible, primarily because the behavior patterns of people are affected by so many

external social, economic, and political factors. In fact, sometimes the legislation itself

prevents a proper comparison group from being established. For example, the Work

Incentive Program legislation required that all fathers must be enrolled in the WIN

program within 30 days after receipt of aid for their children. Therefore, a comparison

group of fathers with comparable attributes to those fathers enrolled in the program

could not be established. Even if all the external factors of the economy could be

controlled, it would still be impossible to replicate the social and political environment

affecting any experimental or demonstration program. Thus, it is easy for a decision

maker to discount the results of almost any evaluation study on the basis that it lacks

the precision control group.

Because of problems such as those mentioned in the quotation, experimental designs

frequently cannot be used. (This reason is apart from their often high dollar cost.) Second-

best alternatives must then be utilized, such as a quasiexperimental design using a

nonequivalent control group.

Official Resistance

Evaluating policy, whether it be called policy analysis, measurement of policy impact, or

something else, involves reporting findings and making judgments on the merits of policy.

This is true even if the evaluator is a university researcher who thinks that he or she is

objectively pursuing knowledge. Agency and program officials will be alert to the possible

political consequences of evaluation. If the results do not come out “right” from their

perspective, or worse, if the results are negative and come to the attention of decision-

makers, their program, influence, or careers may be thrown in jeopardy. Consequently,

309

310

50

51

52

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 41/87

program officials may discourage or disparage evaluation studies, refuse access to data, or

keep incomplete records.

Within agencies, evaluation studies are likely to be most strongly supported by higher-

level officials who must make decisions about the allocation of resources among programs

and the continuation of given programs. They may, however, be reluctant to require

evaluations, especially if their results may have a divisive effect within the agencies. Finally,

organizations tend to resist change, and evaluation implies change. Organizational inertia

may thus be an obstacle to evaluation, along with more overt forms of resistance.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 42/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

A Limited Time Perspective

The time horizon of legislators and other elected officials often extends only as far as the

next election. Consequently, they, and others who think like them, often expect quick results

from governmental programs, even social and educational programs whose effects may

take many years to fully appear. This being the case, short-run evaluations of program

accomplishments may be unfavorable.

A good example is the New Deal’s resettlement program, which provided opportunities for

land ownership to thousands of black sharecroppers in the South during the late 1930s and

early 1940s. It was judged as a failure and just another New Deal boondoggle by

contemporary critics. A decades-later evaluation of the program by policy analyst Lester

Salamon concluded, however, that it had significant, positive, long-term effects, although

not as an agricultural policy. At modest cost, it did transform “a group of landless black

tenants into a permanent landed middle class that ultimately emerged in the 1960s as the

backbone of the civil-rights movement in the rural South.” If the time dimension is ignored

in evaluation studies, the results may be flawed and neglect important long-term effects.

The pressure for rapid feedback concerning a policy can then create a dilemma for the

evaluator.

Evaluation Lacks Influence

Once completed, an evaluation of a program may be ignored or attacked as inconclusive or

unsound on various grounds. It may be alleged that the evaluation was poorly designed, the

data used were inadequate, or the findings are inconclusive. Those strongly interested in a

program, however, whether as administrators or beneficiaries, are unlikely to lose their

affection for it merely because an evaluation study concluded that its costs are greater than

its benefits. Moreover, there is also the possibility that the evaluation is flawed.

Governmental programs are not terminated solely as a consequence of an unfavorable

systematic evaluation, although such evaluations did contribute to adoption of the Airline

Deregulation Act. Of course, evaluations frequently lead to incremental changes or

improvements in the design and administration of programs. That is the intent of many

program evaluations done by the GAO, for instance, which, perhaps, is all that should be

asked or expected of most evaluations.

Policy Evaluation: The Use and Misuse of Cost-Benefit Analysis

310

311

53

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 43/87

Cost-benefit analysis is a formal, quantitative evaluation technique that requires identifying

the costs and benefits of either a proposed or actual policy and translating them into

monetary values for purposes of comparison. It assumes that society will be made better off

only by policies (or projects, or programs) whose benefits exceed their costs. Cost-benefit

analysis has been most frequently used to evaluate proposed policies.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 44/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

Sometimes, though, it is employed to appraise existing policies. Thus, economist A. Myrick

Freeman III used it to evaluate national air- and waterpollution- control policies. He found

that the control of air pollution from stationary sources yielded benefits that were much

greater than control costs. On the other hand, the costs of controlling industrial and

municipal sources of water pollution were greater than the benefits realized. In the

following discussion, the focus will be on cost-benefit analysis primarily as a prospective

evaluation technique.

The major steps in performing a cost-benefit analysis can readily be summarized. First,

one identifies all of the effects or consequences of a policy and categorizes them as costs or

benefits for various groups. (Note that this requires establishing which groups are entitled

to be considered in determining costs and benefits.) Both direct and indirect effects should

be analyzed.

Second, dollar values are placed on various costs and benefits. This will be relatively easy

for items that are customarily bought and sold in markets. For such matters as good health,

the prolongation of human life, or scenic vistas, it will be much more difficult.

Third, some of the consequences or effects of a policy will be current or near-term, but

others will occur many years in the future, or even the next generation. Hence, proponents

of cost-benefit analysis call for a discount rate to equate the value of near and long-term

effects. The basic assumption underlying the discount rate is that a dollar today is worth

more than a dollar years from now. Inflation may reduce the dollar’s value or purchasing

power. Alternatively, the money could be invested and its value would increase. My guess is

that most readers would prefer a thousand dollars today to a thousand dollars seventeen

years from now. What is the explanation?

Fourth, the costs and benefits, direct and indirect, current and future, of the policy are

compared. If benefits exceed costs, the policy is acceptable; conversely, if costs exceed

benefits, it should be rejected, or a better way of doing it should be found.

So presented, cost-benefit analysis appears as a reasonably clear-cut method for

appraising policies. In actuality, there are significant problems involved in its application, a

few of which are examined here.

Good data on the costs and benefits of a policy are frequently difficult to come by. How, for

example, does one calculate the value of the health benefits of cleaner air? Or of the esthetic

benefits of reducing haze in national parks? How are dollar values to be assigned to such

311

312

54

55

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 45/87

matters? The value of land flooded (a cost) for a reservoir can readily be determined by

reference to the value of nearby land. But what of the value of an ancestral home located

there? The data and dollar values on which a cost-benefit analysis is based can be of

tenuous and arguable nature. Professor Frank Rourke states that “the numerical values

assigned to both costs and benefits are more often the product of imagination than

mathematical skill.”

It is, further, no easy task to identify the appropriate discount rate. It can be based on such

criteria as the interest rate, the rate of inflation, or the

56

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 46/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

opportunity costs of capital—that is, the rate of return that money would earn if devoted to

private investment rather than public purposes. Despite its importance, there is no scientific

way to decide on a discount rate. A low discount rate preserves the value of future benefits

whereas a high discount rate can sharply reduce their value. During the Reagan years, the

OMB advocated a discount rate of 10 percent. This discount rate meant that the value of

future benefits, such as lives prolonged two or three decades hence by reducing the

incidence of cancer, would have very low value. This, in turn, increased the likelihood that a

cost-benefit ratio would be unfavorable, and that regulation would not be justifiable.

Cost-benefit analysis is based on the premise that efficiency is the primary, if not the only,

value to be realized. Actions are evaluated on the basis of whether resources are used to

improve the aggregate public good. Little attention is accorded alternative or competing

values—equity, human dignity, personal freedom, and equality, to name some. These are

important to most people. The American system of criminal justice, for instance, is not very

efficient because of our concern with equity and due process.

Another problem that arises in the course of many cost-benefit analyses is the need to

place a value on a human life. Some take the position that life is priceless and that

attempting to place a dollar value on life reduces it to just another commodity. In response it

is argued that many policy decisions, such as industrial-safety standards and highway speed

limits, have an impact in terms of human lives. It is better to objectively take into account

the value of life. The EPA in 2008 determined that the “value of a statistical life” was $6.9

million in current dollars, nearly a million dollars less than five years earlier. This figure

was calculated from opinion surveys asking people what they were willing to pay to avoid

specified risks and what employers paid employees to assume additional risks. (Figure 7.1

displays some alternatives for valuing a human life.) The EPA has traditionally put the

highest value on human life of any federal agency. Interestingly, all units within the EPA do

not use the same figure. Under the cost-benefit regimen, the less a life is worth, the less

need for regulatory protection, and vice versa.

Finally, let us note that cost-benefit analysis emphasizes the consequences for society as a

whole. As we know, however, public policies distribute advantages and disadvantages, or

costs and benefits. Those who pay the costs of policies often do not benefit from them, and

vice versa. Put differently, policies have distributive consequences that are of importance.

People may appropriately be more concerned with who benefits from industrial-safety

policies than whether their total costs exceed their benefits.

312

313

57

58

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 47/87

Problems such as those sketched here have not prevented cost-benefit analysis from being

used as a tool in governmental decision-making for several decades. The Flood Control Act

of 1936 specified that flood-control projects could be undertaken by the Army Corps of

Engineers only “if the benefits to whomsoever they may accrue are in excess of the

estimated costs.” This standard must also be used for water projects handled by the Natural

Resources Conservation Service and has been voluntarily employed by the Bureau of

Reclamation. In the 1960s, cost-benefit analysis was first used in evaluating defense

programs and then domestic programs as part of PPBS.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 48/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

FIGURE 7.1 The Valuation of Human Life

In the 1970s, Presidents Ford and Carter directed executive-branch regulatory agencies to

prepare “inflation impact statements” and “regulatory analyses,” respectively, in developing

some proposed regulations. These statements involved analyzing their expected economic

consequences. The Carter administration made it clear, however, that although regulatory

agencies should consider the burdens and gains of proposed regulations, a cost-benefit test

was not to be used in appraising them.

A goal of the Reagan administration when it took office was to substantially reduce

governmental regulation of private economic activity. People who were critical of the

programs under their jurisdiction were appointed to regulatory positions. A second action

involved issuing Executive Order 12291 in

313

314

59

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 49/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

February 1981, which drew heavily upon the Carter administration’s experience. The

order required that proposed major regulations issued by executivebranch agencies (the

independent regulatory commissions were exempt) must be accompanied by regulatory

impact analyses assessing the potential benefits, costs, and net benefits of the regulations,

including effects that could not be quantified in monetary terms, unless such calculations

were prohibited by law. Some statutes ban the use of cost-benefit analysis for the programs

they establish.

Major regulations were defined as those likely to have an annual impact on the economy

of $100 million or more, to lead to major cost or price increases, or to have “significant

adverse effects on competition, employment, investment, productivity, innovation, or the

ability of U.S.-based enterprises to compete with foreign-based enterprises in domestic or

export markets.” The OMB was authorized to make the final determination of what was a

major rule, to supervise the evaluation process, and to delay the issuance of proposed or

final rules if it found the regulatory analyses were unsatisfactory.

Rules could be issued only if their estimated benefits exceeded their estimated costs. If a

choice was available, the less costly alternative was to be selected. The burden of proof that

this standard was met rested with the agency. An action by the OMB holding up a rule could

be appealed to the President’s Task Force on Regulatory Relief, which was staffed by the

OMB and comprised several executive officials under the leadership of Vice President

George Bush. (The word relief in the task force’s title indicates its orientation.) Although the

task force was phased out in 1983, all of this planning was intended to ensure, among other

things, that “Regulatory Action shall not be undertaken unless the potential benefits to

society for the regulation outweigh the potential costs to society.” Thus, cost-benefit analysis

was to be more than an analytical technique; it became a decision rule with a conservative

bias.

The Reagan regulatory-analysis program was a center of controversy. Critics contended

that it was used improperly to reduce the extent of regulation and to delay the issuance of

rules rather than to improve the quality of regulations by encouraging better analysis. The

OMB was also accused of improperly interfering in the regulatory process by usurping

authority vested in the regulatory agencies. The administration denied such accusations. In

practice, though, administration officials demonstrated much more vigilance about the costs

than the benefits of regulation in trying to reduce the burden of regulatory activity on

businesses.

314

31560

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 50/87

The George Bush administration continued the regulatory-analysis program and in time

created the Council on Competitiveness, an interagency committee chaired by Vice

President Dan Quayle, to work with the OMB in perpetuating the use of cost-benefit

analysis. In the final two years of the George Bush administration, the Council on

Competitiveness acted vigorously to represent the business community in the regulatory

process and to reduce the number and strength of new regulations. For the most part, it

avoided publicity and sought to leave few “fingerprints.”

61

62

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 51/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

The Clinton administration quickly abolished the Council on Competitiveness and later

replaced Executive Order 12291 with its Executive Order 12866, entitled “Regulatory

Planning and Review.” The Clinton regulatory-review program continued to make use of

cost-benefit analysis for major rules. Regulatory review, however, was conducted more

openly and less stringently and intrusively than under the Reagan and George Bush

administrations. Regulatory agencies experienced little to complain about.

The George W. Bush administration retained Clinton’s executive order. The Office of

Information and Regulatory Affairs (OIRA), however, now came under the direction of a

true believer in cost-benefit analysis. Much of his activity at a university policy center had

been funded by business corporations. In its first few months under the Bush

administration, OIRA rejected twentyone proposed regulations, more than had been turned

down by the Clinton administration in eight years. This was not surprising given the

strongly conservative, antiregulation stance of the George W. Bush administration. OIRA

continued for the next seven years to take a tough stance on proposed rules.

As the record here presented indicates, the uses of cost-benefit analysis, and presidential

review of proposed regulations, have become regularized features of the regulatory process.

They will continue under the Obama administration, albeit with softening modifications.

How they are implemented depends substantially upon the ideological leanings of a

presidential administration. Clearly, cost-benefit analyses is a malleable evaluation

instrument.

Fairly used, cost-benefit analysis may contribute to rationality in the decisionmaking

process by aiding in the identification and appraisal of alternatives, helping to identify

impacts or consequences, and otherwise developing information and insights that will assist

persons in making reflective, wellconsidered decisions. A careful appraisal of the likely

costs and benefits of a proposed action, and of the persons and groups upon whom they will

fall, is certainly useful, regardless of whether all are converted into dollar figures, and

without converting cost-benefit analysis into a decision rule.

Cost-benefit analysis, however, is open to manipulation to support the values and

preferences of its users. In the instance of Executive Order 12291, because of the stoutly

antiregulatory orientation of its implementers, it became a form of partisan political

analysis dressed up as regulatory rationality. Again, it is doubtful that the Army Corps of

Engineers has ever been unable to undertake a rivers-and-harbors project that its officials

really wanted to construct because a favorable cost-benefit analysis could not be contrived.

315

316

63

64

65

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 52/87

Policy evaluation, as our discussion indicates, is more than a technical or objective

analytical process; it is also a political process. In the next section, a case study of the Head

Start program illustrates how political factors can affect the conduct and results of an

evaluation of a social program. The case also demonstrates that such evaluations, even

when intended to be neutral or objective in form, become political because they can affect

allocation of resources.

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 53/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

CASE STUDY The Politics of Evaluation: Head Start

In January 1965, President Lyndon Johnson announced that a preschool program named

Head Start would be initiated as part of the Community Action Program (CAP) authorized

by the Economic Opportunity Act of 1964. The Head Start program, which was designed to

help overcome the effects of poverty on the educational achievement of poor children,

included early classroom education, nutritional benefits, parent counseling, and health

services.

Initially, $17 million in CAP funds was earmarked for summer 1965 to enable 100,000

children to participate in Head Start. The announcement of the program, however,

produced requests for a much larger volume of funds from many localities. Officials in the

Office of Economic Opportunity (OEO), who had jurisdiction over the program, decided to

meet this demand. Ultimately, $103 million was committed to provide places for 560,000

children. To say the least, the Head Start program was highly popular, undoubtedly because

it directed attention to poor preschool children, who readily aroused the public’s sympathy,

and to the goal of equal opportunity.

Late in the summer of 1965, Head Start became a permanent part of the antipoverty

program. According to President Johnson, Head Start had been “battle-tested” and “proven

worthy.” It was expanded to a full-year program. In fiscal year 1968, $330 million was

allocated to provide places for 473,000 children in summer programs and another 218,000

in full-year programs, making Head Start the largest component of the CAP. Essentially,

Head Start was a multifaceted program for meeting the needs of poor children. More than a

traditional nursery school or kindergarten program, it was designed also to provide poor

children with physical- and mental-health services and nutritious meals to improve their

diet. Further, an effort was made to involve members of the local community in the

operation of the program.

With this background, let us turn to evaluation of the program. The OEO was among the

agency leaders in efforts to evaluate social programs because of statutory requirements.

Within the agency the task of evaluating its programs for overall effectiveness was assigned

to the Office of Research, Plans, Programs, and Evaluations (RPP&E). Some early efforts had

been made to evaluate the effectiveness of Head Start, mostly by Head Start officials and

involving particular projects, but by mid-1967, no solid evidence was available on overall

program effectiveness. This lack was beginning to trouble OEO officials, the Bureau of the

Budget, and some members of Congress.

316

317

66

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 54/87

The Evaluation Division of RPP&E, as part of a series of national evaluations of OEO

programs, proposed an ex post facto study design for Head Start in which former Head

Start children currently in the first, second, and third grades of school would be given a

series of cognitive and affective tests. Their test scores would then be compared with those

of a control group who had not been in the Head Start program. The Evaluation Division

believed such a design would yield results more quickly than a longitudinal study that,

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 55/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

although more desirable, would take longer to complete. (A longitudinal study examines

the effect over a period of time of a program on a given group.)

Within the OEO, Head Start officials opposed the proposed study on various grounds,

including its design, the test instruments to be used, and the focus on only the educational

aspect of the program to the neglect of its other goals—health, nutrition, and community

involvement. The RPP&E evaluators acknowledged the multiplicity of Head Start goals but

contended that cognitive improvement was its primary goal. They agreed with Head Start

officials that there were risks in making a limited study, such as possibly misleading

negative results, but insisted that the need for evaluative data necessitated taking the risks.

In the wake of much internal debate, the OEO director decided the study should be made,

and in June 1968 a contract was entered into with the Westinghouse Learning Corporation

and Ohio University. The study was conducted in relative quiet, but hints of its negative

findings began to surface as it neared completion.

Early in 1969, a White House staff official became aware of the Westinghouse study and

requested information on it because the president was preparing an address on the

Economic Opportunity Act that would include a discussion of Head Start. In response to the

request, OEO officials reported the preliminary negative findings of the study. In his

message to Congress on economic opportunity on February 19, 1969, President Nixon

referred to the study, commenting that “the preliminary reports . . . confirm what many

have feared: the long-term effect of Head Start appears to be extremely weak.” He went on

to say that “this must not discourage us” and spoke well of the program. Nonetheless, his

speech raised substantial doubts about Head Start among many observers in the public

arena.

The president’s speech also touched off considerable pressure for release of the study’s

findings. The OEO officials were reluctant to do this because what had been delivered to

them by Westinghouse was a preliminary draft intended for use in deciding such matters as

what additional statistical tests were needed and what data required reanalysis. From

Congress, where hearings were being held on OEO legislation, claims were made that the

study was being held back to protect Head Start and that the report was going to be

rewritten. The pressure on the White House became sufficiently great that it directed the

OEO to make the study public by April 14. A major conclusion of the report was that the

full-year Head Start program produced a statistically significant but absolutely slight

improvement in participant children.

317

318

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 56/87

The release of the report set off a flood of criticism from Head Start proponents, including

many academicians, directed at the methodological and conceptual validity of the report. A

sympathetic article on the front page of The New York Times bore the headline “HEAD

START REPORT HELD ‘FULL OF HOLES .’ ” Much of the ensuing controversy focused on the

statistical methods used in the study and involved a broad range of claims, charges,

rebuttals, and denials. The proponents of Head Start seemed to fear that their program was

being victimized by devious intent. This fear had several facets. One was that

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 57/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

persons within the OEO who favored Community Action over Head Start wanted a study

that would spotlight Head Start’s deficiencies. Another was that the administration was

going to use the findings to justify a major cutback in Head Start. Finally, there was the fear

that “enemies of the program” in Congress would use the negative results as an excuse for

attacking it. Although there later appeared to have been little factual basis for these fears,

they were real to the proponents of Head Start and contributed to the intensity of their

assault on the evaluation study.

The methodological conflict that arose over the study focused on such standard items as

sample size, validity of the control group, and appropriateness of the tests given to the

children. An examination of these matters would be too lengthy and too technical to

include here. An assessment of the study by liberal economist Walter Williams, however,

provides a balanced view of the controversy:

In terms of its methodological and conceptual base, the study is a relatively good one.

This in no way denies that many of the criticisms made of the study are valid. However,

for the most part, they are the kinds of criticisms that can be made of most pieces of

social science research conducted outside the laboratory, in a real-world setting, with

all of the logistical and measurement problems that such studies entail. And these

methodological flaws open the door to the more political issues. Thus, one needs not

only to examine the methodological substance of the criticisms which have been made

of the study, but also to understand the social concern which lies behind them as well.

Head Start has elicited national sympathy and has had the support and involvement of

the educational profession. It is understandable that so many should rush to the

defense of such a popular and humane program. But how many of the concerns over

the size of the sample, control-group equivalency, and the appropriateness of

covariance analysis, for example, would have been registered if the study had found

positive differences in favor of Head Start? We imagine that this type of positive, but

qualified assessment will fit any relatively good evaluation for some time to come. We

have never seen a field evaluation of a social action program that could not be faulted

legitimately by good methodologists, and we may never see one.

Interestingly, the findings of the Westinghouse study were as favorable to Head Start as

were the earlier evaluations of specific projects made by Head Start officials. These, too,

showed that the program had limited lasting effects on the children. What the

Westinghouse study, and the controversy over it, did was to inject these findings into the

public arena and expand the scope of the conflict over them.

318

319

67

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 58/87

Despite the essentially negative evaluations of its accomplishments, the Westinghouse

report recommended that Head Start be preserved and improved, at least partly on the

ground that “something must be tried here and now to help the many children of poverty

who may never be helped again.” Head Start was, and is, a politically popular program.

Congress and the

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 59/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

executive have generally been favorably disposed toward the program, and it has suffered

little of the criticism directed at other aspects of the antipoverty program. Children are a

potent symbol in policy conflicts.

Ten years after the Westinghouse study was made public, the findings of another group of

researchers on the long-term effects of Head Start were published by the Department of

Health, Education, and Welfare. Based on a series of longitudinal studies, this study

concluded that Head Start had significant, long-lasting social and educational benefits for

its participants. Thus, children who had been in the program had much less need for

remedial classes, were less likely to be retained in grade, and were half as likely to drop out

of high school as were adolescents of comparable age who had not been in the program.

As a consequence, Head Start was now hailed as a success by the communications media.

Why the substantial difference in findings by the two evaluations? The explanation rests

primarily with the different methodological approaches. The Westinghouse study, using an

experimental design, focused on short-run effects, especially as measured by intelligence-

test scores. The second study focused on long-range effects.

In 1981, Head Start was designated part of President Reagan’s “social safety net,” which

provided assistance to the “truly needy,” and thus was not tagged for cutbacks in funding,

as were several other programs that provided aid to poor people. In 1988, approximately

450,000 children were enrolled in Head Start, which now operated year-round, at a cost of

$1.2 billion. Only about a quarter of the eligible children were actually enrolled in the

program, however. Head Start continued to expand under subsequent administrations.

Research studies on the benefits of Head Start and early-childhood education have

continued to yield inconclusive findings. Children who go through Head Start are found to

have improved cognitive abilities, greater self-esteem, and improved social skills. On the

other hand, various studies report that gains in academic achievement are not lasting. After

a few years, when Head Start children are compared with non-Head Start children, the

educational gains fade away.

A major evaluation of Head Start published in the American Economic Review in 1995

illustrates these mixed findings. Using longitudinal data for a sample of nearly 5,000

children, the evaluators examined the impact of Head Start on cognitive achievement,

school performance (whether a child repeated one or more grades), utilization of

preventive medical care, and health and nutritional status. Children who had been enrolled

in Head Start were compared with their siblings who either had been enrolled in other

preschool programs or had had no preschool experiences.

319

320

68

69

70

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 60/87

The evaluation found that Head Start had positive and persistent effects on the cognitive

achievement and school performance of white children. In contrast, although there were

positive effects on the cognitive achievement of African-American children, these effects

soon disappeared. No positive effects were found on the school performance of African-

American children. For both white and African-American children, Head Start had a

positive effect

 

 

7/24/2018 University of Phoenix: Public Policymaking

https://phoenix.vitalsource.com/#/books/9781305332683/cfi/6/18!/4/24/52/80@0:61.9 61/87

PRINTED BY: monkey4990@email.phoenix.edu. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without publisher’s prior permission. Violators will be prosecuted.

on preventive health care, as measured by measles immunization rates. For neither did it

have an impact on health and nutritional status, as measured by conformity with national

height-for-age norms.

Written Analysis Grading Rubric

CRITERIA Outstanding Above Average Satisfactory Below Average Insufficient No Effort

1. Analysis

 

35

Throughout the whole

work, ● evidence is organized

and synthesized to reveal insightful patterns, differences, or similarities related to focus

Throughout most of the

work, ● evidence is organized

and synthesized to reveal insightful patterns, differences, or similarities related to focus

Throughout a considerable amount of

the work, ● evidence is organized

and synthesized to reveal insightful patterns, differences, or similarities related to focus

 

Throughout some of

the work, ● evidence is organized

and synthesized to reveal insightful patterns, differences, or similarities related to focus

Throughout little to none of the work,

● evidence is organized and synthesized to reveal insightful patterns, differences, or similarities related to focus

No effort

2. Support

 

35

Throughout the whole

work, ● sentences in each

paragraph compellingly explain the main idea

● claims are supported with detailed and persuasive examples

Throughout most of the

work, ● sentences in each

paragraph compellingly explain the main idea

● claims are supported with detailed and persuasive examples

Throughout a considerable amount of

the work, ● sentences in each

paragraph compellingly explain the main idea

● claims are supported with detailed and persuasive examples

 

Throughout some of

the work, ● sentences in each

paragraph compellingly explain the main idea

● claims are supported with detailed and persuasive examples

Throughout little to none of the work,

● sentences in each paragraph compellingly explain the main idea

● claims are supported with detailed and persuasive examples

No effort

3. Conclusions

 

20

States a conclusion that is a logical extrapolation from the inquiry findings

States a conclusion that is an extrapolation from the inquiry findings but needs to strengthen the logical connections

States a conclusion focused solely on the inquiry findings. The conclusion arises specifically from and responds specifically to the inquiry findings

States a general conclusion that, because it is so general, also applies beyond the scope of the inquiry findings

States an ambiguous, illogical, or unsupportable conclusion from inquiry findings

No effort

4. Grammar,

Mechanics, and Style

 

10

Throughout the whole

work, the text ● is free of major errors

in grammar, spelling, punctuation, and citation (where applicable)

● is clear and concise ● is effectively fitted to

format and purpose ● uses language

Throughout most of the

work, the text ● is free of major errors

in grammar, spelling, punctuation, and citation (where applicable)

● is clear and concise ● is effectively fitted to

format and purpose ● uses language

Throughout a considerable amount of

the work, the text ● is free of major errors in

grammar, spelling, punctuation, and citation (where applicable)

● is clear and concise ● is effectively fitted to

format and purpose ● uses language

Throughout some of

the work, the text ● is free of major errors

in grammar, spelling, punctuation, and citation (where applicable)

● is clear and concise ● is effectively fitted to

format and purpose ● uses language

Throughout little to none of the work, the

text ● is free of major errors

in grammar, spelling, punctuation, and citation (where applicable)

● is clear and concise ● is effectively fitted to

format and purpose

No effort

 

 

appropriate for intended audience

● demonstrates strong word choice and sentence variety

appropriate for intended audience

● demonstrates strong word choice and sentence variety

appropriate for intended audience

● demonstrates strong word choice and sentence variety

 

appropriate for intended audience

● demonstrates strong word choice and sentence variety

● uses language appropriate for intended audience

● demonstrates strong word choice and sentence variety