INTRODUCTION
Policy analysis is a process of multidisciplinary inquiry aiming at the creation, critical assessment, and communication of policy-relevant knowledge. As a problem-solving discipline, it draws on social science methods, theories, and substantive findings to solve practical problems.
METHODOLOGY OF POLICY INQUIRY
The methodology of policy inquiry refers to the critical investigation of potential solutions to practical problems. Abraham Kaplan, one of the founders of the policy sciences, observed that the aim of the methodology is to help understand and question, not only the products of policy inquiry but the processes employed to create these products. The methodology of policy inquiry contributes to the reflective understanding of theories, methods, and practices of specialized fields such as benefit-cost analysis in economics, implementation analysis in political science, and program budgeting in public administration. These and other specialized fields hold an important place in the process of policy inquiry because they help provide multidisciplinary foundations.
The methodology of policy inquiry is not the same as methods such as regression analysis, ethnographic interviewing, or a benefit-cost analysis, because the methodology is concerned with the philosophical assumptions that justify the use of these methods. Nor is the methodology of policy inquiry equivalent to one or another philosophy of science, for example, logical positivism, hermeneutics, or pragmatism. On the contrary, the methodology of policy inquiry is productively eclectic; its practitioners are free to choose among a range of scientific methods, qualitative as well as quantitative, and philosophies of science, so long as these yield reliable knowledge. In this context, policy analysis includes art and craft, which may be regarded as scientific to the extent that they succeed in producing knowledge that is reliable because it can be trusted. Ordinary commonsense knowing and well winnowed practical wisdom, both products of evolutionary learning across generations of problem solvers, often produce conclusions that are at least as reliable, and sometimes more so, than those produced by specialized social science methods.
The rationale for policy analysis is pragmatic. For this reason, policy analysis is unmistakably different from social science disciplines that prize knowledge for its own sake. The policy relevance of these disciplines depends, not only on their status as sciences but on the extent to which they are successful in illuminating and alleviating practical problems. Practical problems, however, do not arrive in separate disciplinary packages addressed, as it were, to social science departments. In today’s world, multidisciplinary policy analysis seems to provide the best fit with the manifold complexity of public policymaking.
MULTIDISCIPLINARY POLICY ANALYSIS
Policy analysis is partly descriptive. It relies on traditional social science disciplines to describe and explain the causes and consequences of policies. But it is also normative, a term that refers to value judgments about what ought to be, in contrast to descriptive statements about what is. To investigate problems of efficiency and fairness, policy analysis draws on normative economics and decision analysis, as well as ethics and other branches of social and political philosophy, all of which are about what ought to be. This normative commitment stems from the fact that analyzing policies demand that we choose among desired consequences (ends) and preferred courses of action (means). The choice of ends and means requires continuing tradeoffs among competing values of efficiency, equity, security, liberty, democracy, and enlightenment. The importance of normative reasoning in policy analysis was well stated by a former undersecretary in the Department of Housing and Urban Development: “Our problem is not to do what is right. Our problem is to know what is right.”
Policy-Relevant Knowledge
Policy analysis is designed to provide policy-relevant knowledge about five types of questions:
Policy problems. What is the problem for which a potential solution is sought? Is global warming a man-made consequence of vehicle emissions, or a consequence of periodic fluctuations in the temperature of the atmosphere? What alternatives are available to mitigate global warming? What are the potential outcomes of these alternatives and what is their value or utility?
Expected policy outcomes. What are the expected outcomes of policies designed to reduce future harmful emissions? Because periodic natural fluctuations are difficult if not impossible to control, what is the likelihood that emissions can be reduced by raising the price of gasoline and diesel fuel or requiring that aircraft use biofuels?
Preferred policies. Which policies should be chosen, considering not only their expected outcomes in reducing harmful emissions but the value of reduced emissions in terms of monetary costs and benefits? Should environmental justice be valued along with economic efficiency?
Observed policy outcomes. What policy outcomes are observed, as distinguished from the outcomes expected before the adoption of a preferred policy? Did a preferred policy actually result in reduced emissions, or did decreases in world petroleum production and consequent increases in gasoline prices and reduced driving also reduce emissions?
Policy performance. To what extent has policy performance been achieved, as defined by valued policy outcomes signaling the reduction of global warming through emissions controls? To what extent has the policy achieved other measures of policy performance, for example, the reduction of costs of carbon emissions and global warming to future generations?
Answers to these questions yield these five types of policy-relevant knowledge, which are shown as rectangles in Figure 1.1.
Policy problems are representations of problem situations, which are diffuse sets of worries, inchoate signs of stress, or surprises for which there is no apparent solution. Knowledge of what problem to solve requires knowledge about the antecedent conditions of a problem situation (e.g., school dropouts as an antecedent of unemployment), as well as knowledge about values (e.g., safe schools or a living wage) whose achievement may lead to the definition of the problem and its potential solutions. Knowledge about policy problems also includes at least two potential solutions to the problem and, if available, the probabilities that each alternative is likely to achieve a solution. Knowledge about policy problems plays a critical role in policy analysis because the way a problem is defined shapes the identification of available solutions. Inadequate or faulty knowledge may result in serious or even fatal errors: defining the wrong problem.
Expected policy outcomes are likely consequences of adopting one or more policy alternatives designed to solve a problem. Knowledge about the circumstances that gave rise to a problem is important for producing knowledge about expected policy outcomes. Such knowledge is often insufficient, however, because the past does not repeat itself, and the values that shape behavior may change in the future. For this reason, knowledge about expected policy outcomes is not “given” by the existing situation. To produce such knowledge may require creativity, insight, and the use of tacit knowledge. A preferred policy is a potential solution to a problem. To select a preferred policy, it is necessary to have knowledge about expected policy outcomes as well as knowledge about the value or utility of the expected outcomes. Another way to say this is that factual, as well as value premises, are required for a policy prescription. The fact that one policy is more effective or efficient than another does not alone justify the choice of a preferred policy. Factual premises must be joined with value premises involving equality, efficiency, security, democracy, enlightenment, or some other value.
An observed policy outcome is a present or past consequence of implementing a preferred policy. It is sometimes unclear whether an outcome is actually an effect of a policy. Some effects are not policy outcomes, because many outcomes are the result of other, extra policy factors. It is important to recognize that the consequences of an action cannot be fully stated or known in advance, which means that many consequences are neither anticipated nor intended. Fortunately, knowledge about observed policy outcomes can be produced after policies have been implemented.
Policy performance is the degree to which an observed policy outcome contributes to the solution of a problem. In practice, policy performance is never perfect. Problems are rarely “solved”; most often problems are resolved, reformulated, and even “unsolved.” To know whether a problem has been solved requires knowledge about observed policy outcomes, as well as knowledge about the extent to which these outcomes contribute to the valued opportunities for improvement that gave rise to a problem.
Knowledge Transformations
The five types of policy-relevant knowledge are interdependent. The solid lines connecting each pair of components in Figure 1.1 represent knowledge transformations, where one type of knowledge is changed into another so that the creation of knowledge at any point depends on knowledge produced in an adjacent (and most often previous) phase. Knowledge about policy performance, for example, depends on the transformation of prior knowledge about observed policy outcomes. The reason for this dependence is that any assessment of how well a policy achieves its objectives assumes that we already have reliable knowledge about the outcomes of that policy. Note, however, that types of knowledge are connected with solid lines rather than arrows to show that knowledge can be transformed backward and forward in an iterative fashion. The process of transformation is rarely linear.
Knowledge about policy problems is a special case. Knowledge about policy problems contains other types of knowledge. Some types of knowledge may be included—for example, knowledge about preferred policies—and others excluded. What is included or excluded in the formulation of a problem affects which policies are eventually prescribed, which values are chosen as criteria of policy performance, and which expected outcomes warrant or do not warrant attention. At the risk of being repetitious, it is worth stressing that a fatal error of policy analysis is a Type III error—defining the wrong problem.
Policy-Analytic Methods
The five types of policy-relevant knowledge are produced and transformed by using policy-analytic methods, which are the vehicles driving the production and transformation of knowledge. Methods involve judgments of different kinds: judgments to accept or reject an explanation, to affirm or dispute the rightness or wrongness of a preferred policy, to prescribe or not prescribe a preferred policy, to accept or reject a prediction about an expected outcome, to formulate a problem in one way rather than another.
In policy analysis, these methods have special names:
- Problem structuring. Problem-structuring methods are employed to produce knowledge about what problem to solve. Problem-structuring methods include the influence diagram and decision tree presented in Case 1.2 of this chapter (“Using Influence Diagrams and Decision Trees to Structure Problems of Energy and Highway Safety Policy”). Other examples of problem-structuring methods include argument mapping (Case 1.3, “Mapping International Security and Energy Crises”). Chapter 3 of this book covers problem-structuring methods more extensively.
Forecasting. Forecasting methods are used to produce knowledge about expected policy outcomes. Although many kinds of forecasting methods are covered in Chapter 4, an example of a simple forecasting tool is the scorecard described in Case 1.1 (The Goeller Scorecard and Technological Change). Scorecards, which are based on the judgments of experts, are useful in identifying the expected outcomes of science and technology policies.
- Prescription. Methods of prescription are employed to create knowledge about preferred policies. An example of a prescriptive method is the spreadsheet (in Case 1.2, “Using Influence Diagrams and Decision Trees”). The spreadsheet goes beyond the identification of expected policy outcomes by expressing consequences in terms of monetary benefits and costs. Benefit-cost analysis and other methods of prescription are presented in Chapter 5.
Monitoring. Methods of monitoring are employed to produce knowledge about observed policy outcomes. The scorecard (Case 1.1) is a simple method for monitoring observed policy outcomes as well as forecasting expected policy outcomes. Chapter 6 covers methods of monitoring in detail.
Evaluation. Evaluation methods are used to produce knowledge about the value or utility of observed policy outcomes and their contributions to policy performance.
The first method, problem structuring, is about the other methods. For this reason, it is a metamethod (method of methods). In the course of structuring a problem, analysts typically experience a “troubled, perplexed, trying situation, where the difficulty is, as it were, spread throughout the entire situation, infecting it as a whole.” Problem situations are not problems, because problems are representations or models of problem situations. Hence, problems are not “out there” in the world but stem from the interaction of the thoughts of many persons and the external environments in which they work or live. It is important to understand that analysts with different perspectives see the same problem situation in different ways. Imagine a graph showing increased national defense expenditures in trillions of dollars. The problem situation, represented by the graph, may be seen by one stakeholder as evidence of increasing national security (more of the budget is allocated to defense) and by another as an indication of declining resources for social services (less of the budget can be allocated to social services). Problem structuring, a process of testing different formulations of a problem situation, governs the production and transformation of knowledge produced by other methods. Problem structuring, which is important for achieving approximate solutions to ill-structured or “wicked” problems, is the central guidance system of policy analysis.
Policy-analytic methods are interdependent. It is not possible to use one method without using others. Thus, although it is possible to monitor past policies without forecasting their future consequences, it is usually not possible to forecast policies without first monitoring them. Similarly, analysts can monitor policy outcomes without evaluating them, but it is not possible to evaluate an outcome without first monitoring the existence and magnitude of an outcome. Finally, to select a preferred policy typically requires that analysts have already monitored, evaluated, and forecasted outcomes. This is another way of saying that policy prescription is based on factual as well as value premises.
Figure 1.1 supplies a framework for integrating methods from different disciplines and professions. The five policy-analytic methods are used across political science, sociology, economics, operations research, public administration, program evaluation, and ethics.
Some methods are used solely or primarily in some disciplines, and not others. Program evaluation, for example, employs monitoring to investigate whether a policy is causally relevant to an observed policy outcome. Although program evaluation has made extensive use of interrupted time-series analysis, regression-discontinuity analysis, causal modeling, and other techniques associated with the design and analysis of field experiments, implementation research within political science has not. Instead, implementation researchers have relied mainly on techniques of case study analysis. Another example comes from forecasting. Although forecasting is central to both economics and systems analysis, economics has drawn almost exclusively on econometric techniques. Systems analysis has made greater use of qualitative forecasting techniques for synthesizing expert judgment, for example, the Delphi technique and other qualitative techniques of synthesizing expert judgments.
FORMS OF POLICY ANALYSIS
Relationships among types of knowledge and methods provide a basis for contrasting different forms of policy analysis (Figure 1.2).
Prospective and Retrospective Analysis
Prospective policy analysis involves the production and transformation of knowledge before prescriptions are made. Prospective, or ex-ante analysis, shown as the right half of Figure 1.2, typifies the operating styles of economists, systems analysts, operations researchers, and decision analysts. The prospective form of analysis is what Williams means by policy analysis. Policy analysis is a means of analyzing knowledge “to draw from it policy alternatives and preferences stated incomparable, predicted quantitative and qualitative terms . . . it does not include the gathering of knowledge.” Policy research, by contrast, refers to studies that use social science methods to describe or explain phenomena. However, prospective analysis is often attended by gaps between the choice of preferred solutions and efforts to implement them. The reason for these gaps is that only a small amount of work required to achieve a desired set of objectives is carried out before policies are implemented. “It is not that we have too many good analytic solutions to problems,” writes Graham Allison. “It is, rather, that we have more good solutions than we have appropriate actions.” Retrospective policy analysis, a potential solution, is displayed in the left half of Figure 1.2. This form of ex-post analysis involves the production and transformation of knowledge after policies have been
Discipline-oriented analysts. This group, comprised mainly of political scientists, economists, and sociologists, seeks to develop and test discipline-based theories about the causes and consequences of policies. This group is not concerned with the identification of “policy” variables that are subject to manipulation and those that are not. For example, the analysis of the effects of party competition on government expenditures provides little or no knowledge about specific policy goals and how to attain them, because party competition is not a variable that policymakers can manipulate to change government expenditures.
Problem-oriented analysts. This group, again composed mainly of political scientists, economists, and sociologists, also seeks to describe the causes and consequences of policies. However, problem-oriented analysts are less concerned, if at all, with the development and testing of theories that are important to social science disciplines. Rather they are concerned with identifying variables that may explain a problem. Problem-oriented analysts are not concerned with specific objectives, because the orientation of problem-oriented analysts is general, not specific. For example, the analysis of quantitative data on the effects of ethnicity and poverty on achievement test scores helps explain the causes of inadequate test performance but does not provide knowledge about specific policies that can be manipulated to solve the problem.
Applications-oriented analysts. A third group includes groups of applied political scientists, applied economists, applied sociologists, and applied psychologists, as well as members of professions such as public administration, social work, and evaluation research. This group also seeks to describe the causes and consequences of policies and pays little attention to the development and testing of theories. However, this group, in contrast to the previously described groups, is dedicated to the identification of manipulable policy variables that can potentially achieve specific objectives that can be monitored and evaluated to evaluate the success of policies. For example, applications-oriented analysts may address early childhood reading readiness programs that can be used to achieve higher scores on standardized tests.
The operating styles of the three groups reflect their characteristic strengths and weaknesses. Discipline-oriented and problem-oriented analysts seldom produce knowledge that is directly useful to policymakers. Even when problem-oriented analysts investigate problems such as educational opportunity, energy conservation, and crime control, the result is macro negative knowledge, that is, the knowledge that describes the basic (or “root”) causes of policies with the aim of showing why policies do not work. By contrast, micro positive knowledge shows what policies do work under specified conditions. In this context, it is practically important to know that a specific form of gun control reduces the commission of crimes or that intensive police patrolling is a deterrent to robberies. But it is of little practical value to know that the crime rate is higher in urban than rural areas, or that there is an association between crime and poverty.
Even when applications-oriented analysts provide micro-positive knowledge, however, they may find it difficult to communicate with practitioners of ex-ante policy analysis, because applications-oriented analysts focus on observed (retrospective) outcomes rather than expected (prospective) ones. In agency settings, ex-ante analysts, whose job it is to find optimally efficient future solutions, generally lack knowledge about policy outcomes observed through retrospective analysis. Practitioners of ex-ante analysis also may fail to specify in sufficient detail the kinds of policy-relevant knowledge that will be most useful for monitoring, evaluating, and implementing their recommendations. For this reason, as Alice Rivlin has noted, “the intended outcomes of a policy are so vague that “almost any evaluation of it may be regarded as irrelevant because it missed the ‘problem’ toward which the policy was directed.” Moreover, legislators often formulate problems in general terms in order to gain acceptance, forestall opposition, and maintain political flexibility.
Contrasts among the operating styles of analysts suggest that discipline-oriented and problem-oriented analysis provides knowledge that is less useful than applications-oriented analysis because retrospective (ex-post) analysis as a whole is perhaps less effective in solving problems than prospective (ex-ante) analysis. This conclusion, frequently argued by economists, has merit from the point of view that policymakers need advice on what actions to take in the future. Nevertheless, retrospective analysis has important benefits. It places primary emphasis on the results of observed outcomes of action—that is, what has actually worked or not—and is not content with speculations about expected policy outcomes. Discipline-oriented and problem-oriented analysis may offer new frameworks for understanding policymaking, challenging conventional wisdom, challenging social and economic myths, and shaping the climate of opinion. Retrospective analysis, while it has affected intellectual priorities and understandings, has performed less well in offering potential solutions for specific problems.
Descriptive and Normative Analysis
Figure 1.2 also captures another important contrast, the distinction between descriptive and normative analysis. Descriptive policy analysis parallels descriptive decision theory, which refers to a set of logically consistent propositions that describe or explain the action. Descriptive decision theories may be tested against observations obtained through monitoring and forecasting. Descriptive theories and conceptual frameworks tend to originate in political science, sociology, and economics. The main function of these theories and frameworks is to explain, understand, and predict policies by identifying patterns of causality, also known as causal mechanisms. The function of approaches to monitoring such as field experimentation is to establish the approximate validity of causal inferences relating policies to their presumed outcomes. In Figure 1.2, descriptive policy analysis can be visualized as an axis moving from the lower-left quadrant (monitoring) to the upper right quadrant (forecasting).
Normative policy analysis parallels normative decision theory, which refers to a set of logically consistent propositions that evaluate or prescribe action. In Figure 1.2, normative policy analysis can be visualized as an axis running from the lower right quadrant (prescription) to the upper left quadrant (evaluation). Different kinds of knowledge are required to test normative, as distinguished from, descriptive decision theories. Methods of evaluation and prescription provide knowledge about policy performance and preferred policies, policies that have been or will be optimally efficient because benefits outweigh costs, or optimally equitable because those most in need are made better off. One of the most important features of normative policy analysis is that its propositions rest on values such as efficiency, effectiveness, equity, responsiveness, liberty, enlightenment, and security.
Problem Structuring and Problem Solving
The internal and external cycles of Figure 1.2 provide another important distinction. The inner cycle designates processes of problem structuring. Procedures of problem structuring are designed to identify elements that go into the definition of a problem, but not to identify solutions. What are the main elements of a problem? Are they political, economic, social, ethical, or all of these? How is the problem structured, that is, organized into a specific configuration of elements, for example, a linear sequence or as a complex system? Who are the most important stakeholders who affect and are affected by the problem? Have appropriate objectives been identified? Which alternatives are available to achieve the objectives? Are there uncertain events that should be taken into account? Are we solving the “right” problem rather than the “wrong” one?
By contrast, problem-solving methods are located in the outer cycle of Figure 1.2. They are designed to solve rather than structure a problem. Problem-solving is primarily technical in nature, in contrast to problem structuring, which is primarily conceptual. Methods such as econometric forecasting or benefit-cost analysis are problem-solving methods. For problems to be solved, they must be appropriately structured. For instance, problem-solving methods answer questions about the amount of variance in an outcome that may be explained by an independent variable. What is the probability of obtaining variance as large or larger than that obtained? What are the net benefits of different policies? What is their expected utility or payoff?
Integrated and Segmented Analysis
Integrated policy analysis links the two halves of Figure 1.2, the four quadrants, and the inner and outer cycles. Retrospective and prospective forms of analysis are joined in one continuous process. Descriptive and normative forms of analysis are also linked, as are methods designed to structure problems and also solve them. Practically speaking, this means that integrated policy analysis bridges the several main segments of multidisciplinary policy analysis, especially economics and political science.
Today, this need is not being properly met by specialized social science disciplines, which often produce segmented policy analysis. Today, the job of bridging these segmented disciplines is carried out within multidisciplinary professions including public administration, planning, management, and policy analysis. The American Society for Public Administration (ASPA), the National Association of Schools of Public Affairs and Administration (NASPAA), the American Planning Association (APA), the International Association of Schools and Institutes of Administration (IASIA), the Academy of Management (AM), the Operations Research Society of America (ORSA), and the Association for Public Policy and Management (APPAM) are multidisciplinary professions. So far, these professions have been far more open to economics, political science, and other social sciences disciplines than these disciplines have been open to them, notwithstanding a virtual consensus among policy analysts and practitioners that multidisciplinary research is essential for solving real world problems, which do not respect disciplinary boundaries.
The framework for integrated policy analysis with which we began this chapter (Figure 1.1) helps examine the assumptions, strengths, and limitations of methods employed in disciplines that are so specialized that they are difficult to employ for practical problem-solving. However, the framework for integrated policy analysis identifies and relates the goals of methods of policy analysis, enabling us to see the specific functions performed by methods of problem structuring, monitoring, evaluation, forecasting, and prescription. The framework identifies different types of policy analysis: prospective (ex-ante) and retrospective (ex-post), descriptive and normative, and problem structuring and problem-solving. The framework explains why we define policy analysis as a problem-solving discipline that links social science theories, methods, and substantive findings to solve practical problems.
Public policy is whatever governments choose to do or not to do.
Policy analysis is one activity for which there can be no fixed program, for policy analysis is synonymous with creativity, which may be stimulated by theory and sharpened by practice, which can be learned but not taught.
In large part, it must be admitted, knowledge is negative. It tells us what we cannot do, where we cannot go, wherein we have been wrong, but not necessarily how to correct these errors. After all, if current efforts were judged wholly satisfactory, there would be little need for analysis and less for analysts.
Policy analysis is often limited by disagreements over the nature of societal problems, by subjectivity in the interpretation of results, by limitations to the design of policy research, and by the complexity of human behavior.
Thomas R. Dye (1992)
“Public Policies are those policies developed by governmental bodies and officials”
James E. Anderson (1970)
Analisis kebijakan adalah sebagai suatu disiplin ilmu sosial terapan yang menggunakan argumentasi rasional dengan menggunakan fakta-fakta untuk menjelaskan, menilai, dan membuahkan pemikiran dalam rangka upaya memecahkan masalah publik
ilmu sosial terapan, artinya suatu hasil nyata dari suatu misi ilmu pengetahuan yang terlahir dari gerakan profesionalisme ilmu-ilmu sosial.
Duncan MacRae (1976)
