«Julia Abelson François-Pierre Gauvin Research Report P|06 Public Involvement Network March 2006 Assessing the Impacts of Public Participation: ...»
Assessing the Impacts
of Public Participation:
Concepts, Evidence and
Research Report P|06
Public Involvement Network
Assessing the Impacts of Public
Participation: Concepts, Evidence, and
Julia Abelson, Ph.D.
François-Pierre Gauvin, M.A., Doctoral Student
Centre for Health Economics and Policy Analysis McMaster University March 2006 Assessing the Impacts of Public Participation: Concepts, Evidence, and Policy Implications Canadian Policy Research Networks is a not-for-profit organization. Our mission is to help make Canada a more just, prosperous and caring society. We seek to do this through excellent and timely research, effective networking and dissemination, and by providing a valued neutral space within which an open dialogue among all interested parties can take place. www.cprn.org © Canadian Policy Research Networks Inc. – March 2006
Canadian Policy Research Networks Inc. (CPRN) 600-250 Albert Street Ottawa, ON K1P 6M1 Tel: (613) 567-7500 Fax: (613) 567-7640 Web site: http//www.cprn.org i Assessing the Impacts of Public Participation: Concepts, Evidence, and Policy Implications
TABLE OF CONTENTSEXECUTIVE SUMMARY
1.1 APPROACH TO THE REVIEW
1.2 THE QUESTION AND PURPOSE OF EVALUATION
2 ASSESSING THE CURRENT STATE OF PUBLIC PARTICIPATION EVALUATION.......... 4 2.1 UNPACKING THE CHALLENGES OF EVALUATION
2.2 TRACING THE HISTORY OF EVALUATION FRAMEWORK DEVELOPMENTS
2.3 FROM FRAMEWORKS TO CRITERIA: OPERATIONALIZING EVALUATION
3 A PLURALITY OF EVALUATION APPROACHES AND METHODS
3.1 EVALUATION APPROACHES
3.2 MEASUREMENT PROBLEMS
4 A CONCEPTUAL MAP OF PUBLIC PARTICIPATION EVALUATION
5 REVIEWING THE RESEARCH EVIDENCE
5.1 CHARACTERISTICS OF THE EMPIRICAL EVALUATION LITERATURE
5.2 DISTILLING THE REVIEWS OF PUBLIC PARTICIPATION IMPACTS
6 PRACTITIONER PERSPECTIVES
6.1 PERSPECTIVES FROM PRIOR STUDIES
6.2 PERSPECTIVES FROM KEY INFORMANT INTERVIEWS
7 RESEARCH GAPS
7.1 EVALUATE CONTEXT MORE RIGOROUSLY
7.2 DEFINE AND CATEGORIZE PUBLIC PARTICIPATION MECHANISMS MORE CONSISTENTLY.......... 32
7.3 LINK EMPIRICAL RESEARCH STUDIES MORE CLOSELY TO WELL-ARTICULATED HYPOTHESES. 327.4 USE MULTIPLE DISCIPLINARY PERSPECTIVES AND METHODS IN EVALUATION DESIGN............ 33
7.5 MAKE BETTER USE OF REAL-WORLD DELIBERATIVE EXPERIMENTS TO ADVANCE PROCESS ANDOUTCOME EVALUATION
7.6 EXPLORE DECISION MAKERS AND THEIR ORGANIZATIONS MORE FULLY AS CONTEXT ANDOUTCOME VARIABLES
8 PRACTICE AND POLICY GAPS: BARRIERS TO PUBLIC PARTICIPATION,EVALUATION AND ITS UPTAKE
8.1 PROMISING TREND?
APPENDIX 1 - POSITION DESCRIPTIONS OF INTERVIEWEES
EXECUTIVE SUMMARYThe degree of attention being given to expanding ordinary citizens’ roles in the policy process underscores the need to consider what effects these processes might and will have on policy decisions and on those who participate in them. This paper explores what is known about the extent to which the goals of public participation in policy have been met. It also examines the extent to which research evidence has been used by policy makers and public participation practitioners to design and improve public participation.
The authors review the current state of knowledge about the impact of public participation on policy and civic literacy and identify different conceptual and methodological approaches to evaluation and their associated challenges. In addition to theoretical and conceptual literature, the authors also review published (English and French) empirical public participation evaluation literature and incorporate the results of key informant interviews with policy makers and public participation practitioners. The paper concludes with a discussion of the limits to evaluation and its uptake, and recommends strategies for promoting further practice and methods of public participation evaluation.
Some progress has been made in improving the rigour with which public participation evaluation is undertaken, chiefly through the development of more explicit and agreed upon evaluation criteria that have both process and outcome evaluation properties.
However, much of the progress made has been focused on improving what are still largely formative evaluation studies (i.e. efforts to improve on existing practice and to assess public participation against an a priori set of objectives for what constitutes successful public participation).
There are a plurality of evaluation approaches and methods but it is possible to identify three main approaches: 1) user-based which assumes different participants have different goals and that evaluation must take these into account; 2) theory-based which is driven by theories and models of public participation and applies normative evaluation universally to any public participation effort; and 3) goal-free evaluation which is not constituted by any stated goals and is conducted in the absence of any theory. Most evaluation studies to date fall under the user-based category. Another distinguishing feature of public participation evaluation is the emphasis on either process or outcome evaluation.
Evaluation processes face a number of theoretical and practical challenges. First, the task of defining the end-point of a participation exercise for purposes of measuring effectiveness is often unclear. The ability to measure the institutional and societal impacts of the process, which can take many years, and may be difficult to disentangle from other events that are influential to the policy process, may be limited. Second, the public participation process may be well run according to some criteria but not others. How do we determine how much credence to give to an apparently acceptable, democraticallydriven recommendation? The third big challenge lies with measurement criteria.
Participant satisfaction is routinely used as a measure of success despite problems iii Assessing the Impacts of Public Participation: Concepts, Evidence, and Policy Implications associated with its interpretation. The issue of perceived versus actual impacts is problematic in any evaluation. Likewise the absence of properly tested measurement tools is another area in need of attention. The authors offer a conceptual map of public participation evaluation that includes context evaluation (public policy process), process evaluation (public participation process) and outcome evaluation (public policy, decisionmakers and participants/general public) as an aid in understanding the different types and approaches to evaluation.
The research evidence from empirical evaluation literature is grouped under processoriented research and outcome-oriented studies. The former reveals that “process matters” and that different types of public participation should be designed for different types of issues, decision-making conditions and groups of participants. What is much less clear is which of these contextual variables matter most and which processes are better suited to each of these different sets of arrangements.
Most of the outcome-oriented empirical research has focused on assessing the impacts of public participation processes on a range of citizen participant attributes. These studies have reported: increased levels of interest and knowledge of public issues; improved capacity for future public involvement; increased propensity for social bond formation;
and improved trust of fellow citizens. There has been less research about the direct impacts of public participation on the policy process and political decision-making. What has been produced offers mixed and ambiguous results (e.g., public deliberation can produce outcomes that influence policy but the conditions under which this occurs are not easily identifiable). However, more recent literature finds a strong association between the broad acceptance of the decision outcomes and ‘processes in which agencies are responsive, participants are motivated, the quality of deliberation is high, and the participants have at least a moderate degree of control over the process’ (Beierle and Cayford 2002).
To supplement the literature review, the authors sought the views of policy makers / public participation practitioners working within various levels of government across Canada. Key informant interviews undertaken for the paper were organized around four issues: approaches to public participation design; approaches to evaluation; barriers to evaluation; and how to foster evaluation/improve its quality and use. The key themes emerging from the key informant interviews are captured below.
The literature consulted identified a dearth of good quality research evidence to inform either policy makers or public participation practitioners of the impacts of public participation on political discourse and/or democratic participation. The following
research gaps are identified by the authors:
1. Evaluate the context more rigorously.
There is a remarkable convergence in the literature about the need for more rigorous study of the role context plays in the public participation process. Rowe and Frewer call for categories of contextual attributes that are associated with the implementation of public involvement to be developed (e.g., characteristics of the issue, attributes of the sponsoring organization, the type of decision being made, and the decision timeline).
2. Define and categorize public participation mechanisms more consistently.
This would help to improve the generalizability of the current evaluation literature.
3. Link empirical research studies more closely with well-articulated hypotheses.
Bridge the two solitudes between scholars and policy makers’ interests by defining a set of organizationally derived hypotheses that can be tested within a public participation evaluation.
4. Use multiple disciplinary perspectives and methods in evaluation design.
Include interviews, surveys, documentation and observation.
5. Make better use of real-world deliberative experiments to advance process and outcome evaluation.
Organizations conducting public participation innovations should be open to collaboration with academic and NGO researchers to design and implement evaluation.
6. Explore decision makers and their organizations more fully as context and outcome variables.
The articulation of a clearer set of relationships between decision makers, their organizations and the influences they exert on each other through the design, implementation and evaluation of public participation is a useful avenue for further research.
Despite years of documenting public participation experiences, the practice of public participation evaluation is still in its infancy. More work is needed to reach agreement about a common set of evaluation criteria, the defining features of public participation mechanisms and how to categorize and evaluate the role of contextual variables in shaping and influencing public participation. To achieve these goals, the authors encourage forums that bring together public participation scholars, practitioners and policy makers from a variety of policy sectors and levels of governance for the purposes of general knowledge exchange, but also with the specific objectives of seeking v Assessing the Impacts of Public Participation: Concepts, Evidence, and Policy Implications agreement about evaluation frameworks and criteria and in particular, the balance between generic and specific frameworks. Should they be successful, these exchanges may help to shift current views toward public participation evaluation from “frill” to “essential”.