Evaluating the impact of economic policy research: Concepts, practices and lessons
Ryan, J.
Evaluating the impact of economic policy research: Concepts, practices and lessons - Mexico, DF (Mexico) CIMMYT : 2003 - p. 32-33 - Printed
Abstract only
Evaluating the impact of economic policy research remains a challenging assignment, with few â€best practices†to draw on. Case studies remain the preferred way forward in the absence of an agreed- upon impact evaluation paradigm. The major problems are those of measurement, sampling and attribution. The approach to impact evaluation is conditioned, to a significant extent, by the primary purpose that underpins such studies. For a research institution, this is to improve accountability and credibility; quality and relevance; program/project design and implementation; future planning and prioritizing. However, depending on the primary purpose, the unit of analysis and approach tend to differ. The importance of understanding policy processes when undertaking impact evaluation is discussed in the paper. This includes the pros and cons of adopting a “supply-side†versus “demand-side†perspective and recognition of the policy formulation, decision-making and implementation phases when judging the influence of policy research. The products of economic policy research can be delineated into four elements: outputs, outcomes, policy responses and final environmental and socio-economic impacts. The paper describes various indicators that can be used to articulate and measure these products and alternative qualitative and quantitative approaches that can be employed to elicit them. Key issues that continue to arise in the actual conduct of impact evaluations of economic policy research are discussed under nine headings: scale and scope; time horizon; supply- versus demand-side approaches; the importance of surprise; attribution; choice of indicators; case studies; time lags; and ex ante and ex post assessments. Methods of enhancing impact are then discussed, including the importance of effective communications and an understanding of the policy environment and processes. Lessons are drawn from a number of case studies commissioned by the International Food Policy Research Institute (IFPRI) which enable an assessment of the influence and impact of IFPRI's research and related activities on a number of policy themes. The lessons are intended to guide the conduct of future impact studies; assist in articulating, measuring, and documenting successes and failures; and help enhance impacts in the future. Aspects addressed include the importance of research quality, timeliness, and communications; type of collaboration and collaborators; the policy environment; data availability; time scale; consensus building; and the breadth of experience on which to draw. The paper concludes that, in the last five years, progress has been made in the conduct of case studies designed to assess the impact of economic policy research and lessons have been drawn for the future. There has also been some progress in the development of methodologies for quantifying impact in economic terms. However, a number of issues remain. These include attribution, measurement and the enhancement of impact. As the need for more accountability has driven much of the work on impact assessment in recent years, attribution has been, and is still is, a challenging question to which answers remain to be found. It seems that it is preferable to begin from the "demand- side" in impact assessment. This entails using major policy events as the starting point and working retrospectively to establish the separate influences of the many research suppliers and other factors on policy responses. When the interest is on the impact of particular institutions in a "supply-side" approach, whole bodies of work on topics rather than individual projects of limited duration would appear to be preferred. As far as possible, joint impacts of various players should be measured rather than trying to separate the contributions of individual institutions. There is a need for more research on sampling and elicitation techniques to ensure objectivity and devise best practices. In this quest, it seems that exploring the elements of surprise in research information that policy makers were exposed to offers considerable promise. This is especially so with Bayesian approaches. The most appropriate approaches to impact assessment should invo1ve a mixture of both qualitative and quantitative methods. Retrospective narratives are an essential component of the former and, indeed, provide the basis for quantitative estimates and the related issue of attribution. In order to be effective, impact evaluation must be institutionalized and not simply an exercise in accountability. Staff and management should have responsibility for recording outputs, outcomes and policy responses from their research. Independent evaluators can verify these and endeavor to translate them into meaningful measures of their impact on economic welfare. Researchers have a responsibility to ensure that their research is disseminated to policy makers using appropriate communications media. A degree of advocacy is also appropriate. However this should not be taken to lengths that might compromise the independence and credibility of the researcher. With the increased availability of IT and the growing role of participatory democracy and good governance in developing countries, there is increased scope for credible policy research to be accessed by disparate groups and thereby better inform the policy process. A remaining difficulty is the attribution of economic value when economic policy research reinforces current policy settings. Economic policy research that illuminates the distributional consequences of current and/ or prospective policies seems to have the most influence on policy makers. Estimates of deadweight economic losses do not seem to be nearly as influential. In order to further refine approaches and methodologies, we need to continue to undertake more case studies and apply the lessons learnt to enhance future impact and help to define "best practices."
English
970-648-076-5
Agricultural economics
Economic policies
Economic systems
Experimentation
Food policies
Planning
Project design
Quality
Agricultural research
CIMMYT
338.91 / WAT
Evaluating the impact of economic policy research: Concepts, practices and lessons - Mexico, DF (Mexico) CIMMYT : 2003 - p. 32-33 - Printed
Abstract only
Evaluating the impact of economic policy research remains a challenging assignment, with few â€best practices†to draw on. Case studies remain the preferred way forward in the absence of an agreed- upon impact evaluation paradigm. The major problems are those of measurement, sampling and attribution. The approach to impact evaluation is conditioned, to a significant extent, by the primary purpose that underpins such studies. For a research institution, this is to improve accountability and credibility; quality and relevance; program/project design and implementation; future planning and prioritizing. However, depending on the primary purpose, the unit of analysis and approach tend to differ. The importance of understanding policy processes when undertaking impact evaluation is discussed in the paper. This includes the pros and cons of adopting a “supply-side†versus “demand-side†perspective and recognition of the policy formulation, decision-making and implementation phases when judging the influence of policy research. The products of economic policy research can be delineated into four elements: outputs, outcomes, policy responses and final environmental and socio-economic impacts. The paper describes various indicators that can be used to articulate and measure these products and alternative qualitative and quantitative approaches that can be employed to elicit them. Key issues that continue to arise in the actual conduct of impact evaluations of economic policy research are discussed under nine headings: scale and scope; time horizon; supply- versus demand-side approaches; the importance of surprise; attribution; choice of indicators; case studies; time lags; and ex ante and ex post assessments. Methods of enhancing impact are then discussed, including the importance of effective communications and an understanding of the policy environment and processes. Lessons are drawn from a number of case studies commissioned by the International Food Policy Research Institute (IFPRI) which enable an assessment of the influence and impact of IFPRI's research and related activities on a number of policy themes. The lessons are intended to guide the conduct of future impact studies; assist in articulating, measuring, and documenting successes and failures; and help enhance impacts in the future. Aspects addressed include the importance of research quality, timeliness, and communications; type of collaboration and collaborators; the policy environment; data availability; time scale; consensus building; and the breadth of experience on which to draw. The paper concludes that, in the last five years, progress has been made in the conduct of case studies designed to assess the impact of economic policy research and lessons have been drawn for the future. There has also been some progress in the development of methodologies for quantifying impact in economic terms. However, a number of issues remain. These include attribution, measurement and the enhancement of impact. As the need for more accountability has driven much of the work on impact assessment in recent years, attribution has been, and is still is, a challenging question to which answers remain to be found. It seems that it is preferable to begin from the "demand- side" in impact assessment. This entails using major policy events as the starting point and working retrospectively to establish the separate influences of the many research suppliers and other factors on policy responses. When the interest is on the impact of particular institutions in a "supply-side" approach, whole bodies of work on topics rather than individual projects of limited duration would appear to be preferred. As far as possible, joint impacts of various players should be measured rather than trying to separate the contributions of individual institutions. There is a need for more research on sampling and elicitation techniques to ensure objectivity and devise best practices. In this quest, it seems that exploring the elements of surprise in research information that policy makers were exposed to offers considerable promise. This is especially so with Bayesian approaches. The most appropriate approaches to impact assessment should invo1ve a mixture of both qualitative and quantitative methods. Retrospective narratives are an essential component of the former and, indeed, provide the basis for quantitative estimates and the related issue of attribution. In order to be effective, impact evaluation must be institutionalized and not simply an exercise in accountability. Staff and management should have responsibility for recording outputs, outcomes and policy responses from their research. Independent evaluators can verify these and endeavor to translate them into meaningful measures of their impact on economic welfare. Researchers have a responsibility to ensure that their research is disseminated to policy makers using appropriate communications media. A degree of advocacy is also appropriate. However this should not be taken to lengths that might compromise the independence and credibility of the researcher. With the increased availability of IT and the growing role of participatory democracy and good governance in developing countries, there is increased scope for credible policy research to be accessed by disparate groups and thereby better inform the policy process. A remaining difficulty is the attribution of economic value when economic policy research reinforces current policy settings. Economic policy research that illuminates the distributional consequences of current and/ or prospective policies seems to have the most influence on policy makers. Estimates of deadweight economic losses do not seem to be nearly as influential. In order to further refine approaches and methodologies, we need to continue to undertake more case studies and apply the lessons learnt to enhance future impact and help to define "best practices."
English
970-648-076-5
Agricultural economics
Economic policies
Economic systems
Experimentation
Food policies
Planning
Project design
Quality
Agricultural research
CIMMYT
338.91 / WAT