Health research: measuring the social, health and economic benefits
Cyril Frank and Edward Nason
Department of Surgery, University of Calgary (Frank), Calgary, Alta.; and the Canadian Policy Research Network (Nason), Ottawa, Ont.
CMAJ • MARCH 3, 2009 • 180(5) - Canadian Medical Association Journal
Available online at: http://www.cmaj.ca/cgi/content/full/180/5/528?etoc
URL: http://www.cmaj.ca/cgi/reprint/180/5/528
"…… the authors discuss two approaches to measure return on investment in health research. The "payback model," which was recently adopted and modified by the Canadian Institutes of Health Research, is favoured and may help optimize future evaluations.
"……..despite intense interest in defining the social, health and economic impacts of health research investments globally 1�5 and in Canada 6�10 as proof of value for- money, no validated method for measuring return on investments yet exists. Until now, issues of complexity combined with major gaps in methodology have limited the ability to link health research products to outcomes at a relevant level (e.g., to be useful to stakeholders: individual funders, decision-makers, institutions, researchers or clinicians).
In this article, we discuss current approaches to measuring returns on investment, analyze key issues and gaps that need to be bridged to improve returns on investment, and present a new method that may help overcome them………"
Key points
• Health research is expensive, and its explicit social, health and economic impacts are hard to define.
• There are many challenges and assumptions in defining specific returns on investment in health research.
• There is no common approach to tracking health research impacts.
• The payback model can be populated with validated indicators to track overall outcomes or outcomes in a specific health research area.
• Collaboration among funders will ensure cost-effective implementation of the new framework to quantify return on investment.
• The framework can be fine-tuned as necessary to improve indicator sets, overcome gaps and progressively define returns on investment in health research.
This analysis article comments on
MAKING AN IMPACT
A Preferred Framework and Indicators to Measure Returns on Investment in Health Research
Report of the Panel on the Return on Investments in Health Research
Canadian Academy of Health Sciences/Académie canadienne des sciences de la santé
January 2009
Volume 1 available online PDF file [136p.] at: http://www.cahs-acss.ca/e/pdfs/ROI_FullReport.pdf
Apendices available online as PDF [340p.] at: http://www.cahs-acss.ca/e/pdfs/ROI_Appendices.pdf
"……Twenty three different organizations sponsored this assessment. They all share an interest in defining the impacts of health research and learning how to improve the returns on investments in health research. Our remit from these sponsors was: Is there a "best way" (best method) to evaluate the impacts of health research in Canada, and are there "best metrics" that could be used to assess those impacts (or improve them)?
Based on our assessment, we propose a new impacts framework and a preferred menu of indicators and metrics that can be used for evaluating the returns on investment in health research. The CAHS impact framework demonstrates how research activity informs decision making, eventually resulting in changes in health and economic and social prosperity (left to right arrow). The framework also shows how research impacts feed back upstream, potentially influencing the diffusion and impacts of other research, and creating inputs for future research (right to left arrow).
This framework builds on the combined logic model and impacts approach of the "payback model" (Buxton, M.J., and Hanney, S.R., 1996 � adapted by CIHR in Canada in 2005 and 2008), revised by our panel into a "systems approach" to capture impacts (this is shown at the bottom of the Figure). It is designed to be used as a roadmap to track health�research impacts in five main categories:
1) advancing knowledge,
2) building capacity,
3) informing decision�making,
4) health impacts, and
5) broad socio�economic impacts
Content
Volume 1
The Chair's Perspective Rationale (and CAHS Prospectus) for this Assessment
Executive Summary
Panel Recommendations
1. Chapter 1: Background
1.1. Defining Health Research
1.1.a. The Canadian Health Research Landscape
1.1.b. What Returns are Expected from Canadian Health Research?
1.1.c. Many Stakeholders, Many Views
1.1.d. Different Evaluation Purposes
1.1.e. Why Measure Returns on Investment for Health Research in Canada?
1.1.f. What Evaluation of "Returns" is Already Taking Place in Canada?
1.2. Are "Health Research Impacts" Already Defined Elsewhere?
1.3. Economic Evaluations to Date
1.4. Summary of the Landscape
2. Chapter 2: Frameworks
2.1. Rationale for a Framework to Understand Health Research
2.1.a. Definitions
2.1.b. Stakeholders Needs
2.2. A Review of Frameworks and Their Use
2.3. Developing a Health Research Evaluation Framework for Canada
2.3.a. Identifying What Should be Modelled
2.3.b. Building a Framework for R&D Uptake
2.3.c. Impact Categories
2.3.d. Impact Categories and the Logic Model
3. Chapter 3: Strategies for Using the Framework
3.1. Using the Framework Appropriately
3.1.a. Avoiding Misuse of the Framework
3.1.b. The Four Pillars and the Framework
3.2. Costs of Evaluation
3.3. Issues in Evaluation: Attribution, the Counterfactual, Time�lags and Levels of Aggregation
3.4. Evaluation Methods
3.5. Data Collection
4. Chapter 4: Choosing Sets of Indicators and Metrics
4.1. Overview of Indicators and Metrics and Their Use
4.2. Defining "Appropriate Indicators"
4.3. Identifying Appropriate Indicators
4.3.a. Advancing Knowledge
4.3.b. Capacity Building
4.3.c. Informing Decision Making
4.3.d. Health Impacts
4.3.e. Broad Economic and Social Impacts
4.3.f. Theoretical examples of indicator sets for evaluation
5. Chapter 5: Conclusions
6. References
Volume 2: http://www.cahs-acss.ca/e/pdfs/ROI_Appendices.pdf - Appendices
The appendices present commissioned papers in areas where the report is not able to provide details, cover the background for the main report, and present the approach taken to the assessment process. The commissioned papers cover assessing the impacts of research in pillars II, III and IV but do not cover pillar I, since basic biomedical research is the area where most has been said on understanding the impacts of health research.
Appendix A: Commissioned Papers
- Pillar II: Clinical Research � "How to Optimally Measure the Impact of Health Research Funding in Clinical Research" by Ralph M. Meyer
- Pillar III: Health Services Research � "Estimating the Return on Investment from Health Services Research: A Theoretical and Empirical Analysis
Steven Lewis, Patricia J. Martens and Louis Barre
- Pillar IV: Population and Public Health Research �"Assessing the Return on Canada's Public Investment in Population and Public Health Research:
Methods and Metrics" by Alan Shiell and Erica Di Ruggiero
- Meso�Level Metrics for Impact � "Metrics for the Treatment Sector or Meso Level of the Canadian Health Care System" by Jerald Hage
- Ethics and Evaluating Health Research � "The Return on Investments (ROI) in Health Research: Ethical Aspects" by Michael McDonald and Bartha Knoppers
- Public Perspective on Health Research Funding � "Translating Science into Hope: The Public Perspective on Health Research Funding" by André Picard
- Health Research Evaluation Frameworks: An International Comparison by Philipp� Bastian Brutscher, Steven Wooding and Jonathan Grant
Appendix B: The Canadian Landscape for health research
Appendix C: Evaluation frameworks and methods
Appendix D: Issues for research evaluation
Appendix E: Indicators
Appendix F: Glossary
Appendix G: Methods
Appendix H: External Interviewees
Appendix I: Prospectus for a Major Assessment � The Return on Investments in Health Research: Defining the Best Metrics
* * *
This message from the Pan American Health Organization, PAHO/WHO, is part of an effort to disseminate
information Related to: Equity; Health inequality; Socioeconomic inequality in health; Socioeconomic
health differentials; Gender; Violence; Poverty; Health Economics; Health Legislation; Ethnicity; Ethics;
Information Technology - Virtual libraries; Research & Science issues. [DD/ KMC Area]
"Materials provided in this electronic list are provided "as is". Unless expressly stated otherwise, the findings
and interpretations included in the Materials are those of the authors and not necessarily of The Pan American
Health Organization PAHO/WHO or its country members".
------------------------------------------------------------------------------------
PAHO/WHO Website
Equity List - Archives - Join/remove: http://listserv.paho.org/Archives/equidad.html
