[EQ] Making an Impact: Framework and Indicators to Measure Returns on Investment in Health Research

MAKING AN IMPACT

A Preferred Framework and Indicators to Measure Returns on Investment in Health Research

 

Report of the Panel on the Return on Investments in Health Research

Canadian Academy of Health Sciences/Académie canadienne des sciences de la santé
January 2009

 

Available online PDF file [136p.] at: http://www.cahs-acss.ca/e/pdfs/ROI_FullReport.pdf

 

"……Twenty three different organizations sponsored this assessment. They all share an interest in defining the impacts of health research and learning how to improve the returns on investments in health research. Our remit from these sponsors was: Is there a "best way" (best method) to evaluate the impacts of health research in Canada, and are there "best metrics" that could be used to assess those impacts (or improve them)?

 

Based on our assessment, we propose a new impacts framework and a preferred menu of indicators and metrics that can be used for evaluating the returns on investment in health research. The CAHS impact framework demonstrates how research activity informs decision making, eventually resulting in changes in health and economic and social prosperity (left to right arrow). The framework also shows how research impacts feed back upstream, potentially influencing the diffusion and impacts of other research, and creating inputs for future research (right to left arrow).

 

This framework builds on the combined logic model and impacts approach of the "payback model" (Buxton, M.J., and Hanney, S.R., 1996 � adapted by CIHR in Canada in 2005 and 2008), revised by our panel into a "systems approach" to capture impacts (this is shown at the bottom of the Figure). It is designed to be used as a roadmap to track health�research impacts in five main categories:
1) advancing knowledge,
2) building capacity,
3) informing decision�making,
4) health impacts, and
5) broad socio�economic impacts………."

 

Content

 

Volume 1

The Chair's Perspective

Rationale (and CAHS Prospectus) for this Assessment

Executive Summary

Panel Recommendations

1. Chapter 1: Background

1.1. Defining Health Research

1.1.a. The Canadian Health Research Landscape

1.1.b. What Returns are Expected from Canadian Health Research?

1.1.c. Many Stakeholders, Many Views

1.1.d. Different Evaluation Purposes

1.1.e. Why Measure Returns on Investment for Health Research in Canada?

1.1.f. What Evaluation of "Returns" is Already Taking Place in Canada?

1.2. Are "Health Research Impacts" Already Defined Elsewhere?

1.3. Economic Evaluations to Date

1.4. Summary of the Landscape

2. Chapter 2: Frameworks

2.1. Rationale for a Framework to Understand Health Research

2.1.a. Definitions

2.1.b. Stakeholders Needs

2.2. A Review of Frameworks and Their Use

2.3. Developing a Health Research Evaluation Framework for Canada

2.3.a. Identifying What Should be Modelled

2.3.b. Building a Framework for R&D Uptake

2.3.c. Impact Categories

2.3.d. Impact Categories and the Logic Model

3. Chapter 3: Strategies for Using the Framework

3.1. Using the Framework Appropriately

3.1.a. Avoiding Misuse of the Framework

3.1.b. The Four Pillars and the Framework

3.2. Costs of Evaluation

3.3. Issues in Evaluation: Attribution, the Counterfactual, Time�lags and Levels of Aggregation

3.4. Evaluation Methods

3.5. Data Collection

4. Chapter 4: Choosing Sets of Indicators and Metrics

4.1. Overview of Indicators and Metrics and Their Use

4.2. Defining "Appropriate Indicators"

4.3. Identifying Appropriate Indicators

4.3.a. Advancing Knowledge

4.3.b. Capacity Building

4.3.c. Informing Decision Making

4.3.d. Health Impacts

4.3.e. Broad Economic and Social Impacts

4.3.f. Theoretical examples of indicator sets for evaluation

5. Chapter 5: Conclusions

6. References

 

Volume 2

Appendices

The appendices present commissioned papers in areas where the report is not able to provide details, cover the background for the main report, and present the approach taken to the assessment process. The commissioned papers cover assessing the impacts of research in pillars II, III and IV but do not cover pillar I, since basic biomedical research is the area where most has been said on understanding the impacts of health research.

 

Appendix A: Commissioned Papers

- Pillar II: Clinical Research � "How to Optimally Measure the Impact of Health Research Funding in Clinical Research" by Ralph M. Meyer

- Pillar III: Health Services Research � "Estimating the Return on Investment from Health Services Research: A Theoretical and Empirical Analysis
   Steven Lewis, Patricia J. Martens and Louis Barre

- Pillar IV: Population and Public Health Research �"Assessing the Return on Canada's Public Investment in Population and Public Health Research:
  Methods and Metrics" by Alan Shiell and Erica Di Ruggiero

- Meso�Level Metrics for Impact � "Metrics for the Treatment Sector or Meso Level of the Canadian Health Care System" by Jerald Hage

- Ethics and Evaluating Health Research � "The Return on Investments (ROI) in Health Research: Ethical Aspects" by Michael McDonald and Bartha Knoppers

- Public Perspective on Health Research Funding � "Translating Science into Hope: The Public Perspective on Health Research Funding" by André Picard

- Health Research Evaluation Frameworks: An International Comparison by Philipp� Bastian Brutscher, Steven Wooding and Jonathan Grant

Appendix B: The Canadian Landscape for health research

Appendix C: Evaluation frameworks and methods

Appendix D: Issues for research evaluation

Appendix E: Indicators

Appendix F: Glossary

Appendix G: Methods

Appendix H: External Interviewees

Appendix I: Prospectus for a Major Assessment � The Return on Investments in Health Research: Defining the Best Metrics

 

 

 

Download the Appendices [pdf] 5.27 MB, (340 pages) ISBN: 978-0-9811589-1-4
Download the Assessment Summary [pdf] 1.15 MB, 24 pages
Pour Télécharger de « Résumé de l'évaluation » [pdf] 1.56 MB, 30 pages
Menu of Preferred Indicators [pdf] 1.56 MB, 8 pages

 

Website: http://www.cahs-acss.ca/e/assessments/completedprojects.php

 

 

 *      *     *

This message from the Pan American Health Organization, PAHO/WHO, is part of an effort to disseminate
information Related to: Equity; Health inequality; Socioeconomic inequality in health; Socioeconomic
health differentials; Gender; Violence; Poverty; Health Economics; Health Legislation; Ethnicity; Ethics;
Information Technology - Virtual libraries; Research & Science issues.  [DD/ KMC Area]

"Materials provided in this electronic list are provided "as is". Unless expressly stated otherwise, the findings
and interpretations included in the Materials are those of the authors and not necessarily of The Pan American
Health Organization PAHO/WHO or its country members".
------------------------------------------------------------------------------------
PAHO/WHO Website
Equity List - Archives - Join/remove: http://listserv.paho.org/Archives/equidad.html