A historical reflection on research evaluation studies,
their recurrent themes and challenges
Sonja Marjanovic, Stephen Hanney, Steven Wooding
2009 RAND Corporation
Available online PDF [55p.] at: http://www.rand.org/pubs/technical_reports/2009/RAND_TR789.pdf
“……This report presents a historical reflection on research evaluation studies, their recurrent themes and challenges, and their implications. It critically examines studies of how scientific research drives innovation and socioeconomic benefits.
…First, it provides a predominantly descriptive historical overview of some landmark studies in the research evaluation field, from the late 1950s until the present day, and highlights some of their key contributions. Then, it reflects on the historical overview analytically, in order to discuss some of the methodological developments and recurrent themes in research evaluation studies. The report concludes by discussing the enduring challenges in research evaluation studies and their implications.
We emphasise that this report does not address all of the key studies in the research evaluation field. The evaluation literature today is so extensive that a selective approach is necessary to focus on those studies that we feel provide the most valuable insights in the context of biomedical and health research evaluation. However, we do go back in history to revisit some early landmark studies outside of the biomedical or health research sectors, which had a particularly strong influence on informing evaluation thinking, policy and practice…..”
Contents
Summary
CHAPTER 1 A historical overview of research evaluation studies
1.1 Early landmark studies: the 1960s and 1970s
1.2 The evolution of studies concerned with innovation processes and outputs: from the 1980s to the present day
1.2.1 Process-oriented evaluation studies
1.2.2 Output-oriented evaluation studies
CHAPTER 2 Methodological developments:
case study-based approaches and conceptual frameworks in evaluation research
2.1 The Payback framework
2.2 Economic and Social Research Council framework
2.3 Royal Netherlands Academy of Arts and Sciences framework
2.4 Framework for impact segmentation by audience
2.5 US Agency for Healthcare and Research Quality impact model
CHAPTER 3 Consistent themes in research evaluation studies
3.1 Scientific accountability and justification of investments into research
3.2 Lack of convincing evidence on success factors
3.3 Nature of the innovation process
3.4 Factors influencing innovation processes
3.4.1 Funding mechanisms and research settings
3.4.2 The relative contributions of basic and applied science to innovation
3.4.3 Knowledge-exchange environments
3.5 Diversity and range of benefits from research
CHAPTER 4 Challenges in research evaluation studies
4.1 Apparent contradictions between studies as an artefact of research design
4.1.1 Types of innovations examined
4.1.2 Timeframes
4.2 Biases in the selection of cases
4.3 Lack of clarity and unity in the definitions of concepts
4.4 Challenges to the ability to replicate methodology: unclear descriptions of techniques used for data collection and analysis
4.5 The challenge of setting boundaries in research
4.6 Challenges in the attribution of impacts
4.7 Issues of sector idiosyncrasies
CHAPTER 5 Conclusion
Bibliography
Appendix: A summary of key studies and their insights
* * *
This message from the Pan American Health Organization, PAHO/WHO, is part of an effort to disseminate
information Related to: Equity; Health inequality; Socioeconomic inequality in health; Socioeconomic
health differentials; Gender; Violence; Poverty; Health Economics; Health Legislation; Ethnicity; Ethics;
Information Technology - Virtual libraries; Research & Science issues. [DD/ KMC Area]
“Materials provided in this electronic list are provided "as is". Unless expressly stated otherwise, the findings
and interpretations included in the Materials are those of the authors and not necessarily of The Pan American
Health Organization PAHO/WHO or its country members”.
------------------------------------------------------------------------------------
PAHO/WHO Website
Equity List - Archives - Join/remove: http://listserv.paho.org/Archives/equidad.html
Twitter http://twitter.com/eqpaho
IMPORTANT: This transmission is for use by the intended recipient and it may contain privileged, proprietary or confidential information. If you are not the intended recipient or a person responsible for delivering this transmission to the intended recipient, you may not disclose, copy or distribute this transmission or take any action in reliance on it. If you received this transmission in error, please notify us immediately by email to infosec@paho.org, and please dispose of and delete this transmission. Thank you.
