Reassessing RCTs as the ‘gold standard’: synergy not separatism in evaluation designs

Pam Hanley, Bette Chambers, Jonathan Haslam

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.
LanguageEnglish
Pages287-298
Number of pages12
JournalInternational Journal of Research and Method in Education
Volume39
Issue number3
Early online date2 Feb 2016
DOIs
Publication statusPublished - 2016
Externally publishedYes

Fingerprint

separatism
gold standard
synergy
evaluation
development of science
cooperative learning
privilege
research method
primary school
pupil
expertise
leader
paradigm
classroom
teacher
interview
school
knowledge
community
evidence

Cite this

@article{75554b7a00ad475dac1bb644cfc37b91,
title = "Reassessing RCTs as the ‘gold standard’: synergy not separatism in evaluation designs",
abstract = "Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.",
keywords = "evaluation, evidence-based education, mixed methods, RCT, STEM",
author = "Pam Hanley and Bette Chambers and Jonathan Haslam",
year = "2016",
doi = "10.1080/1743727X.2016.1138457",
language = "English",
volume = "39",
pages = "287--298",
journal = "International Journal of Research and Method in Education",
issn = "1743-727X",
publisher = "Taylor and Francis Ltd.",
number = "3",

}

Reassessing RCTs as the ‘gold standard’ : synergy not separatism in evaluation designs. / Hanley, Pam; Chambers, Bette; Haslam, Jonathan.

In: International Journal of Research and Method in Education, Vol. 39, No. 3, 2016, p. 287-298.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Reassessing RCTs as the ‘gold standard’

T2 - International Journal of Research and Method in Education

AU - Hanley, Pam

AU - Chambers, Bette

AU - Haslam, Jonathan

PY - 2016

Y1 - 2016

N2 - Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.

AB - Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.

KW - evaluation

KW - evidence-based education

KW - mixed methods

KW - RCT

KW - STEM

U2 - 10.1080/1743727X.2016.1138457

DO - 10.1080/1743727X.2016.1138457

M3 - Article

VL - 39

SP - 287

EP - 298

JO - International Journal of Research and Method in Education

JF - International Journal of Research and Method in Education

SN - 1743-727X

IS - 3

ER -