Experimental research into teaching innovations

One of Keith's publications is a research review/perspective article:

Taber, K. S. (2019). Experimental research into teaching innovations: responding to methodological and ethical challenges. Studies in Science Education, 55(1), 69-119. doi:10.1080/03057267.2019.1658058

Abstract:

Experimental studies are often employed to test the effectiveness of teaching innovations such as new pedagogy, curriculum, or learning resources. This article offers guidance on good practice in developing research designs, and in drawing conclusions from published reports. Random control trials potentially support the use of statistical inference, but face a number of potential threats to validity. Research in educational contexts often employs quasi-experiments or natural experiments rather than true experiments, and these types of designs raise additional questions about the equivalence between experimental and control groups and the potential influence of confounding variables. Where it is impractical for experimental studies to employ samples that fully reflect diverse populations, generalisation is limited. Series of small-scale replication studies may be useful here, especially if these are conceptualised as being akin to multiple case studies, and complemented by qualitative studies. Control conditions for experimental studies need to be carefully selected to provide the most appropriate test for a particular intervention, and considering the interests of all participants. Control groups in studies that replicate innovations that have been widely shown to be effective in other settings should experience teaching conditions that reflect good practice and meet expected teaching standards in the research context.

Download the article

Keywords:

Content:

  • Introduction
  • The use of random control trials in education
  • Potential threats to the validity of findings from RCT
    • Participants' expectations can influence outcomes
    • Participants can respond to perceived novelty
    • Fair testing should involve teachers in different treatment groups having comparable levels of experience of their assigned teaching 'treatment'
    • Participants may make gains during a study due to maturation
    • Participants may learn from pre-tests
    • Deciding when learning is best measured
    • Measurement instruments may be considered to be biased towards one treatment
    • Other potential confounds
  • Quasi-experiments and natural experiments employed when randomisation is not plausible
  • Testing for equivalence between groups
    • Equivalence is more than a lack of significance difference
    • Using statistics to respond to non-equivalence
  • Choosing comparison conditions
    • Does the experimental treatment have any educational effect?
    • Does the intervention represent an improvement on current practice?
    • How does an innovation compare with currently recognised good practice?
    • Guidance on selecting control conditions: logical considerations
    • Guidance on selecting control conditions: ethical considerations
  • Generalising from experimental studies
  • Replication studies
  • Planning ethical comparison conditions in replication studies
    • Rhetorical experiments
    • Avoiding detrimental control conditions
    • Good practice in selecting productive control treatments
  • Conclusions

From the introduction:

  • "The present article offers a thematic review of some key issues and challenges that arise in the design and interpretation of experimental studies in education, drawing upon selected illustratory examples of published studies.
  • It is intended that this review will be useful both as guidance for those looking to undertake experimental studies of teaching innovations, and also for those seeking to be informed by reading research reports of such studies.
  • The article considers the particular practical challenges of carrying-out experimental studies in education. This analysis highlights some inherent limitations in many small-scale experimental studies which cannot be assumed to generalise to other contexts.
  • The article considers notions of generalisability and replication to both argue for how such studies can best be understood to contribute to our understanding of teaching and learning and to suggest how individual studies can be best designed to usefully add to the literature.
  • Particular attention is given to the selection of the most informative 'control' conditions with which experimental treatments may be compared. The article suggests guidelines for best practice in establishing control conditions for studies that will be both ethical and informative." (p.70-72)

The author's manuscript version is available for download.

The version of record can be found at: https://www.tandfonline.com/doi/abs/10.1080/03057267.2019.1658058 (If you would like the published version and do not have access to the journal site, you can contact me.)

facebookShare on Facebook
TwitterTweet