A topic in research methodology
Read about experiment as research methodology
"Experiments are set up to test specific hypotheses. In a 'true' experiment the researcher controls variables, so that only the factor which is hypothesised to have an effect differs between the experimental and control treatments. In reality, such control is rarely (if ever) possible in enquiries into teaching and learning – even if the range of potentially significant variables can be identified" (Taber, 2013, p.83).
Researchers working in educational contexts will not always have a free choice over the comparison conditions, but ideally the control is set up in view of the research questions to best meet the needs of the study.
Three levels of comparison condition
"Table 2 sets out a simple typology of three levels depending upon the nature of the educational input provided for the learners in a control or comparison group. The activity undertaken with a group of learners that could potentially be educative is here referred to as a 'treatment'. In experimental work, the experimental/intervention group is subject to a treatment that differs in some well- characterised way from the treatment of the control or comparison group. The three levels suggested set different tests for the experimental treatment. These are, in effect,
- Does it have any educational value? (level 1);
- Is it better than standard educational provision? (level 2);
- How does it compare to what is already recognised as good practice? (level 3)" (Taber, 2019, p.88).
Treatment versus no treatment: Does the experimental treatment have any educational effect?
"The first level of experimental design suggested in Table 2 simply looks to see if outcomes on some educational measure are better after some treatment than in a matched group of learners who did not experience any treatment. This level of design is potentially useful when the research question concerns whether there is any value in introducing some new educational provision or resource that would be additional to current provision. That is, this type of study is not concerned with doing something differently, but rather whether there is sufficient value in committing additional resources to do something extra, that is not currently done, to consider recommending it should be added to existing educational provision" (Taber, 2019, p.89).
Innovation versus standard treatment: Does the intervention represent an improvement on current practice?
"The second type of experimental design represented in Table 2 concerns the testing of an innovation which is conjectured to offer an improved form of educational provision in relation to some specific educational aim(s). In this situation, the innovation is compared with what is considered the 'standard' or 'normal' form of provision" (Taber, 2019, p.91).
Innovation versus enhanced treatment: How does an innovation compare with currently recognised good practice?
"…the third type of experimental design in Table 2 that sets a higher standard for an innovation to be measured against.
- Where at the first level researchers seek to find out if some educational treatment has some effect in comparison to no treatment at all; and
- at the second level researchers look to see if an innovative approach has a more positive effect than standard provision;
- at the third level a comparison is made with educational provision considered to reflect good practice.
In effect, researchers are asking if an innovation is as good as, or even better than, something that is already considered to be effective. In studies with level 3 control conditions, a failure to find a significant difference between outcomes in the two conditions may be seen as reflecting positively on the experimental treatment.
A fourth alternative: How does an innovation compare with currently recognised poor practice?
Sadly, some studies in the research literature seem to choose the research question "is the experimental treatment better than an educational input which is known to fall short of good practice?" This does not seem a very useful question from a research perspective (it seems to be a kind of 'rhetorical research' -seeking to demonstrate the obvious) and may lead to unethical control conditions.
Read about unethical control conditions
Making a choice
When to, and not to, use level 1 controls
"The choice between (a) level 1 control conditions where a teaching innovation is compared with a treatment without teaching (or where standard teaching that is supplemented by an additional teaching input is compared with only the standard provision) and (b), level 2 and 3 control conditions that offer an equivalent level of teaching input intended to meet the same educational objectives as the innovatory treatment, will derive from the motivation for the study. In many teaching contexts, there will be an existing provision which, even if not considered effective, will be assumed to bring about learning objectives to some extent.
In these situations, a level 1 control condition is of limited use as such a study will simply show that the tested teaching treatment produces some level of learning – something that is to be expected (as even mediocre teaching is likely to facilitate some level of learning), and, without a meaningful comparison with existing practice, offers little guidance for teachers" (Taber, 2019, p.93).
Choosing between levels 2 and 3
"The choice between levels 2 (the comparison treatment being standard provision) and 3 (the comparison treatment being recognised good practice) may depend upon what the innovation is hoped to provide. If existing provision is considered to draw upon too high a resource level, or is found to have some undesirable side effects, then seeking an alternative that is just as effective may be well motivated. So, a hypothetical school level biology course using animal dissection might lead to satisfactory levels of learning of anatomy, but lead to a minority of students declining to take part. In such a situation an experiment to test an alternative to dissection may only be seeking to find an approach that produces learning outcomes that are as good as in the comparison condition. In this situation, current standard practice provides an effective comparison condition and there is a sensible rationale for a 'level 2ʹ control (see Table 2).
Many published studies argue that the innovation being tested has the potential to be more effective than current standard teaching practice, and seek to demonstrate this by comparing an innovative treatment with existing practice that is not seen as especially effective. This seems logical where the likely effectiveness of the innovation being tested is genuinely uncertain, and the 'standard' provision is the only available comparison. However, often these studies are carried out in contexts where the advantages of a range of innovative approaches have already been well demonstrated, in which case it would be more informative to test the innovation that is the focus of the study against some other approach already shown to be effective" (Taber, 2019, p.93).
Sources cited:
- Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.
- Taber, K. S. (2019). Experimental research into teaching innovations: responding to methodological and ethical challenges. Studies in Science Education. doi:10.1080/03057267.2019.1658058 [Download the article]
My introduction to educational research:
Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.
