Incremental generalisation

Page contents

'Building up a profile of the pedagogic effectiveness of some innovation'

Generalising from a single study

Generalisation is the process of applying findings in a research study beyond the immediate context of that study. Common approaches are statistical generalisation (from a sample to a population), analytic or theoretical generealisation (from a case to other cases argued to be similar in the most pertinent ways) and reader generalisation (from a case described in sufficient detail to another context that is judged sufficient similar).

Read about generalisation in research

Generalising through programmatic research

The term 'incremental generalisation' was intended to be used in relation to a programme of research rather than a single study. The assumption behind this concept is that social contexts (for example, teaching and learning contexts) vary across a wide range of variables, and this makes it very difficult to reach general conclusions.

Poetic chemistry?

As an example consider the hypothesis that teaching chemistry in verse leads to greater learning gains than teaching chemistry in prose. (Of course, this is a hypothetical example, which would only be tested in practice if there was a strong conceptual framework for the study providing a reasonable rationale for suspecting this might be the case!)

Imagine someone tested this experimentally with 14-16 year olds students in a 11-18 secondary school in Cambridge, England, and found that the hypothesis was supported as the mean of the learning gains of those taught chemistry in verse was significantly greater than the mean learning gains for those taught chemistry in prose. Moreover, the difference found in outcomes between the results in the two conditions was not only statistically significant (so unlikely to be due to chance factors) but had a large effect size, suggesting that the difference was not just measurable, but likely to make a substantive difference to learners. ¹

We might wonder if the same results would apply to

a diffrent cohort of students (for example, the following year), or
with different teachers, or
in another secondary school the other side of the city, or,
indeed, one of the ('private' or 'independent') Cambridge schools which select their students from among those who can afford high school fees and are not accessible to most young people, or
with 16-18 year old students studying at a higher level, or
students in school in a socially deprived Lincolnshire coastal town, or
in high schools in the United States, or
for students in Paris being taught chimie en français, or
in classes in a refugee camp in Chad, or…

Clearly the further we get away from the orignal context, the less likely we are to assume we can generalise form a single study.

Learning from series of studies in different contexts

But what if there were a good many studies, in different contexts, and most, but not all, found an advantage to teaching chemistry in rhyme?

The argument here then is that large-scale RCT that use representative samples from populations of interest are necessarily rare in education. What are more common are individual small-scale experiments that cannot be considered to offer highly generalisable results. Despite this, where these individual studies are seen as being akin to case studies (and reported in sufficient detail) they can collectively build up a useful account of the range of application of tested innovations. That is, some inherent limitations of small-scale experimental studies can be mitigated across series of studies, but this is most effective when individual studies offer thick description of teaching contexts and when contexts for 'replication' studies are selected to best complement previous studies.
Taber, 2019: 106

Under these circumstances, we can work towards 'incremental generalisation'.