Generalisation

A topic in research methodology

The problem of generalisation

Generalisation refers to the extent to which a finding from a specific context or case or sample can be applied more widely.

Generalisation can sometimes be applied unproblematically in the physical sciences (this molecule of benzene has six carbon atoms – so all molecules of benzene must have six carbon atoms), but is much more problematic in social sciences (and even in the life sciences),

"…the [American Psychological Association] Task Force [on evidence-based practice] acknowledged that findings based on groups may not apply to all individuals and that individual characteristics may be relevant to the outcomes of interventions. In particular, they conceded the difficulty of applying evidence across populations."

Lovasz & Clegg, 2019

In fields such as education, it is common for research papers to refer to small samples of individuals from a poorly (or not) defined population as though they can be assumed to reflect the wider population.

"Consider some titles of published research papers…
• Secondary school teachers' pedagogic practices when teaching mixed and structured ability classes…
• Constructivist-compatible beliefs and practices among US teachers
• A study of schoolchildren's alternative frameworks of the concept of force"…

The titles … suggest that the paper is about a group of people: …'secondary school teachers'… 'US teachers' or even … 'schoolchildren'…These are very large groups, and – of course – none of these papers present data from such large groups."

Taber, 2013: 80

So, the question raised is whether a study that, say, interviewed a dozen schoolchildren in the London area (the 'sample') can be reasonably assumed to produce findings that apply to 'schoolchildren' (as a 'population') in general. The title of papers may suggest such broad populations, but often within the text there are caveats that because of the nature of the sample results can only be assumed to apply to a more limited population.

Read about populations of interest in research

This still raises the question of the extent to which generalisation is reasonable or justified:

Can a sample of London school children be assumed to reflect European schoolchildren? British schoolchildren? Even English schoolchildren?

The answer to that question might depend on what precisely the focus of the study was – if the topic was attitudes to school science one would not assume this would be the same in different countries with different science curricula for example.

Any claim of generalisation has to be be based on some kind of rationale. There are different types of argument that can be made, for different kinds of generalistion

Statistical generalisation

Research studies that sample larger populations look for statistical generalisability. This means that the sampling techniques used are designed to ensure that the results obtained when analysing data collected from the sample allow inferences to be made about the wider population.

Taber, 2013: 189

When all members of a population are accessible for sampling (but the population is too large to all be included in the study) the simplest approach would be to select a random sample. Experimental research relies on random sampling. Statistical methods can then be used to find the 'statistical significance' of the results obtained form the sample.

Read about sampling

Read about experimental research

Theoretical or analytical generalisation

Another type of generalisation uses theoretical arguments about why the original research focus might be able to stand for a wider population. This will be "based upon the conceptual framework supporting the research design" Taber, 2013: 195 – that is, the study will be based on assumptions about the nature of the phenomena inquired into that led to the sample selected as able to represent a wider population.

If that sounds a little vague, it is actually what happens in natural science all the time.

For example, if it is found that a pure sample of a salt dissolved in pure water to give a basic solution, then we would expect the same thing to be true of other pure samples of the same salt. We have a conceptual framework for what it means to be a chemical substance, such that we know (can safely assume) that certain properties are invariant between samples: melting temperature, solubility, conductivity, and so on.

If we found two samples of the 'same' salt did have different properties, then our theoretical framework would suggest they are not really the same kind and we should look for a difference (levels of impurity, different crystal form…) In the natural sciences this kind of logic is so taken-for-granted that it is often not make explicit.

Generally, the subjects of scientific studies are samples/specimens of natural kinds where it can either be assumed (i) that swapping the particular specimen would not change the results, or (ii) that there is some relevant variation between specimens such that we should work with a sample and draw conclusions statistically, so that drawing a new sample from the same population should give substantially the same results. If we find a pure copper rod is a good conductor, then this applies to all pure copper rods and not just the one(s) we decided to test.

Taber, 2020: 16

Even so, this statement in a popular science book about comets seems a little bold:

"After the virgin comet Kohouteck was scrutinised by radio and visible, ultraviolet and infrared light, one could sum up the detectable constituents of comets in general in one word: noisome."

Calder, 1980 [emphasis added]

In the social sciences, however, explicit argument need to be made about why (for example) student alternative conceptions elicited in one context are likely to also be relevant on other contexts. So, for example, if an alternative conception is believed to derive from early experience of acting on/in the physical environment (from experience of pushing, dropping, lifting, pulling, etc. objects in general) then it can be argued it should be found in all human children; but if it seems to be based on transferring meaning from a synonymous word then it would not be assumed to arise in groups of children speaking other languages.

Reader generalisation

In reader generalisation the researchers do not claim that their result will apply in another context such as a potential reader's teaching context, but offer enough information about the context of the study for a reader to make judgements of similarity:

The researcher needs to provide enough context ('thick description') – of the students, of the physical environment, of the teacher-class rapport, of the subject matter, etc. – for a reader to make a judgement about how likely it is that insights drawn from that particular case may be useful in understanding the [context] that we are interested in (as teachers, as researchers, as appraisers of teachers, etc.). This is called reader generalisability, because the reader has to make the judgement. The author cannot tell us, as they do not know about our context.

Taber, 2013: 190

Read about reader generalisation from individual cases

Incremental generalisation

The term 'incremental generalisation' has been used to describe the process of series of research studies that collectively test out the range of applicability of findings across wider contexts.

Experimental research into teaching innovations: responding to methodological and ethical challenges

If researchers carefully consider the results of previous trials of an innovation in relation to the specific contexts of those studies when planning their own research, then the community of researchers can collectively build up a body of research which incrementally explores the range of effectiveness of different innovations. For this to occur, it is important that reports of teaching experiments are sufficiently detailed, not just in terms of technical matters, but also in terms of the specific teaching and learning contexts where the work takes place.

Taber, 2019: 105

Read about the idea of incremental generalisation

Works cited:

My introduction to educational research:

Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.

facebookShare on Facebook
TwitterTweet

Leave a Reply

Your email address will not be published. Required fields are marked *