Drawing conclusions

Caveats and provisos

A topic in research methodology

Ideally, a claim made in a research paper is supported by a logical chain of argument, supported by the interpretation of evidence. For the logic to work, the research design has to have a high level of coherence (so the data collected, and the analysis made, can logically inform the researcher about the research question).

Read about principles of coherent research design

However, research normally involves some compromises over what would be ideal in a study. For example:

  • A whole population can seldom be studied, so must instead be sampled – and it may not be possible to obtain a sample that is known to be fully representative of the wider population of interest*
  • It may not be possible to run a true experiment if that might, for example, requires students and teachers in established classes to randomly reassigned to experimental and control conditions (I have seen studies where schools allow this, BUT this is unusual), so a 'natural experiment' has to be employed**
  • It may be necessary to use self reports when direct observations may be preferable but not possible (consider research to find out how long people spend in the bath/shower), and this relies upon good self-knowledge as well as honest informants! (Teacher: 'Yes, I always give full feedback on all my students work within two days of submission')

And so on!

Researchers are often only able to do what is feasible given resources and what their participants will be happy to consent to (Teacher: 'well, you can observe MOST of my Y9 classes, but, erm, not set 9/5 as, er, they are easily distracted by classroom visitors').

Good research writing

The 'feasibility gap' between an ideal research design and what is possible in practice often means that drawing conclusion form research studies is not straightforward. In good research writing the authors make an argument for what they think can be concluded form their study (putting aside limitations that they consider are not serious enough to undermine the logical chain), but ALSO discuss those limitations. For example, where it is not possible to control all variables that could be having effect, they discuss alternative interpretations of how their findings may be explained in terms of those other factors.

In good research writing, then, authors presents conclusions with caveats and provisos, so that readers can judge for themselves the extent to which study limitations undermine the ability to draw firm conclusions.

Critical reading of research

Although good research writing raises such issues for readers, not all studies will be explicit about such matters, perhaps because

  • the authors feel here is no reasonable expectations that any limitations actually impinge on their conclusions;
  • the authors thought it weakened their paper in peer review to point out the weaknesses of the study (even though best practice requires they do)
  • the authors had not noticed the limitations.

So, critical readers of research need to carefully interrogate research papers to see if there is a coherent design and a logical chain of argument to support the conclusions reached.

Read about critical reading of research

  • Things to consider when critically reading research reports
  • The reader needs to look for limitations that might undermine confidence in research findings

Some examples:

Might a self-selecting sample distort findings?

Consider this examples that I have discussed elsewhere (Taber, 2013: 91):

"In a study published in the British Educational Research Journal, Brooks and Everett (2009) reported on graduates' perceptions of the value of having obtained a degree. Clearly, the potential population of graduates is immense, and in selecting their 90 participants, Brooks and Everett recruited graduates from six institutions that they considered to occupy different 'market positions' within higher education:
• an Oxbridge college
• a college of the University of London
• a redbrick university
• a 1960s campus university
• a post-92 university
• and a college of higher education"

The authors were aware there was possible issue with the way they have built a sample, as only people who had volunteered for the research responded, and volunteers my not be representative of a wider population:

Brooks and Everett, instead, draw the reader's attention to another severe limitation of their study with potential to bring their results into question:

"It is important to remember that our sample was self-selecting: respondents offered to be interviewed after receiving a letter about the project from their alumni office, or seeing an advert on the 'Friends Reunited' website. It is, therefore, likely that the sample over-represents those who have had more successful learning careers and/or transitions from higher education into work–as it is probably much easier to talk about perceived successes than perceived failures."

(Brooks & Everett, 2009, p. 347)

"interviewing a self-selected sample was likely to have biased their sample…[so] the self-selecting nature of the sample is a major methodological flaw in the study, and so the quote above offers a major caveat to the study findings. Yet the paper is published in a prestigious journal, after being subject to peer review. Often research involves compromises." (Taber, 2013: 91, emphasis added).

Although this is a weakness of the study, this reflects a general issue about the nature of the society we live in. If there was an accessible central register of all graduates detailing where they studied, when they graduated, and their contact details, then in principle researchers could make a sample reflecting the population (in this case, by inviting a random selection of graduates from each of their six classes of institution). However, the ready availability of information useful to researchers has to be balanced against both the costs of maintaining that kind of database, and, more importantly, people's rights to privacy and having information about them kept confidential: an important ethical issue.

Read about research ethics

Confounding variables

Ponikwer and Patel report a study of flipped learning, where an undergraduate course in analytical chemistry was segmented into three parts, with topics in each part taught by normal lectures, flipped learning, normal lectures. The authors reported some data suggesting that student attendance and examination scores were better for the topics taught in the flipped learning segment rather than the parts of the course taught by lecturing. However Ponikwer and Patel were aware that it was not possible to hold all other potentially relevant variables constant so findings could be influenced by so-called 'confounding' variables.

For example, students were expected to meet a minimum attendance requirement on classes (that is lecturer or the exercises classes held instead during the flipped learning) across the course. So, near the end of the course, students who knew they had already attended enough sessions not to be failed on that criterion would have less incentive to attend class regularly. Moreover, at the end of the course, students were revising for examinations, which could lead to the choice of missing a class to spend more time to revise.

Ponikwer and Patel were not suggesting student attendance dropped at the end of the course due to these reasons, but rather that they provide feasible alternative explanations to students finding the lectures less useful for their learning after experiencing the flipped learning.

Read about this study: Alternative interpretations and a study on flipped learning

* read about sampling

** read about natural experiments

Sources cited:

My introduction to educational research:

Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.

facebookShare on Facebook
TwitterTweet