Category: experimental design

Reviewing initial teacher education

Some responses to the "Initial teacher training market review"

A 'market' review

Image by Pexels from Pixabay

The UK Government's Department for Education (responsible for the school system in England) is currently undertaking what it called a 'market review' of initial teacher education (ITE) or initial teacher 'training' as it prefers to describe ite. (Arguably, 'education' suggests broad professional preparation for someone who will need to make informed decisions in complex situations, whereas 'training' implies learning the skills needed for a craft.)

The aims of the review are certainly unobjectionable:

The review has aimed to make well informed, evidence-based recommendations on how to make sure:

• all trainees receive high-quality training
• the ITT market maintains the capacity to deliver enough trainees and is accessible to candidates
• the ITT system benefits all schools1

https://www.gov.uk/government/publications/initial-teacher-training-itt-market-review/initial-teacher-training-itt-market-review-overview

Despite such intentions clearly being laudable, the actual proposals (which, inter alia, can be seen as looking to further increase central government control over the professional preparation of teachers) raised concerns among many of those actually involved in teacher education.

The consultation

There was a public consultation to which all interested were invited to respond. Since the consultation closed, the Secretary of State (i.e., senior minister) for Education has changed, so we await to see whether this will derail the review.

The review is wide ranging, but there is a widespread view that once again government is seeking to reduce the influence of academic education experts (see for example, 'Valuing the gold standard in teacher education'), and my colleagues still working in university-school based ITE partnerships certainly felt that if all the proposals were brought to fruition such partnership would be at risk. Not that Universities would not be able to contribute, but they would not be able to do so in a way that allowed full quality control and proper planning and sustainable commitment.

My own University, Cambridge, has suggested

We cannot, in all conscience, envisage our continuing involvement with ITT should the proposals be implemented in their current format.

Government ITT market review consultation, Faculty of Education website

Some discussion on one teachers' email list I subscribe to, provoked me me decide to look back at my own consultation responses.

A selective response – and a generic default hole-filler

I have not worked in I.T.E. for some years, and so did not feel qualified to comment on all aspects of the review. However, there were some aspects of the plans (or at least my interpretation of what  was intended) that I felt would put at risk some of the strongest and most important aspects of high quality teacher preparation.

As being able to submit a response to the consultation required providing a response at every section (a cynic might suggest that expecting full completion of such a long consultation document is a disincentive for most people to contribute), I used a generic statement to cover those questions where I  did not feel I had anything informed and useful to say:

I am aware of concerns raised in responses by the Russell group of Universities, the University of Cambridge (of which I am an emeritus officer), and Homerton College, Cambridge (of which I am a senior member). I concur with these concerns, and rather than seek to reproduce or mirror all of their comments (already available to you), I refer you to those responses. Further, I am offering some specific comments on particular issues where I have strong concerns based on my past experiences as a PGCE student teacher; as a teacher in comprehensive secondary schools; as a school/college-based mentor supporting graduates preparing for teaching in schools and also in a further education context; as a researcher exploring aspects of student learning and the teaching that supports it; as a lecturer and supervisor on initial teacher education courses as part of University-School training partnerships; as a supervisor for teachers in post undertaking school-based research; as an advisor to schools undertaking context-directed research; and as a lecturer teaching graduates how to undertake research into educational contexts.

Here are my more specific responses highlighting my particular concerns:

Individual differences

Having worked in initial teacher education as well as having been a school teacher, I am well aware that one of the most important things anyone working in the education sector has to appreciate is individual differences – between pupils, between teachers, between classes, between schools, and between new-entrants. Too much focus on uniformity is therefore unwelcome and likely to reduce the quality of the highest provision which takes this into diversity into account, Similarly, genuinely 'rigorous' sequencing of the educational experience will be responsive to individual needs and that would be welcome. However, uniform and inflexible sequencing, which would be far from rigorous, would be damaging.

Being equipped to engage with research

I am aware that the diversity in routes for new entrants now available has reduced the quality of training experience available to some new teachers. In particular, the fully professional teacher has to be a critical reader of research, and to have the tools and confidence to undertake their own small-scale context based enquiry to develop their own practice.

Table 1 from Taber, 2010

This is essential because the research shows clearly that whilst it is sometimes possible to identify some features of best practice that generalise across most teaching contexts, this is by no means always the case. Teaching and learning are highly complex phenomena and are strongly influenced by contextual factors. So, what has been found to 'normally' work best will not be the best approach in all teaching contexts. Teachers needs to be able to read research claims critically

(there are always provisos

  • most studies are small-scale where strict generalisation is simply not possible,
  • few studies are sufficiently supported with the resources to test ideas across a wide range of contexts; and
  • experimental studies which are the gold standard in the natural sciences are usually problematic in education
    • as randomisation {a critical aspect of true experimental research} is seldom possible, and
    • there is seldom the information or means to characterise populations sufficiently to build representative samples;
    • moreover the complexity of educational contexts does not allow the identification (let alone control) of all relevant variable, and
    • there are some key known factors which influence results when double-blind methods are not viable
      • – a situation that is very common when testing innovations in educational practices as teachers and learners are usually well aware of deviations from normal practice)

and identify the most promising recommendations when taking into account their own teaching context (i.e., what is referred to as reader or naturalistic generalisation) and test out ideas in their own classrooms, and iteratively develop their own practice.

Sadly, whilst the M-level PGCE type programmes usually support new teachers in introducing these skills, this does not seem to necessarily be the case on some other routes.

On 'intensive' practice placements

I consider this is a misguided notion based on a flawed conceptualisation of teaching and teacher skills. It is certainly the case that generally speaking teachers develop their skills over time with greater teaching experience, and that all other things being equal, the more direct teaching experience a new entrant has during the period of initial teacher education the better, as long as this is productive experience.

However, teaching is a highly complex activity that requires making myriad in the moment decisions in response to interactions with unique classes of unique people. The quality of those decisions tends to increase over time with experience, but only if the teacher is well prepared for the teaching in terms of subject knowledge, general and specialist pedagogic knowledge, and knowledge of the particular learners.

This requires that the teacher has extensive preparation time especially when new to teaching a topic, age, group or pedagogic approach, and opportunities for productive debrief and reflection. Given the intensity of teaching as an experience, it is much better for new entrants to initially focus on parts of lessons with plenty of opportunity for preparation and reflection than to too quickly progress to whole lessons where much of the experience will not be fully processed before moving on. Similarly, it is better that new teachers have sufficient time between classes to focus intensely on those classes rather than be moving directly from class to class.

In the same way, the programmes that allow regular movements between the teaching context and an HEI or similar context offer an ideal context for effective learning. The intense focus on the school is broken up by time in faculty (still focused, but as a student without the intense scrutiny in school), where there are extensive opportunities for peer support (especially important given the extreme highs and lows often experienced by new teachers).

Partnerships of Universities with Schools offer new entrants complementary expertise, and opportunities for 'iteration' – moving between the 'graduate student' and 'teaching department member' contexts 2 (Figure 1 from Taber, 2017)

This is also critical for developing teaching that is informed by research-informed and evidence-based theories and constructs. Being taught 'theory' in an academic context, and expecting such content to be automatically applied in a teaching context is unrealistic – rather the new teacher has to learn to conceptualise actual classroom experience in terms of the theory, and to see how to apply the theory in terms of actual teaching experience. 2

This learning is best supported by an iterative process – where there are plenty of opportunities to reflect on and analyse experience, and compare and discuss experiences with peers, as well as with mentors, other experienced teachers, and with academic staff. Over time, as new teachers build experiences, especially ones they recognise as productive and successful, they will come to automatically apply ideas and skills and techniques, and will be able to 'chunk' component teaching moves into longer sequences – being able to work effectively for sequences of whole classes, with less reflection time, and less explicit support. 3

The aim is for the new teachers to be able to prepare, teach, assess, on something approaching a teaching timetable whilst working in school full-time. However, efforts to move to such a state too quickly will [be counter-productive] for many potentially excellent teachers, and will likely increase drop-out rates.

Ultimately, the quality of the teaching experience, and the ability to manage increasing workload according to individual needs, is what is important. Any attempts to increase the intensity of the teaching placements, or to accelerate the rate at which new teachers take on responsibility without recourse to individual circumstances is likely to be counterproductive in terms of retention, and the quality of the 'training' experience in supporting the development of excellent teachers.

I am very pleased that I would not be 'training' nor still working in teacher education under such expectations as I think the incidents of crises, mental health issues, and drop-out, would be likely to increase.

On common timetables for progress

As suggested above, any attempt to tightly quantify these things would be misplaced as it removes the ability of providers to manage the process to seek the best outcomes for individual trainees, and it ignores the responsibilities of teachers and schools to ensure that trainees are only given responsibilities as and when they are ready.

Please remember that every class taught by a trainee contains children or young people who are required to be in school and are entitled to be taught by someone who

  • is prepared for class,
  • confident they are ready to teach that class, and
  • is not under such intense stress that they cannot perform to their potential.

You have a responsibility to consider the pupils as well as to your 'market'.

On applying research evidence

A postgraduate award is meant to include a strong research component. As suggested in earlier comments, it is essential for the fully professional teacher who will need to make informed decisions about her own classroom practice to be be provided with the skills to access research (including understanding strengths and weaknesses of methodology), critique it, evaluate its potential relevance to the immediate teaching and learning contexts, and to evaluate it in the context. Many PGCE-MEd and PGCE-MA programmes already support this.

I totally agree that this should be provided to all new trainees, and would have thought there are enough HEIs with expertise in educational research for this to be possible (as it is on the PGCE-M route already). However, it is not enough to simply provide teachers the skills, they also have to have

  • access to research publications,
  • time to
    • read them and
    • undertake small-scale context-directed enquiry, and
    • to give them the confidence that this aspect of professional practice is recognised and appreciated.

For example, a teacher has to know that if they are doing something differently to some government advice because they have looked at the research, considered it in relation to their specific context, and evaluated approaches in their own teaching context and concluded that for a particular class/course/students some approach other than that generally recommended is indicated, THEN this would be recognised (e.g., in Inspections) as praiseworthy.

On 'incentives that could encourage schools and trusts to participate in ITT'

I would think it is dangerous and perhaps foolish to add to schools' expected responsibilities where they do not welcome this.

On proposed reforms on the recruitment and selection process

To me, this seems to complicate matters for a PGCE applicant who at the moment has to only select a university-schools partnership.

Potential equality impacts

As discussed above, in my experience current arrangements, at least for the PGCE route, offer flexibility to meet the individual needs of a range of new entrants. My sense is the proposals would be unhelpful in this regard.

Comments on 'any aspect'

I was lucky enough to undertake my PGCE at a university that at the time was recognised as one with excellent provision in my teaching subjects (chemistry and physics, at Nottingham Trent). At that time the structure of the teaching placement (two isolated blocks, one of 4 weeks, one of 8 weeks) did not allow the kind of incredibly valuable iterative experience of moving between the university and school contexts I discuss above, and the teachers in the schools did not act as mentors, but merely handed over their classes for a period of time.

Otherwise I was very happy with my 'training' experience.

I was also privileged to work for about 10 years in initial teacher education in a PGCE university-schools partnership that has consistently been awarded the very top inspection grades across categories. I have therefore seen much excellent initial teacher education practice in a stable partnership with many committed (if diverse) schools. We were also able to be pretty selective in recruitment, so were working with incredibly keen and committed new teachers.

If (some) university-schools partnerships (such as that based at the University of Cambridge) are recognised as excellent, why change the system in ways that threaten those providers?

Despite this, I know some of our excellent new recruits went through serious periods of doubt and crises in their teaching due to the intense and highly skilled nature of the work. In the context where I was lucky enough to work, the structure of the training year and the responsive and interactive nature of managing the graduates in their work meant that nearly always these setbacks were temporary, and so could be overcome.

I am concerned that some of this good practice may not continue if some of the proposals in the review are carried through – and that consequently a significant number of potentially excellent new teachers will not get the support they need to develop at the pace that best matches their needs. This will lead to drop-out, and early burn-out – or potentially candidates doing enough to cope, without meeting the high standards they wish to set for themselves to the benefit of their pupils.

Keith S. Taber

1 It strikes me that the third bullet point might seem a little superfluous – after all, surely a system of initial teacher education that both maintains the supply of new teachers at the level needed (which in some subjects would be a definite improvement on the existing system) and ensures they all receive high quality preparation should inherently benefit all schools by making sure there was always a pool of suitably qualified and well-prepared teachers to fill teaching vacancies across the school curriculum.

Perhaps, however, this means something else – such as (in view of the reference to 'incentives that could encourage schools and trusts to participate in ITT' in the consultation) making sure all schools receive funding for contributing to the preparation of new teachers (by making sure all schools make a substantial contribution to the preparation of new teachers).

2 It strikes me that the way in which teachers in preparation are able to move back and forth between a study context and a practitioner context, giving opportunities to apply learning in practice, and to 'stand back' and reflect on and conceptualise that practice, reflects the way science proceeds – where theory motivates new practical investigations, and experience of undertaking the empirical enquiry informs new theoretical refinements and insights (which then…).

3 That is, the pedagogic principles which teachers are expected to apply when working with their students are, in general terms, just as relevant in their own professional education.

Work cited:

Not a great experiment…

What was wrong with The Loneliness Experiment?

Keith S. Taber

The loneliness experiment, a.k.a. The BBC Loneliness Experiment was a study publicised through the BBC (British public service broadcaster), and in particular through it's radio programme All in the Mind, ("which covers psychology, neuroscience & mental health" according to presenter, Claudia Hammond's website.)1 It was launched back in February 2018 – pre-COVD.2

"All in the Mind: The Loneliness Experiment launches the world's largest ever survey of its kind on loneliness." https://www.bbc.co.uk/programmes/b09r6fvn

Claudia Hammond describes herself as an "award-winning broadcaster, author and psychology lecturer". In particular "She is Visiting Professor of the Public Understanding of Psychology at the University of Sussex" where according to the University of Sussex  "the post has been specially created for Claudia, who studied applied psychology at the University in the 1990s", so she is very well qualified for her presenting role. (I think she is very good at this role: she has a good voice for the radio and manages to balance the dual role of being expert enough to exude authority, whilst knowing how to ask necessarily naive questions of guests on behalf of non-specialist listeners.)

A serious research project

The study was a funded project based on a collaboration between academics from a number of universities, led by Prof Pamela Qualter, Professor of Education at the Manchester Institute of Education at the University of Manchester. Moreoever, "55,000 people from around the world chose to take part in the BBC Loneliness Experiment, making it the world's largest ever study on loneliness" (https://claudiahammond.com/bbc-loneliness-experiment/)

Loneliness is a serious matter that affects many people, and is not be made light of. So this was a serious study, on an important topic – yet every time I heard this mentioned on the radio (and it was publicised a good deal at the time) I felt myself mentally (and sometimes physically) cringe. Even without hearing precise details of the research design, I could tell this was simply not a good experiment.

This was not due to any great insight on my behalf, but was obvious from the way the work was being described. Readers may wish to see if they can spot for themselves what so irked me.

What is the problem with this research design?

This is how the BBC described the study at its launch:

The Loneliness Experiment, devised by Professor Pamela Qualter and colleagues, aims to look at causes and possible solutions to loneliness. And we want as many people as possible to fill in our survey, even if they've never felt lonely, because we want to know what stops people feeling lonely, so that more of us can feel connected.

https://www.bbc.co.uk/programmes/b09r6fvn

This is how Prof. Hammond described the research in retrospect:

55,000 people from around the world chose to take part in the BBC Loneliness Experiment, making it the world's largest ever study on loneliness. Researchers from the universities of Manchester, Brunel and Exeter, led by Professor Pamela Qualter and funded by the Wellcome Trust, developed a questionnaire asking people what they thought loneliness was, when they felt lonely and for how long.

https://claudiahammond.com/bbc-loneliness-experiment/

And this is how the work is described on the University of Manchester's pages:

The Loneliness Experiment was a study conducted by BBC Radio 4's All in the Mind….

The study asked respondents to give their opinions and record their experiences of loneliness and related topics, including friendship, relationships, and the use of technology – as well as recording lifestyle and background information. Respondents also engaged in a number of experiments.

The survey was developed by Professor Pamela Qualter, from The University of Manchester's Manchester Institute of Education (MIE), with colleagues from Brunel University London, and the University of Exeter. The work was funded by a grant from The Wellcome Trust.

https://www.seed.manchester.ac.uk/education/research/impact/bbc-loneliness-experiment/

When is an experiment not an experiment?

These descriptions make it obvious that the The Loneliness Experiment was not an experiment. Experiment is a specific kind of research – a methodology where the researchers randomly assign participants randomly to conditions, intervene in the experimental condition,and take measurements to see what effect the intervention has by comparing with measurements in a control condition. True experiments are extremely difficult to do in the social sciences (Taber, 2019), and often quasi-experiments or natural experiments are used which do not meet all the expectations for true experiments. BUT, to be an experiment there has to be something that can be measured as changing over time in relation to specified different conditions.

Experiment involves intervention (Image by Gerd Altmann from Pixabay)

Experiment is not the only methodology used in research – there are also case studies, there is action research and grounded theory, for example – and non-experimental research may be entirely appropriate in certain situations, and can be of very high quality. One alternative methodology is the survey which collects data form a sample of a population at some particular time. Although surveys can be carried out in various ways (for example, through a series of observations), especially common in social science is the survey (a methodology) carried out by using participant self-responses to a questionnaire (a research instrument).

it is clear from the descriptions given by the BBC, Professor Hammond and the University of Manchester that the The Loneliness Experiment was not actually an experiment at all, but basically a survey (even if, tantalisingly, the Manchester website suggests that "Respondents also [sic] engaged in a number of experiments". )

The answer to the question 'when is an experiment not an experiment?' might simply be: when it is something other than an experiment

Completing a questionnaire (Image by Andreas Breitling from Pixabay)

What's in a name: does it really matter?

Okay, so I am being pedantic again.

But I do think this matters.

I think it is safe to assume that Prof. Hammond, Prof. Qualter and colleagues know the difference between an experiment and a survey. Presumably someone decided that labelling the research as the loneliness study or the loneliness survey would not be accessible (or perhaps not as impressive) to a general audience and so decided to incorrectly use the label experiment as if experiment was synonymous with study/research.

As a former research methods lecturer, that clearly irks as part of my job was to teach new researchers about key research concepts. But I would hope that people actually doing research or learning to do research are not going to be confused by this mislabelling.

But, as a former school science teacher, I know that there is widespread public misunderstanding of key nature of science terms such as theory and experiment. School age students do need to learn what is meant by the word experiment, and what counts as an experiment, and the BBC is being unhelpful in presenting research that is not experimental as an experiment – as this will simply reinforce common misconceptions of what the term experiment is actually used to denote in research .

So, in summary, I'll score The BBC Loneliness Experiment

  • motivation – excellent;
  • reach – impressive;
  • presentation – unfortunate and misleading
Further reading:

Read about methodology

Read about experiments

Read about surveys

Work cited:

Taber, K. S. (2019). Experimental research into teaching innovations: responding to methodological and ethical challenges. Studies in Science Education, 55(1), 69-119. doi:10.1080/03057267.2019.1658058 [Download manuscript version]

Note:

1: Websites cited accessed on 28th August, 2021.

2: It would have been interesting to repeat when so many people around the world were in 'lock-down'. (A comparison between pre-COVID and pandemic conditions might have offered something of a natural experiment.)

Those flipping, confounding variables!

Keith S. Taber

Alternative interpretations and a study on flipped learning

Image by Please Don't sell My Artwork AS IS from Pixabay

Flipping learning

I was reading about a study of 'flipped learning'. Put very simply, the assumption behind flipped learning is that usually teaching follows a pattern of (a) class time spent with the teacher lecturing, followed by (b) students working through examples largely in their own time. This is a pattern that was (and perhaps still is) often found in Universities in subjects that largely teach though lecture courses.

The flipped learning approach switches the use the class time to 'active' learning activities, such as working through exercises, by having students undertake some study before class. That is, students learn about what would have been presented in the lecture by reading texts, watching videos, interacting with on-line learning resources, and so forth, BEFORE coming to class. The logic is that the teacher's input is more useful  when students are being challenged to apply the new ideas than as a means of presenting information.

That is clearly a quick gloss, and clearly much more could be said about the rationale, the assumptions behind the approach,and its implementation.

(Read more about flipped learning)

However, in simple terms, the mode of instruction for two stages of the learning process

  • being informed of scientific ideas (through a lecture)
  • applying those ideas (in unsupported private study)

are 'flipped' to

  • being informed of scientific ideas (through accessing learning resources)
  • applying those ideas (in a context where help and feedback is provided)

Testing pedagogy

So much for the intention, but does it work? That is where research comes in. If we want to test a hypothesis, such as 'students will learn more if learning is flipped' (or 'students will enjoy their studies more if learning is flipped', or 'more students will opt to study the subject further if learning is flipped', or whatever) then it would seem an experiment is called for.

In principle, experiments allow us to see if changing some factor (say, the sequence of activities in a course module) will change some variable (say, student scores on a test). The experiment is often the go-to methodology in natural sciences: modify one variable, and measure any change in another hypothesised to be affected by it, whilst keeping everything else that could conceivably have an influence constant. Even in science, however, it is seldom that simple, and experiments can never actually 'prove' our hypothesis is correct (or false).

(Read more about the scientific method)

In education, running experiments is even more challenging (Taber, 2019). Learners, classes, teachers, courses, schools, universities are not 'natural kinds'. That is, the kind of comparability you can expect between two copper sulphate crystals of a given mass, or two specimens of copper wire of given dimensions, does not apply: it can matter a lot whether you are testing this student or that student, or if the class is taught one teacher or another.

People respond to conditions different to inanimate objects – if testing the the conductivity of a sample of a salt solution of a given concentration it should not matter if it is Monday morning of Thursday afternoon, or whether it is windy outside, or which team lost last's night's match, or even whether the researcher is respectful or rude to the sample. Clearly when testing the motivation or learning of students, such things could influence measurements. Moreover, a sample of gas neither knows or cares what you are expecting to happen when you compress it, but people can be influenced by the expectations of researchers (so called expectancy effect – also known as the Pygmalion effect).

(Read about experimental research into teaching innovations)

Flipping the fundamentals of analytic chemistry

In the study, by Ponikwer and Patel, researchers flipped part of a module on the fundamentals of analytical chemistry, which was part of a BSc honours degree in biomedical science. The module was divided into three parts:

  1. absorbance and emission spectrosocopy
  2. chromatography and electrophoresis
  3. mass spectroscopy and nuclear magnetic resonance spectroscopy

Students were taught the first topics by the usual lectures, then the topics of chromatography and electrophoresis were taught 'flipped', before the final topics were taught through the usual lectures. This pattern was repeated over three successive years.

[Figure 1 in the paper offers a useful graphical representation of the study design. If I had been prepared to pay SpringerNature a fee, I would have been allowed to reproduce it here.*]

The authors of the study considered the innovation a success

This study suggests that flipped learning can be an effective model for teaching analytical chemistry in single topics and potentially entire modules. This approach provides the means for students to take active responsibility in their learning, which they can do at their own pace, and to conduct problem-solving activities within the classroom environment, which underpins the discipline of analytical chemistry. (Ponikwer & Patel,  2018: p.2268)

Confounding variables

Confounding variables are other factors which might vary between conditions and have an effect.

Read about confounding variables

Ponikwer and Patel were aware that one needs to be careful in interpreting the data collected in such a study. For example, it is not especially helpful to consider how well students did on the examination questions at the end of term to see if students did as well, or better, on the flipped topics that the other topics taught. Clearly students might find some topics, or indeed some questions, more difficult than others regardless of how they studied. Ponikwer and Patel reported that on average students did significantly better on questions from the flipped elements, but included important caveats

"This improved performance could be due to the flipped learning approach enhancing student learning, but may also be due to other factors, such as students finding the topic of chromatography more interesting or easier than spectroscopy, or that the format of flipped learning made students feel more positive about the subject area compared with those subject areas that were delivered traditionally." (Ponikwer & Patel,  2018: p.2267)

Whilst acknowledging such alternative explanations for their findings might seem to undermine their results it is good science to be explicit about such caveats. Looking for (and reporting) alternative explanations is a key part of the scientific attitude.

This good scientific practice is also clear where the authors discuss how attendance patterns varied over the course. The authors report that the attendance at the start of the flipped segment was similar to what had come before, but then attendance increased slightly during the flipped learning section of the course. They point out this shift was "not significant", that is statistics suggested it could not be ruled out to be a chance effect.

However Ponikwer and Patel do report a statistically "significant reduction in the attendance at the non-flipped lectures delivered after the flipped sessions" (p.2265) – that is, once students had experienced the flipped learning, on average they tended to attend normal lectures less later in their course. The authors suggest this could be a positive reaction to how they experienced the flipped learning, but again they point out that there were confounding variables, and other interpretations could not ruled out:

"This change in attendance may be due to increased engagement in the flipped learning module; however, it could also reflect a perception that a more exciting approach of lecturing or content is to be delivered. The enhanced level of engagement may also be because students could feel left behind in the problem-solving workshop sessions. The reduction in attendance after the flipped lecture may be due to students deciding to focus on assessments, feeling that they may have met the threshold attendance requirement" (Ponikwer & Patel,  2018: p.2265).

So, with these students, taking this particular course, in this particular university, having this sequence of topics based on some traditional and some flipped learning, there is some evidence of flipped learning better engaging students and leading to improved learning – but subject to a wide range of caveats which allow various alternative explanations of the findings.

(Read about caveats to research conclusions)

Pointless experiments?

Given the difficulties of interpreting experiments in education, one may wonder if there is any point in experiments in teaching and learning. On the other hand, for the lecturing staff on the course, it would seem strange to get these results, and dismiss them (it has not been proved that flipped learning has positive effects, but the results are at least suggestive and we can only base our action on the available evidence).

Moreover, Ponikwer and Patel collected other data, such as students' perceptions of the advantages and challenges of the flipped learning approach – data that can complement their statistical tests, and also inform potential modifications of the implementation of flipped learning for future iterations of the course.

(Read about the use of multiple research techniques in studies)

Is generalisation possible?

What does this tell us about the use of flipped learning elsewhere? Studies taking place in a single unique teaching and learning context do not automatically tell us what would have been the case elsewhere – with different lecturing staff, different demographic of students, when learning about marine ecology or general relativity. Such studies are best seen as context-directed, as being most relevant to here they are carried out.

However, again, even if research cannot be formally generalised, that does not mean that it cannot be informative to those working elsewhere who may apply a form of 'reader generalisation' to decide either:

a) that teaching and learning context seems very similar to ours: it might be worth trying that here;

or

b) that is a very different teaching and learning context to ours: it may not be worth the effort and disruption to try that out here based on the findings in such a different context.

(Read about generalisation)

This requires studies to give details of the teaching and learning context where they were carried out (so called 'thick description'). Clearly the more similar a study context is to one's own teaching context, and the wider the range of teaching and learning contexts where a particular pedagogy or teaching approach has been shown to have positive outcomes, the more reason there is to feel it is with trying something out in own's own classroom.

I have argued that:

"What are [common in the educational research literature] are individual small-scale experiments that cannot be considered to offer highly generalisable results. Despite this, where these individual studies are seen as being akin to case studies (and reported in sufficient detail) they can collectively build up a useful account of the range of application of tested innovations. That is, some inherent limitations of small-scale experimental studies can be mitigated across series of studies, but this is most effective when individual studies offer thick description of teaching contexts and when contexts for 'replication' studies are selected to best complement previous studies." (Taber, 2019: 106)

In that regard, studies like that of Ponikwer and Patel can be considered not as 'proof' of the effectiveness of flipped learning, but as part of a cumulative evidence base for the value of trying out the approach in various teaching situations.

Why I have not included the orignal figure showing the study design

* I had hoped to include in this post a copy of the figure in the paper showing the study design. The paper is not published open access and so the copyright in the 'design' (that, is the design of the figure **, not the study!) means that it cannot be legally reprodiced without permission. I sought permission to reproduce the figure here through (SpringerNature) the publisher's on line permissions request system, explaining this was to be used in an acdemics scholar's personal blog.

Springer granted permission for reuse, but subject to a fee of £53.83.

As copyright holder/managers they are perfectly entitled to do that. However, I had assumed that they would offer free use for a non-commercial purpose that offers free publicity to their publication. I have other uses for my pension, so I refer readers interested in seeing the figure to the original paper.

** Under the conventions associated with copyright law the reproduction of short extracts of an academic paper for the purposes of criticism and review is normally considered 'fair use' and exempt from copyright restrictions. However, any figure (or table) is treated as a discrete artistic design and cannot be copied from a work in copyright without permission.

(Read about copyright and scholarly works)

 

Work cited:

Nothing random about a proper scientific evaluation?

Keith S. Taber

Image by annca from Pixabay 

I heard about an experiment comparing home-based working with office-based working on the radio today (BBC Radio 4 – Positive Thinking: Curing Our Productivity Problem, https://www.bbc.co.uk/sounds/play/m000kgsb). This was a randomised controlled trial (RCT). An RCT is, it was explained, "a proper scientific evaluation". The RCT is indeed considered to be the rigorous way of testing an idea in the social sciences (see Experimental research into teaching innovations: responding to methodological and ethical challenges).

Randomisation in RCTs

As the name suggests, a key element of a RCT is randomisation. This can occur at two levels. Firstly, research often involves selecting a sample from a larger population, and ideally one selects the sample at random from the population (so every member of the wider population has exactly the same chance of being selected for the sample), so that it can be assumed that what is found with the sample is likely to reflect what would have occurred had the entire population been participating in the experiment. This can be very difficult to organise.

More critically though, it is most important that the people in the sample each have an equal chance of being assigned to each of the conditions. So, in the simplest case there will be two conditions (e.g., here working at home most workdays vs. working in the office each workday) and each person will be assigned in such a way that they have just as much chance as being in one condition as anyone else. We do not put the cleverest, more industrious, the tallest, the fastest, the funniest, etcetera, in one group – rather, we randomise.

If we are randomising, there should be no systematic difference between the people in each condition. That is, we should not be able to use any kind of algorithm to predict who will be in each condition because assignments are made randomly – in effect, according to 'chance'. So, if we examine the composition of the two groups, there is unlikely to be any systematic pattern that distinguishes the two groups.

Two groups – with elements not selected at random (Image by hjrivas from Pixabay)

Now some scientists might suspect that nothing happens by chance – that if we could know the precise position and momentum of every particle in the universe (contra Heisenberg) … perhaps even that probabilistic effects found in quantum mechanics follow patterns due to hidden variables we have not yet uncovered…

How can we randomise?

Even if that is not so, it is clear that many ways we use to randomise may be deterministic at some level (when we throw a die, how it lands depends upon physical factors that could in principle, even if not easily in practice, be controlled) but that does not matter if that level is far enough from human comprehension or manipulation. We seek, at least, a quasi-randomisation (we throw dice; we mix up numbered balls in a bag, and then remove them one at a time 'blind'; we flip a coin for each name as we go down a list, until we have a full group for one condition; we consult a table of 'random' numbers; whatever), that is in effect random in the sense that the researchers could never know in advance who would end up in each condition.

When I was a journal editor it became clear to me that claims of randomisation reported in submitted research reports are often actually false, even if inadvertently so (see: Non-random thoughts about research). A common 'give away' here is when you ask the authors of a report how they carried out the randomisation. If they are completely at odds to answer, beyond repeating 'we chose randomly', then it is quite likely not truly random.

To randomise, one needs to adopt a technique: if one has not adopted a randomisation technique, then one used a non-random method of assignment. Asking the more confident, more willing, more experienced, less conservative, etc., teacher to teach the innovation condition is not random. For that matter, asking the first teacher one meets in the staffroom is arbitrary and not really random, even if it may feel as if it is.

…they were randomised, by even and odd birthdates…

The study I was hearing about on the radio was the work of Stanford Professor Nick Bloom, who explained how the 'randomisation' occurred:

"…for those volunteers, they were randomised, by even and odd birth dates, so anyone with an even birth date, if you were born on like the 2nd, the 4th, the 6th, the 8th, etcetera,of the month, you get to work at home for four out of five days a week, for the next nine months, and if you are odd like, I'm the 5th May, you had to stay in the office for the next nine months…"

Professor Nick Bloom interviewed on Positive Thinking: Curing Our Productivity Problem
Image by Jeevan Singla from Pixabay 

So, by my definition, that is not randomisation at all – it is totally deterministic. I would necessarily have been in the working at home condition, with zero possibility of being in the office working condition. If this had been random there would have been a 50:50 chance of Prof. Bloom and myself being assigned to the same condition – but with the non-random, systematic assignment used it was certain that we would have ended up in different conditions. So, this was a RCT without randomisation, but rather a completely systematic assignment to conditions.

This raises some questions.

  • Is it likely that a professor of economics does not understand randomisation?
  • Does it really matter?

Interestingly, I see from Prof. Bloom's website that one "area of [his] research is on the causes and consequences of uncertainty", so I suspect he actually understands randomisation very well. Presumably, Prof. Bloom knows that strictly there was no randomisation in this experiment, but is confident that it does not matter here.

Had Prof. Bloom assigned the volunteers to conditions depending on whether they were born before or after midnight on the 31st December 1989, this clearly would have introduced a major confounding variable. Had he assigned the volunteers according to those born in March to August to one condition and those born in September to February to the other, say, this might have been considered to undermine the research as it is quite conceivable that the time of year people were gestated, and born, and had to survive the first months of life, might well be a factor that makes a difference to work effectiveness later.

Even if we had no strong evidence to believe this would be so, any systematic difference where we might conjecture some mechanism that could have an effect has to be considered a potential confound that undermines confidence in the results of a RCT. Any difference found could be due to something other (e.g., greater thriving of Summer babies) than the intended difference in conditions ; any failure to find an effect might mean that a real effect (e.g., home working being more efficient than office working) is being masked by the confounding variable (e.g., season of birth).

It does not seem conceivable that even and odd birth dates could have any effect (and this assignment is much easier to organise than actually going through the process of randomisation when dealing with a large number of study participants). So, in practice, it probably does not matter here. It seems very unlikely this could undermine Prof. Bloom's conclusions. Yet, in principle, we randomise in part because we are not sure which variables will, or will not, be relevant, and so we seek to avoid any systematic basis for assigning participants to conditions. And given the liberties I've seen some other researchers take when they think they are making random choices, my instinct is to want to see an RCT where there is actual randomisation.