Keith S. Taber
I heard about an experiment comparing home-based working with office-based working on the radio today (BBC Radio 4 – Positive Thinking: Curing Our Productivity Problem, https://www.bbc.co.uk/sounds/play/m000kgsb). This was a randomised controlled trial (RCT). An RCT is, it was explained, "a proper scientific evaluation". The RCT is indeed considered to be the rigorous way of testing an idea in the social sciences (see Experimental research into teaching innovations: responding to methodological and ethical challenges).
Randomisation in RCTs
As the name suggests, a key element of a RCT is randomisation. This can occur at two levels. Firstly, research often involves selecting a sample from a larger population, and ideally one selects the sample at random from the population (so every member of the wider population has exactly the same chance of being selected for the sample), so that it can be assumed that what is found with the sample is likely to reflect what would have occurred had the entire population been participating in the experiment. This can be very difficult to organise.
More critically though, it is most important that the people in the sample each have an equal chance of being assigned to each of the conditions. So, in the simplest case there will be two conditions (e.g., here working at home most workdays vs. working in the office each workday) and each person will be assigned in such a way that they have just as much chance as being in one condition as anyone else. We do not put the cleverest, more industrious, the tallest, the fastest, the funniest, etcetera, in one group – rather, we randomise.
If we are randomising, there should be no systematic difference between the people in each condition. That is, we should not be able to use any kind of algorithm to predict who will be in each condition because assignments are made randomly – in effect, according to 'chance'. So, if we examine the composition of the two groups, there is unlikely to be any systematic pattern that distinguishes the two groups.
Now some scientists might suspect that nothing happens by chance – that if we could know the precise position and momentum of every particle in the universe (contra Heisenberg) … perhaps even that probabilistic effects found in quantum mechanics follow patterns due to hidden variables we have not yet uncovered…
How can we randomise?
Even if that is not so, it is clear that many ways we use to randomise may be deterministic at some level (when we throw a die, how it lands depends upon physical factors that could in principle, even if not easily in practice, be controlled) but that does not matter if that level is far enough from human comprehension or manipulation. We seek, at least, a quasi-randomisation (we throw dice; we mix up numbered balls in a bag, and then remove them one at a time 'blind'; we flip a coin for each name as we go down a list, until we have a full group for one condition; we consult a table of 'random' numbers; whatever), that is in effect random in the sense that the researchers could never know in advance who would end up in each condition.
When I was a journal editor it became clear to me that claims of randomisation reported in submitted research reports are often actually false, even if inadvertently so (see: Non-random thoughts about research). A common 'give away' here is when you ask the authors of a report how they carried out the randomisation. If they are completely at odds to answer, beyond repeating 'we chose randomly', then it is quite likely not truly random.
To randomise, one needs to adopt a technique: if one has not adopted a randomisation technique, then one used a non-random method of assignment. Asking the more confident, more willing, more experienced, less conservative, etc., teacher to teach the innovation condition is not random. For that matter, asking the first teacher one meets in the staffroom is arbitrary and not really random, even if it may feel as if it is.
…they were randomised, by even and odd birthdates…
The study I was hearing about on the radio was the work of Stanford Professor Nick Bloom, who explained how the 'randomisation' occurred:
"…for those volunteers, they were randomised, by even and odd birth dates, so anyone with an even birth date, if you were born on like the 2nd, the 4th, the 6th, the 8th, etcetera,of the month, you get to work at home for four out of five days a week, for the next nine months, and if you are odd like, I'm the 5th May, you had to stay in the office for the next nine months…"
Professor Nick Bloom interviewed on Positive Thinking: Curing Our Productivity Problem
So, by my definition, that is not randomisation at all – it is totally deterministic. I would necessarily have been in the working at home condition, with zero possibility of being in the office working condition. If this had been random there would have been a 50:50 chance of Prof. Bloom and myself being assigned to the same condition – but with the non-random, systematic assignment used it was certain that we would have ended up in different conditions. So, this was a RCT without randomisation, but rather a completely systematic assignment to conditions.
This raises some questions.
- Is it likely that a professor of economics does not understand randomisation?
- Does it really matter?
Interestingly, I see from Prof. Bloom's website that one "area of [his] research is on the causes and consequences of uncertainty", so I suspect he actually understands randomisation very well. Presumably, Prof. Bloom knows that strictly there was no randomisation in this experiment, but is confident that it does not matter here.
Had Prof. Bloom assigned the volunteers to conditions depending on whether they were born before or after midnight on the 31st December 1989, this clearly would have introduced a major confounding variable. Had he assigned the volunteers according to those born in March to August to one condition and those born in September to February to the other, say, this might have been considered to undermine the research as it is quite conceivable that the time of year people were gestated, and born, and had to survive the first months of life, might well be a factor that makes a difference to work effectiveness later.
Even if we had no strong evidence to believe this would be so, any systematic difference where we might conjecture some mechanism that could have an effect has to be considered a potential confound that undermines confidence in the results of a RCT. Any difference found could be due to something other (e.g., greater thriving of Summer babies) than the intended difference in conditions ; any failure to find an effect might mean that a real effect (e.g., home working being more efficient than office working) is being masked by the confounding variable (e.g., season of birth).
It does not seem conceivable that even and odd birth dates could have any effect (and this assignment is much easier to organise than actually going through the process of randomisation when dealing with a large number of study participants). So, in practice, it probably does not matter here. It seems very unlikely this could undermine Prof. Bloom's conclusions. Yet, in principle, we randomise in part because we are not sure which variables will, or will not, be relevant, and so we seek to avoid any systematic basis for assigning participants to conditions. And given the liberties I've seen some other researchers take when they think they are making random choices, my instinct is to want to see an RCT where there is actual randomisation.
