Shortlisting for disease

False positives on screening tests can be understood in relation to job applications

Keith S. Taber

I rather liked an analogy used by Dr Kit Yates of Bath University comparing medical screening to being shortlisted for a job. The context was a Royal Institution podcast entitled: Can We Trust Maths? ¹

This was a very informative discussion of aspects of statistics, and one of the questions addressed was:

How often do false positive and false negative test results occur in medical screenings?

Screening for disease

Screening programmes test apparently healthy members of the population for serious medical issues in order to catch problems at an early stage when treatment offers the best prognosis.

No tests are perfect, so tests will sometimes give misleading results – called false positives and false negatives.

a test result that is:	when an ideal perfect test would have shown positive	when an ideal perfect test would have shown negative
positive	is called a true positive	is called a false positive
negative	is called a false negative	is called a true negative

Sometimes tests can be tuned to avoid many false negatives by tolerating a higher rate of false positive (or vice versa). This is similar to what happens in statistical hypothesis testing when the choice of 'confidence level' (the p {for probability} value used as a cut-off criterion for 'statistical significance') can be chosen according to whether it is more important to avoid false positives or to avoid false negatives.

The notion of 'beyond reasonable doubt' used in criminal trials can be understood as based on the principle that it is better that some guilty perpetrators are not convicted at trial than to risk miscarriages of justice where innocent people may lose their liberty (or indeed in some jurisdictions, perhaps their lives). That is, it is better to have false negatives than false positives in criminal convictions.

In medical screening programmes, it is common to have an initial test which might give quite a few positive results (but hopefully not produce many false negatives, where a person with a disease appears to be clear according to the test), even though most of the positive results will prove to be false alarms (false positives) when followed up by a more sophisticated test that it is impractical or too expensive to use for mass screening.

The bias towards false positives built into some medical screening trials means that a person should not be too despondent at getting a positive result in the initial screen. Dr Yates worked through one example to show that based on the rates of false positives on certain screening tests, a person called for regular screenings over a number of years was actually more likely than not to get at least one positive screening result – but still unlikely to be unlucky enough to have the disease.

A teaching analogy

What I most liked was the use of an analogy to compare the logic of the screening process with a familiar everyday situation. Teaching can be seen as a process of making the unfamiliar familiar, and teachers often do this by comparing the unfamiliar they are charged with teaching about with something already familiar to the their students. That is only a starting point for supporting a developing understanding of the new concept or phenomenon, but it often is very useful in making abstract new ideas seem less threatening or inaccessible.

Read about making the unfamiliar familiar

One common way of making the unfamiliar familiar is through analogy: showing that what is new has a familiar conceptual structure – mapping onto a set of ideas already understood.

Read about teaching analogies

An 'outreaching' analogy?

Scientists charged with giving talks to a public audience as part of 'public communication' of science ('outreach') or attempts to improve 'public understanding' of science also have the job of making the unfamiliar familiar and may also use teaching analogies – as Dr Yates did here:

"I would make the analogy to screenings with a job interview. So, when a company wants to hire someone for a job, they send out an advert, and people send in their c.v.s. And the company can read those c.v.s quickly and make a shortlist. And that's a really cheap way, just as the first screen is a really cheap way of identifying people, who might be suitable for the job, people who might have breast cancer. And then for the job interview you call people in and you interview them and you throw 'assessment centres' at them, you do tests which are too expensive to do to the whole population at large to identify someone good for the job, but you can do it to this smaller population. And in the same way, with the screen we invite people in and we throw more expensive, more accurate tests at them to give them a diagnosis. And the point is, just because you would get invited to an interview for a job you had applied for, you wouldn't assume that you had got the job, right? So, in the same way, just because you get invited for further tests after a screen, you shouldn't assume you have the disease that is being screened for. You should wait and go to the follow-up test and see what that follow-up test says."

Dr Kit Yates explaining the logic of screening programmes

This seemed a well-considered analogue, one that would be very accessible to most people in the audience. It is a common experience to have applied for jobs: perhaps sometimes not being shortlisted; sometimes called in for interview but not appointed; and sometimes being offered the job. ²

The explanations flowed nicely between the target concept (screening) and the analogue (shortlisting) – as can be seen in the tabulated version below.

"I would make the analogy to screenings	with a job interview.
	So, when a company wants to hire someone for a job, they send out an advert, and people send in their c.v.s.
	And the company can read those c.v.s quickly and make a shortlist.
	And that's a really cheap way,
just as the first screen is a really cheap way of identifying people,
	who might be suitable for the job,
people who might have breast cancer.
	And then for the job interview you call people in and you interview them and you throw 'assessment centres' at them, you do tests which are too expensive to do to the whole population at large to identify someone good for the job, but you can do it to this smaller population.
And in the same way, with the screen we invite people in and we throw more expensive, more accurate tests at them to give them a diagnosis.
	And the point is, just because you would get invited to an interview for a job you had applied for, you wouldn't assume that you had got the job, right?
So, in the same way, just because you get invited for further tests after a screen, you shouldn't assume you have the disease that is being screened for.
You should wait and go to the follow-up test and see what that follow-up test says."

An effective teaching analogy needs to have an analogue that is sufficiently familiar for an audience to appreciate its conceptual structure – and that structure must fit well when mapped across to the target concept. 'Medical screening is like job shortlisting' seems to work well on both these criteria.

Work cited:

• Taber, K. S. (2019). Experimental research into teaching innovations: responding to methodological and ethical challenges. Studies in Science Education, 55(1), 69-119. doi:10.1080/03057267.2019.1658058

Footnotes:

¹: "If you see a newspaper headline with a big, bold statistic, how do you know that you can trust it? How often do false positive and false negative test results occur in medical screenings? And how do you safely bet whether or not 2 people in any room will share a birthday?
This month we hear from Kit Yates about the maths of medicine, crime and the media, exploring real-world data from his book, 'The Maths of Life and Death'.
This talk was recorded from our theatre at the Royal Institution, on 21 January 2020." https://soundcloud.com/royal-institution/maths-trust

2. It might be suggested that this process reflects a middle class /professional/white collar employment experiences, whereas for many jobs, such as much shop or factory work, an employer is likely to employ the first apparently suitable candidate that applies, rather than using a slower and more expensive two stage process. This is so, but the situation of short-listing is still generally familiar through story lines in fiction, such as in television dramas.