What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process.

Qualitative data analysis examples: These short extracts from published research papers are provided to support class discussion.

Suto, W. M. I., & Greatorex, J. (2007). What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal, 34(2), 213-233.

Abstract

The process of examination marking is complex, requiring examiners to engage in a variety of cognitive operations. While consideration has been given to marking practices in a few specific contexts, those of General Certificate of Secondary Education (GCSE) examiners have yet to receive serious attention. This study’s aims, therefore, were: first, to investigate the cognitive strategies used when marking GCSEs; and second, to interpret them within the context of psychological theories of human judgement. Two contrasting GCSE examination papers were considered: Mathematics used a ‘points-based’ marking scheme, while Business Studies relied on a ‘levels-based’ scheme. Small groups of experienced examiners marked script samples, and using a concurrent ‘think aloud’ method, verbal protocols were obtained. Using a semi-structured interview schedule, the examiners were then questioned retrospectively. Qualitative data analysis supported a model of five distinct cognitive marking strategies, which can be interpreted within dual-processing theories of judgement. The implications for GCSE marking are discussed.

"Each examiner was asked to ‘think aloud’ whilst marking the fourth and final sample of scripts. Digital audio equipment was used to record the ‘verbal protocols’ that resulted from this process." p.218

"Using a pre-prepared semi-structured interview schedule (see Appendix 1), each examiner was questioned retrospectively about his or her marking strategies. For both mathematics and business studies examiners, the interviews centred on three question parts which had been pre-selected from the appropriate examination paper by the researchers. In each case, these question parts were chosen for their diversity of style, length, and expected difficulty for the candidate. Digital audio equipment was used to record the interviews." p.218

"The audio files from Stages 4 and 5 were transcribed, and a qualitative analysis of the data was conducted by the researchers jointly. It comprised two distinct stages: verbal protocol analysis, and analysis of the post-verbal protocol interviews." p.219

"Initially, to obtain an overview of these data, the verbal protocol transcripts were given a broad reading. This led to the general inference that when an examiner first glances at a given question or question part within a script, the first thing he or she must do, consciously or otherwise, is to recollect which one it is." p.219

"The first reading of the verbal protocol transcripts led to the formulation of a tentative framework of five main cognitive marking strategies used by the examiners, which we labelled: matching, scanning, evaluating, scrutinising and no response. In a second and more detailed reading, this framework of strategies was applied consistently to the data. In order to do this, each examiner’s protocol was broken down or ‘segmented’ (Green, 1998) by script, and also by the question parts as defined by the appropriate mark scheme. Each examiner’s marking of each candidate’s response to each question part was coded according to the cognitive strategies used." p.220

"The transcripts of the post-verbal protocol interviews were analysed with the objective of establishing whether this data supported the model of marking strategies p.//p. formulated in the verbal protocol analysis. This appeared to be the case. Although the no response strategy was not discussed, both mathematics and business studies examiners described all of the other four marking strategies. The following interview extracts illustrate this finding:…" pp.226-227

(Brief extracts such as these can only give you a flavour of a study. You can use the citations to access the full papers to explore the extracts here in the contexts of the full studies.)

Index of topics