Instructions for using the Critical Reading of Empirical Studies Tool
A topic in research methodology
These instructions are intended to support use of the Critical Reading of Empirical Studies Tool which is designed to support students and others needing to interrogate published research studies.
Read about the importance of critical reading of research
Read about, and download, the tool
Using the research confidence evaluation tool
Purpose
The tool is intended to be used when critically reading empirical research – studies that involve collecting and analysing research data.
Background
The tool is founded on an assumption that published research studies
- make new knowledge claims (they claim something new has been found out); and
- provide arguments for those knowledge claims – to persuade readers that the claims are justified.
That is, although a research report may seem to be a descriptive narrative of what was done, and why, and what was found out; the rationale of the research paper is that it draws some conclusions which, it is argued, follow from the enquiry reported in the paper.
Read about 'The research paper as an argument'
This process can be summarised in a (convoluted) sentence: the conclusions depend upon the interpretation of the outcomes of the analysis of data that has been collected in accordance with a research design that was planned to answer certain research questions which were seen as motivated by a conceptualisation of the literature reviewed.
This can be usefully broken down. It can be seen as a logical chain, such as:
If
you accept the authors' conceptualisation of the current state of knowledge in the field;
and if
you agree that the study's research question(s) is/are suggested by that understanding;
and if
you agree that the methodology chosen is appropriate for a study designed to answer the research question(s);
and if
you agree that the design of the research (i.e., the sampling of participants or data sources; the data collection instruments; the means of analysing the data, etc.) allows the research question(s) to be answered;
and if
you believe that the design was implemented as planned;
and if
you believe that the analysis is sound;
and if
you believe the researchers have interpreted their findings appropriately;
then
you have good reasons to accept their conclusions.
The tool is intended to act as a visual device to help keep some of these features in mind when reading a study, and to help the scholar 'keep tabs' on their judgments about aspects of the paper as they read it. In particular, it asks the user to offer impressionist elevations of how convincing some key features of the research report are.
This is meant purely as a visual aid, and it is not intended that any of the ratings should be seen as objective measures: BUT if you give a study low ratings, then this suggests you do not have strong grounds for simply accepting the authors' conclusions.
What is the tool?
The original version of the tool comprised a series of 11-point scales aligned with some key aspects of research reports,
where the user is asked to evaluate their confidence in that aspect of the work reported.
That is, the scale is 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0. This can be considered as a series of 10% increments (0%, 10%, 20%…)
So 0.0 (0%) confidence (as above) would mean you have no confidence in that aspect of the study, and 1.0 (100%, as below) would mean that you have absolute confidence in that aspect of the study.
As a result of feedback during a small-scale valuation of the tool, an alternative version of the tool uses a six-point verbal scale in place of the numeric scale.
How to use the tool
The tool is provided to support effective study/reading of research. The instructions are meant as guidance – please feel free to adapt and modify according to what best supports your style, level of expertise and needs.
As you read through a study, note which sections of the report relate to the key features highlighted on the tool. For each section, consider how convincing you found the research paper, and indicate this by circling one of the points on the scale. Do NOT worry about being precise, as this is meant to be purely impressionistic.
You are not required to explain your reasons (but you might wish to make related notes for your own purposes in case you later write about the study in an assignment/dissertation). Alternative versions of the tool are available with space for making more specific notes.
If you do not feel that one of the features listed on the tool is relevant to the study you are reading, or you feel you have no basis for evaluating the strength of that feature, just leave that specific scale blank.
Features to evaluate:
1. Confidence in the overall research design: to what degree do the authors convince you that the study they have designed is suitable for answering their research questions? You might wish to consider whether the researchers are using suitable methodology to address the research questions, as well as whether the different aspects of the design (data collection methods, instruments, analytical techniques) work together and fit the overall methodology.
2. Confidence in the sampling (in behavioural/social science subjects). A research report should make it clear to what population the study is intended to relate to (e.g., first year undergraduates at Oxbridge colleges; students starting secondary school in Chiapas, Mexico). In many studies, it is not possible to collect data from an entire population of interest (e.g., modern language teachers working in the U.K.; 4 year olds attending pre-school in Western Australia; children attending hagwons after the school day finishes in Seoul) and therefore a sample of the population is selected to 'represent' the full population.
To what extent are you confident that the population of interest has been sufficiently well-sampled?
Read about 'Sampling a population'
3. Confidence in the data collections tools (tests, interview schedules, questionnaire items, observation schedules, laboratory instruments, etc.) To what degree are you confident that the data collection instruments do their job: are they are valid (measure what they are intended to measure), reliable (give reproducible results), accurate/precise?
4. Confidence in the analysis: To what degree are you confident that the data collected have been appropriately analysed to provide results?
5. Confidence in the interpretation of results: To what degree are you confident that the researchers' conclusions (e.g., answers to research questions) actually follow from their findings (results)?
When you have finished reading the paper, reflect on how convincing you find the paper overall:
Confidence in the study overall: To what degree are you confident that the new 'knowledge claims' made by the authors of the paper are justified in terms of the quality of the study overall?
When the tool was evaluated there was a sense that participants might tend to over-estimate their overall confidence in the conclusions of a study, given their ratings of the discrete components of the study. You might wish to consider this before reaching your overall rating.
Read about how users of the tool rated features of research studies
All research is subject to limitations and caveats/provisos, so you should not expect to be totally confident in all aspects of the studies you read.
Read about 'Caveats and provisos in research reports'
Reports may give inadequate information for you to feel you can evaluate some aspects of the work. You should also bear in mind that sometimes reports (inadvertently, or – even occasionally – deliberately) contain inaccurate details of studies carried out.
In reporting on your evaluations (in assignments, dissertations/theses or publications) you will need to explain your reasons for your evaluations – so you may want to make notes on the features that lead you to rate a study as you do.
Alternative versions of the tool
Feedback received during a small-scale evaluation of the tool suggested that some users might prefer
- a non-numerical scale for making evaluations
- space next to the scales to make specific notes about the reasons for making evaluations
I have therefore prepared some modified versions of the tool:
