A topic in research methodology
Research can change reality
Research reports make claims about new knowledge, and often make recommendations about such matters as
- directions for future research (what further research is needed, which approaches are likely to be more fruitful)
- modifications to professional practice (e.g., such as changing teaching behaviours in classrooms)
- policy revision – recommendations to changes in policy direction, implementation, or regulation
Research is therefore important as it can help us improve society (for example, via the education system) by developing a strong knowledge base and providing information about current strengths and weaknesses, indicated changes, and where valuable resources should be focused on further enquiry.
Yet research can be of variable quality.
Limitations to research
Even high quality research will be subject to some provisos due to (often unavoidable) limitations in study designs. Resources limit sample size, or the geographic range of a study; ethical considerations (rightly) limit what can be undertaken; inherent issues (such as how asking an interview question can promote a line of thought that may not have occurred to a respondent otherwise) introduce bias or subjectivity that may be unavoidable.
Not all researchers are fully competent in all aspects of their project as might be hoped. Occasionally authors deliberately report work fraudulently, and may leave tell-tale signs of the fraud that an experienced reader will notice.
Peer-review processes are intended to ensure that only high quality research is published. However:
- in some contended fields, peer reviewers may be asked to look for an argument with merit even where this would not be persuasive to most people in the relevant research community – a reader may need to make a judgment about published work in the light light of their wider reading of a field
- peer reviewers are human, and have their own biases and blind spots, so although peer review doubtless excludes much poor quality work from publication in reputable outlets, it is not a fail-safe mechanism
- not all published work has passed through rigorous peer review; there is much 'grey literature' (reports, conference papers, discussion papers) and, increasingly in some academic areas, alternative models of publication such as publicly shared pre-prints appearing ahead of peer review. (More traditionally, such pre-prints may have been circulated privately by an authors to those invited to offer feedback before finalising a paper: now such materials are often freely searchable and downloadable from the web.)
- there is now an infestation of a great many supposedly academic journals (often with impressive enough titles) being published by commercial publishers regardless of the quality of editorial and peer review support. Some of these lack the support of experienced academics that are needed to run a good journal: but many will publish material for income regardless of quality. As I have discussed in some of the blog postings, some complete nonsense gets published as research papers in some of these journals.
Two stances to reading research
Research reports can be treated as authoritative – that is, by assuming that the conclusions and recommendations made in research reports should be taken at face value. That is generally appropriate for a lay population – the typical member of the public does not have the expertise to critique research, and so is probably wise to generally take advice from experts (which is why people generally consult medical doctors, solicitors, electricians, and others with privileged knowledge and experiences in a specialist area.)
Alternatively, research can be read critically, where the reader does not rely on the conclusions and recommendations alone, but looks to interrogate a report to ascertain for themselves the extent to which such conclusions and recommendations are justified by an argument from evidence presented. That is, rather than take outcomes on authority, the reader looks to reach an informed judgment for themselves. This stance is feasible when the reader of a research report themselves has sufficient expertise to be able to test the arguments made in the report.
An expectation on researchers
Researchers themselves are expected to develop sufficient expertise in their area of work to be able to critical evaluate other people's research in their field for themselves.
Students (especially those working for higher degrees) are usually expected to build up knowledge about methods and standards of enquiry in their own discipline so that they can read scholarly work critically and demonstrate this critical faculty in their own assignments.
Research students are expected to acquire quite high levels of skills in evaluating research in their own topic, field or discipline, and to apply this in their own work – for example in developing a conceptual framework (an account of the current state of knowledge relevant to a research project) to inform their own research plans.
Research reports as arguments
(After, Figure 3.4, Taber, 2013)
A research report usually makes claims for some new knowledge, something the author(s) claim(s) is now known as a result of the research. Drawing this conclusion depends on a chain of argument that is inherently linked to the research design (see the figure). In principle, judging that one link in this chain is flawed (use of a research instrument that is invalid in the study context; misinterpretation of a statistical result, etc.) is enough to throw doubt upon the entire argument.
Reading research critically
In reading research studies critically, a reader needs to identify the steps in the argument supporting a claim, determine whether there is a complete chain of argumentation, and judge whether each step in the argument is convincing.
Critical Reading of Empirical Studies Tool
The Critical Reading of Empirical Studies Tool was designed as a simple aid to help those undertaking such reading – especially those new to evaluating research report.
Read about, and access, the Critical Reading of Empirical Studies Tool
Work cited:
- Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.
