A topic in academic standards
One aspect of academic honesty, is being objective in reporting the outcomes of research.
Yet studies are sometimes published which make claims that are clearly unsupported by the evidence presented in the study. In some cases, it is likely that researchers are so convinced in the merits of their work that they are blind to obvious flaws in the case they are making. These problems should be spotted in peer review such that an editor would not publish the paper until (at least) the errors are corrected, and suitable provisos and feasible alternative explanations for outcomes are included in the report.
Other times authors may go as far as to falsify their conclusions in the sense of including a conclusion that is contrary to their results. Whilst this should also be spotted by peer reviewer, it is a mistake researchers should not make.
- It is acceptable for researchers disappointed by the outcomes of a study to suggest reasons why their results may be influenced by factors that have distorted the outcomes, and suggests that further more careful research may lead to different outcomes (and then readers can decide if they agree)
- It should not be acceptable for researchers obtaining a negative (non-significant) result to simply ignore this and frame their conclusions as though the outcome had been positive (significant).
These flaws are common in papers published in predatory journals that have little concern for the quality of what is published. But sometimes they also slip through peer review in more prestigious journals, so readers need to check they are convinced by the chain of argument in reported studies.
A failure to make the case
Grades are improved by casting out demons
This paper was published in the 'European Journal of Education and Pedagogy' (which I suspect from the quality of this work is a predatory journal). The paper claims that "hallucinatory disorders can be overcome". The basis for this claim was that the graves of five students (who's grades dipped when they were suffering from conditions that included hallucinations) recovered after the hallucinations stopped following an educational module of soul purification. Whilst it may indeed be true that these poor students' grades increased after their hallucinations stopped following an educational module of soul purification there is no reason to assume any logical connection.
The author seems to assume that hallucinations are due to possession by some kind of demon or spirit, and that exorcism therefore stops the hallucinations. No consideration at all is given to medical diagnoses (surely an ethical issue) or indeed medical treatments that may have coincided with the educational module. On the basis of this kind of argument this journal would publish a paper claiming that singing Christmas Carols in church in December is the cause of the Northern spring returning each year. (Ironically, perhaps, as that's the ultimate reason why Christmas was scheduled near the Winter solstice.)
Fabricating conclusions
Discounting negative (non-significant) results
The authors of this study in Chemistry Education Research and Practice (a prestigious journal) claimed that their research findings "refute those of [authors of a previous study] who found no statistical difference in learning gains…" in comparing two teaching approaches. One issue with this claim was that the two studies were dissimilar in ways such that the later study could not really be be seen as a replication of the earlier one, so could hardly refute it.
A bigger issue was that the authors' results show no significant differences in the learning gains between their two conditions – so this could not refute a previous claim of no statistical difference in learning gains. This seems to be an example of authors using inferential statistical tests and then deciding to ignore them in drawing their conclusions. (The journal later published a 'comment' highlighting these issues.)
Discounting negative (non-significant) results [again]
The authors of this study in the Journal of Chemical Education (a prestigious journal) compared the average class scores after implementing an innovation with those obtained previously. They had no test of equivalence between cohorts, and must have changed their assessment methodology given the nature of the shift in teaching approach. Despite this, the average class scores remained virtually unchanged (80% to the nearest percentage point). As one would expect given this, these differences were found to be non-significant (a negative result). This did not stop the authors concluding that their results suggested the innovation improved teaching and learning outcomes (a positive result)! (The journal declined to publish a letter pointing out the problems with the study.
My introduction to educational research:
Taber, K. S. (2013). Classroom-based Research and Evidence-based Practice: An introduction (2nd ed.). London: Sage.
