Peer review comments

Peer review is the process by which articles (or book chapters, book  manuscripts, book or journal proposals, funding applications, etc) are scrutinised by experts in the field who comment on their strengths and weaknesses, recommend whether they should be published (or funded etc) and/or what changes might be required. Referees will comment on a wide variety of features of submissions.

Peer review reports highlight strengths and weaknesses of submissions. Sometimes journals offer reviewers categories of response, or ask specific questions. Generally it is assumed reviewers will apply academic standards that are shared within a research field.

Here are some examples of the kinds of things reviewers criticise (note, praise it not included here) about submissions as making them unsuitable for publication in research journals:

“The paper cannot be published in a research journal in its present form, as it does not make any new contribution to knowledge.”

“...clear research questions do not seem to be established”

“The sample consisted of 610 students. Nevertheless, the authors opt for a 'qualitative case-study analysis'. However, quite in line with the large sample size, the analysis has been quite shallow: the fragments of student discourse are presented without any contextual interpretation, which makes it impossible as a reader to assess the validity of the given interpretations. The only advantage I could see of having such a large sample, would be if there would be any frequency data, but these are not presented.”

“The main problem with the paper is that the earlier phases of the research involve interpretive (qualitative) or 'exploratory' stages, and yet there is far too little presentation and analysis of data offered in reporting these phases.”

“A theoretical framework for teaching and learning is not clearly elaborated..a much more elaborated theoretical framework that guides the work is needed...”

“Throughout the paper there appears to be a lack of rigor of argumentation.”

“A rather fundamental problem is that the "nature" of the empirical study and the very aims of the study are not explicitly provided.”

“In sum, a less ambitious title that better expresses the real focus of the study would be more appropriate....”

“The study was carefully designed; however, the description of the methods requires further clarification.”

“The literature review section is not structured within a suitable context and the literature chosen is not related to the central issue of the paper in a meaningful way.”

“My concern here goes back to my comment about the use of "interpretations." The authors state an assertion, give some initial interpretation of the quote, offer a teacher quote, and then give more interpretation of the quote. First, the authors' interpretations seem to be more far-reaching than the content of the teacher quotes. I found myself reading and re-reading the teacher quotes, and then reading and re-reading the interpretation of each quote, and then asking myself, "Is that really what the teacher is trying to say here?" Again, I know that a "member checking" process was implemented; however, some of the interpretations seem so far-reaching that it makes me question the validity of the assertions.”

“I do not believe this manuscript should be published. More directly, I do not believe this research can ever be published. No revision of text will ultimately be able to redress the fatal design error of the study.”

“I do not believe the author’s claims are supported primarily because the validity of the constructs being compared are not well established. Yes there appear to be significant correlations and differences in group means, but this can be meaningful only if one believes the anxiety and attitude instruments are valid measures of these latent constructs.”

“In the methodology part, the researchers did not state anything about the trustworthiness of the study. What did they do to ensure it? (e.g., member-check, data triangulation, inter-rater reliability. Etc.)”

“Too much detail was given to explain what happened in the class. It makes reader tired.”

“Because the number of participants in this course is so low a few students shifting from one response to another may appear significant, but could simply be part of the randomness in a sample this small. …If the authors are intent on making claim regarding the significance of changes, I would suggest repeating this exercise to increase the number of participants.”

“The literature on metacognition …is very thin. This leads to a rather simplistic portrayal of the field and also the working definition of metacognition. A deeper understanding of metacognition is needed.”

“Where is ethics and informed consent of student participants addressed in the study? This is particularly important because it is possible that one class of students, for the purposes of the study, might have been disadvantaged by not being given the different instruction.”

“The question scoring rubric needs to be supplied so that sense can be made of the scoring and how it was done. Samples of the scoring of students questions using the rubric also need to be supplied for review.”

“I think the authors overstate their findings when they claim on the effectiveness of …. Again, I did not find solid evidence in the result sections that supports their claims.”

“this study lacks an appropriate theoretical ground and evidence from empirical studies to support the effectiveness of the metacognitive instructional design and support their discussion. The data is also relatively thin.”

“it is then not clear to me how your coding is grounded in the data. You illustrate your descriptions with good examples, but it is not clear how you did the analysis and came to your categories.”

“There are some orthographical mistakes.”

“What I think could use more consideration is a discussion of the exact transcription process and how that affected the data”

“without an example from the data corpus, I really don’t understand the concept behind the XXXX category”

“I am not sure why this is labeled argumentation, because there is no element of the kind of interaction representative of argumentation. Second, this is an overly kind interpretation of what is going on here.“

“Ontologically, it does not make sense to group these two very different purposes into one code category. The first purpose is clearly social in nature and the second is clearly academic. It would be nice to see more of a justification for such a broad category.”

“The evidence indicates there was certain ways the teacher asked questions but there is no evidence regarding retention of those ideas or how they impacted student understanding of the content, which was one of the main reasons provided as justification for the study.”

“Research hypothesis: Page 6. The first hypothesis sounds weird to me”

“The conclusion should be rephrased to make it more representative for the study.”

“there should be some explanations how students in the control group were fulfilled or after the study to make it fair for them.”

“The main issue with the submitted paper is that its conclusions are rather trivial, considering the body of research we currently possess”

“In the opening sections, the meaning you attach to terms, such as meaningful learning, concept and conception, could be briefly explained.”

“Given that the paper is claiming that the dialogic intervention helped to change the preservice teachers’ understandings, I would have expected to see a more systematic analysis of their before and after understandings.”

“The procedures for developing the coding scheme (and its use in analysis) are vague. How were codes/categories developed?”

“a more structured presentation of results is needed, both to illustrate your findings but also to make later claims from this research and show how it adds to the literature“

“There needs to be a closer link between individual data points, their origins, and their relationship to analysis processes and analyses…Here is another example illustrating the need for a tighter analysis process and more structure to the presentation of results: to illustrate conceptual change makes no sense at the class or group level, partly because the data do not support this, but also because the construct refers to an individual’s understanding.”

“The authors need to build a stronger case for the need for their study and more deeply reflect on its implications. After reading the paper, it is not clear to me what was gained, from an educational point of view”

“In general, the authors need to deepen the qualitative analysis of their data, the discussion of their results, and the implications for teaching and learning of their findings. In particular, they need to extend and deepen their analysis and reflection on the educational insights that are gained”

“The title does not seem to fit the paper.“

“What theoretical/conceptual framework does this paper use?”

“he way that the authors tested the validity of the tool was by computing the correlation coefficient of representation cards. Computing the correlation coefficient is a kind of inferential statistics method. The situation of the small number of participants (9 participants) was not suitable for inferential statistics. In addition, the authors did not report the significance of the correlation coefficient”

“Unfortunately, the research design of this study contains irreparable choices and as such, I recommend rejection of the manuscript.”

“First, there are no research questions stated, thus it is unclear what contribution this paper may make“

“Third, many details of the methodology are missing. We don’t know what the measurement instruments … look like, and what validity and reliability evidence was available to support their uses”

“because of its simple design and the self-evident findings it does not contribute much to the literature”

“the paper lacks theoretical rigor and interpretation of the finding. The hypotheses … are rather naïve and they do not serve the development of theory.”

“A major obstruction in the design of the study is the apparent lack of control of an extraneous variable, …which could have a significant impact on the dependent variable.”

“the potential problem with convenient sample used in the study is that the variation in student abilities of the sample may not be wide enough to represent the variation of other student populations.”

“there are lots of limitations such as small sample size and absence of control group. The introduction section is not sufficient to convince the reader about the significance of the study. There are lots of missing in the method section like procedure, treatment fidelity, treatment verification, ethical issues and confidentiality of the participants. The results, discussion and conclusion is very poor. I suggest the authors to conduct a new study with larger sample and control group to strengthen the results of the study.”

“The main weaknesses I found in the manuscript are uninformative abstract, weak arguments in the introduction section as well as in the discussion section, unclear description of research method used in research presented in the manuscript as well as the presentation and explanation of statistical analysis (tables and figures included).”

“this manuscript is suffered from inadequate design and poor writing. The title, research questions, rationale, the research design, and content of findings were not aligned”

“Because of the flaws of research method, the findings are shallow. The transferability of the findings of the study is limited. The authors did a poor job on writing the findings. I could not find their claims of the major themes in the Results sections."

“Although the authors introduce a novel idea, this framework is not productively used in the presentation and discussion of results. The analysis of results needs to be deepened and it must be aligned with the analytical framework advanced by the authors.”

“the article is too ambitious, describing too many “concepts” with little or no data to support the concepts, and the methods are not described well enough leading to confusion for the reader or questions about the methodology.”

“The paper does not include any discussion of the implications for research and/or practice of the core results of this study. There is also no discussion of what new insights about student understanding are gained through this investigation”

“Segments of excerpts and interpretations did not address the research questions; there are also long excerpts presented without interpretation. More importantly, I did not find data and convincing evidence that could answer the research question.”

Index of topics