Tooltip Categories: Comparisons

An example of a teaching analogy:

"In practice, many studies rely on testing for a statistically significant difference, although this is a very dubious criterion for equivalence…. This becomes clear if we consider how inferential tests are interpreted when comparing the final outcome measures in a study. At the end of a true experiment, statistical tests may be used to infer that a difference in final outcomes was unlikely enough that we can confidently assume it is not due to random variations but is due to a systematic difference (i.e., the difference between the experimental treatment and the control condition) and so can be assumed to (probably) apply more generally in the population and not just to this specific permutation of learners. So, a statistically significant difference means a very unlikely one (in practice, normally one with a probability value, p<0.05).

Often a similar approach is used in studies to evaluate the differences found at pre-test. The results are analysed to see if there is a very unlikely difference between the scores in the different conditions. If a statistically significant difference is found, then this clearly suggests the groups cannot be considered equivalent. That is reasonable.

Unfortunately the reverse does not apply: if the differences do not reach significance, we cannot assume that implies equivalence. Say p=0.08 (which means that the differences were unlikely enough to only occur by chance about once in 12 times, rather than once in twenty times as when p=0.05), this still shows there was a difference that was unlikely to be down to random factors. There is a logical difference between what we are seeking to do in these two situations. In one case (comparing post-test results), we are trying to exclude all but those outcomes that are most unlikely to be chance events, and in the other (comparing pre-test scores), we are trying to show that any difference is small enough to be insignificant in affecting later outcomes. So, in the first case, we are trying to show something is very improbable, but in the other case, we are trying to show we have a very probable outcome. So, using the same kind of inferential test as a test of equivalence means (sensibly) excluding cases with very different pre-test outcomes across treatments from being labelled equivalent: but still (dubiously) admitting other substantially different pre-test outcomes across treatments as being equivalent.

If this seems a little abstract, consider this analogy. Consider Table 2.2 which presents two questions that might be posed to a learner and her hypothetical responses. The two questions are looking at two different ends of a spread (that is a continuum from combinations of elements with very different electronegativities to combinations of elements with the same electronegatiity) and a suitable criterion that works for one extreme cannot be simply reversed to be used at the other extreme (which would be like saying anyone who is not over 2 m tall should be considered short). That is, if we agree that an electronegativity difference of >2.5 is a good criterion to identify highly ionic compounds, then it is inappropriate to use the same cut-off as the basis for a criterion (<2.5) to identify highly covalent (i.e., non- polar) compounds. Perhaps we should instead look for an electronegative difference <1.0 or <0.5? The precise choices are open to opinion (you might actually suggest >3.0 for the most ionic compounds): but the invalidity of using the same cut-off to identify both sets of extreme cases is not. If we decide the most covalent compounds are those where the electronegative difference is <0.7, we should not then class any where the difference is >0.7 as ionic.

Taber, Keith S. (2024) Chemical pedagogy. Instructional approaches and teaching techniques in chemistry. Royal Society of Chemistry. [Download Chapter 1]