Researchers from the Universities of Manchester, Oxford and Arkansas are urging scientists to stop treating “non-significant” results as proof that nothing happened.
The common statistical mistake, they warn in a new paper in PNAS on 10 08 26 – one of the world’s leading scientific journals – could be leading researchers to draw the wrong conclusions from their data.
A p-value, which measures how surprising the observed data would be if there were truly no effect, of greater than 0.05 does not, they say, show there is ‘no effect’.
However, the interpretation remains widespread in about 50% of research papers and conference presentations, according to sources.
Instead, the authors say, a non-significant result simply means there is insufficient evidence to conclude a difference exists, not proof that a difference does not exist.
Crucially, the same statistical result can arise either because there is genuinely no meaningful effect or because an important effect is hidden by small sample sizes or highly variable data.
The researchers argue that failing to recognise this distinction risks oversimplifying scientific findings and may cause potentially important effects to be overlooked.
With many studies not having a large-enough sample size to detect the real difference, reporting that there is none can miss promising therapies or fail to detect genuine risk, derailing subsequent research in the area.
Equivalence testing, they say, guards against declaring an effect unimportant when the data cannot support that claim. A small or noisy study will usually return an inconclusive result rather than a verdict of no difference, which is the honest answer.
To address the problem, the team is promoting a statistical approach known as equivalence testing.
Rather than asking whether there is evidence for a difference, equivalence testing asks whether any difference that exists is too small to matter in scientific, clinical or practical terms.
The method allows researchers to distinguish between effects that are genuinely negligible and results that remain inconclusive because there is not enough reliable evidence.
They focus on the two one-sided tests procedure, or TOST, which has already gained traction in psychology, medicine and pharmaceutical regulation but remains underused across many areas of the life and natural sciences.
Wider adoption of equivalence testing, they add, could improve the quality of scientific reporting and help prevent non-significant findings from being misrepresented as evidence of no effect.
“The University of Manchester is a public research university in Manchester, England. The main campus is south of Manchester City Centre on Oxford Road.”
Please visit the firm link to site
