I am about to describe a recent research result from a first-tier referred journal.
It will sound obvious and trivial and unimportant.
But it is not.
And in understanding why it is important, a critically important deep insight into statistics in particular and research, in general, is yours for the taking.
Are you ready?
Here is the result:
People with severe side effects tend to stop taking drugs more than people who don’t have side effects.
Not surprising, is it?
However, decades of research results tell us that side effects generally go down with continued use.
Everyone knows it to be true.
But it’s a lie.
Why has it taken decades for someone to point out (after carefully measuring) that it is not that things get better but that when things are bad (and getting worse), people stop?
The graphs, charts, and calculations were all carefully done—no error there for the decades of reviewers to find. Rather, a fundamental error in all the experimental designs went unnoticed.
Three questions for you:
- Did the result initially sound important?
- Does it seem important now?
- What other mistakes like this are we not noticing?
Recently, I saw a result of the form: 49 out of 50 respondents said our product worked for them. They didn’t mention that 950 out of 1,000 customers surveyed declined to answer.
I’ve seen this particular problem over and over again by people who honestly thought that they had something wonderful when they had nothing. In each case, before I had spoken to them, dozens of people had reviewed their results without asking about the response rate.
In each case, resources were needlessly wasted or at least unnecessarily put at risk.
If you encounter this error or even made it yourself, please add a note in the comments.
PsyPost has a good summary of the article.
