Search This Blog

Showing posts with label Stats. Show all posts
Showing posts with label Stats. Show all posts

July 28, 2017

New proposal to shift statistical threshold from p < .05 to p < .005

  1. Boards
  2. Current Events 
  3. New proposal to shift statistical threshold from p < .05 to p < .005
COVxy 1 day ago#1
http://www.nature.com/news/big-names-in-statistics-want-to-shake-up-much-maligned-p-value-1.22375

Full preprint:
https://osf.io/preprints/psyarxiv/mky9j

Shifting alpha to a new arbitrary value is literally pointless, especially in the way they recommend.

If you are a scientist, you need to understand statistics and data analysis, that's really all. And I don't think that this is largely the issue (I think most do understand the statistics deeply), but it's a good scapegoat for the culture of pushing positive results and the journaling system, which are much more difficult to fix.
=E[(x-E[x])(y-E[y])]
SageHarpuia 1 day ago#2
Literally why would you care?
My name is Harpuia, one of the four Guardians of Master X and General of the Strong Air Battalion, The Rekku Army.
COVxy 1 day ago#3
SageHarpuia posted...
Literally why would you care?


Cuz I science often.
=E[(x-E[x])(y-E[y])]
qyll3 1 day ago#4
SageHarpuia posted...
Literally why would you care?


p-value cutoffs affect significance of findings, which in turn affect chances of being published in reputable journals, which in turn affect career prospects. So shifting the cutoff would substantively affect millions of researchers around the world (including myself).

Fisher never intended for the p-value to be used as a hard cutoff for meaningful findings but as one aspect to be considering among a bunch of other factors. It's too bad that that's what it's become.
No sig here
Darkman124 1 day ago#5
i feel like the problem is not that .05 is too high

it's that media tends to exaggerate the significance of findings with p-values that evoke lesser confidence
And when the hourglass has run out, eternity asks you about only one thing: whether you have lived in despair or not.
COVxy 1 day ago#6
I guess it's less laughable than the time someone published, in a pretty highly reputable (but not well regarded) journal, a suggestion to switch from p-values to confidence intervals, even though they are literally mathematically equivalent.
=E[(x-E[x])(y-E[y])]
This is fantastic news and I hope it happens.
COVxy 23 hours ago#8
Transcendentia posted...
This is fantastic news and I hope it happens.


You're, uh, really bad at this.
=E[(x-E[x])(y-E[y])]
JM_14_GOW 23 hours ago#9
I am kinda rusty on statistics, but isn't a p value of 0.005 way too low to be practical? Like thats a very small part for outliers and in practice it would almost include the whole population?
Playing: Rainbow Six Siege/Battlefield 1/Dark Souls 3
ZMythos 23 hours ago#10
I think 0.01 or 0.02 would be more reasonable. 0.005 is 10 times more significant. That's ridiculous with some distributions.
Rainbow Dashing: "it's just star wars"
AutumnEspirit: *kissu*
luigi13579 23 hours ago#11
Transcendentia posted...
This is fantastic news and I hope it happens.

We've all had enough of experts.
Transcendentia 23 hours ago#12
you guys mad?
Darkman124 23 hours ago#13
JM_14_GOW posted...
I am kinda rusty on statistics, but isn't a p value of 0.005 way too low to be practical? Like thats a very small part for outliers and in practice it would almost include the whole population?


iirc it would correspond to 3-sigma instead of 2-sigma on a normal probability distribution

it's not 'too low to be practical' so much as 'unnecessary and likely to suppress useful data'
And when the hourglass has run out, eternity asks you about only one thing: whether you have lived in despair or not.
(edited 23 hours ago)quote
JM_14_GOW 23 hours ago#14
Darkman124 posted...
JM_14_GOW posted...
I am kinda rusty on statistics, but isn't a p value of 0.005 way too low to be practical? Like thats a very small part for outliers and in practice it would almost include the whole population?


iirc it would correspond to 3-sigma instead of 2-sigma on a normal probability distribution

it's not 'too low to be practical' so much as 'unnecessary and likely to suppress useful data'


Yeah thats more or less what I am trying to say, basically its including the 99.995 of the population and you lose a significant amount of outliers imo.
Playing: Rainbow Six Siege/Battlefield 1/Dark Souls 3
Darkman124 23 hours ago#15
JM_14_GOW posted...

Yeah thats more or less what I am trying to say, basically its including the 99.995 of the population and you lose a significant amount of outliers imo.


p<.005 is equivalent to a 99.5% confidence interval, which in a normal probabiltiy distribution is three standard deviations from the mean

it's common for engineering projects to require 3-sigma for safety of designs as it's treated as being as close to "we know this won't happen" as is possible, since the standard deviation there is typically a manufacturing tolerance.

i think it's a lot less viable in pure science where the error is influenced by things outside your control and cannot be as easily minimized
And when the hourglass has run out, eternity asks you about only one thing: whether you have lived in despair or not.
(edited 23 hours ago)quote
PiOverlord 23 hours ago#16
Nah.
Number of legendary 500 post topics: 26, 500th posts: 15; PiO ATTN: 2
Thank the lord, the PiOverlord! RotM wins 1
CreekCo 23 hours ago#17
Hmm. Tagging for interest later.
*Triggered*
COVxy 21 hours ago#18
Darkman124 posted...
it's not 'too low to be practical' so much as 'unnecessary and likely to suppress useful data'


This is pretty much the primary issue. The false negative rate in a lot of fields is a much larger issue than the false positive issue.

We really shouldn't be paying too much attention to arbitrary cut offs, tbh.
=E[(x-E[x])(y-E[y])]
shockthemonkey 21 hours ago#19
tag
Support local music.
But not if it sucks.
COVxy 14 hours ago#20
Up.
=E[(x-E[x])(y-E[y])]
Polycosm 14 hours ago#21
SageHarpuia posted...
Literally why would you care?

His user name is literally COVxy.
BKSheikah owned me so thoroughly in the 2017 guru contest, I'd swear he used the Lens of Truth to pick his bracket. (thengamer.com/guru)
Shmashed 14 hours ago#22
ZMythos posted...
I think 0.01 or 0.02 would be more reasonable. 0.005 is 10 times more significant. That's ridiculous with some distributions.


The fact that people are saying things like this shows how arbitrary it is. It really depends more on the context. For some areas, like maybe marketing, a p value of 0.1 is fine. In something like astrophysics the p value needs to be far lower than 0.05
/poast
Transcendentia posted...
you guys mad?

Oh man, you are not good at this
- literal garbage
COVxy 14 hours ago#24
literal_garbage posted...
Transcendentia posted...
you guys mad?

Oh man, you are not good at this


Must be meta-trolling from the great Clad.
=E[(x-E[x])(y-E[y])]
MangaFan462 14 hours ago#25
Anteaterking 14 hours ago#26
Sativa_Rose 14 hours ago#27
I don't have a whole lot of knowledge on this subject, but from my perspective, it seems like different fields would want to use different p value thresholds. It doesn't make sense to have a universal threshold that applies across all fields.
I may not go down in history, but I will go down on your sister.
scar the 1 14 hours ago#28
I thought the threshold varied depending on the field and type of study, and that 0.05 was more of an informal standard than anything?
Everything has an end, except for the sausage. It has two.
COVxy 14 hours ago#29
Sativa_Rose posted...
I don't have a whole lot of knowledge on this subject, but from my perspective, it seems like different fields would want to use different p value thresholds. It doesn't make sense to have a universal threshold that applies across all fields.


Meh, it's more like a careful consideration of an argument in the context of the totality of the evidence is more important than an arbitrary cut off on any one of the tests.

I'd be completely fine accepting a paper with no statistically significant results if all of the data pointed towards a single conclusion, if there were enough convergent evidence.
=E[(x-E[x])(y-E[y])]
  1. Boards
  2. Current Events 
  3. New proposal to shift statistical threshold from p < .05 to p < .005