Date: Wed, 25 Apr 2007 10:52:36 -0700
Reply-To: "Pardee, Roy" <pardee.r@GHC.ORG>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: "Pardee, Roy" <pardee.r@GHC.ORG>
Subject: Re: Computing power post-hoc
In-Reply-To: A<6250203B042D8349A920AA610383093602DF40EB@UTHEVS3.mail.uthouston.edu>
Content-Type: text/plain; charset="us-ascii"
Sure, absolutely--this goes to your earlier point, that it's much more
useful to assess power ahead of time, with a view toward assuring that
you will have power enough to detect a meaningful effect, right? No
argument from me on that score. But mightn't it *also* be useful to
assess after the fact, to see if you really got the power you thought
you would & if not, how much you did get?
Incidentally, back when I was in grad school (psych) there was much
lamentation of the fact that journal reviewers would not recommend
publishing reports of experiments that failed to reject the null--no
asterisks, no publication. I'd be interested to hear whether that tide
has turned at all... Off list I suppose, as that's way off-topic.
Thanks for entertaining my questions, all.
Cheers,
-Roy
-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of
Swank, Paul R
Sent: Wednesday, April 25, 2007 10:34 AM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: Computing power post-hoc
Why not just look at the effect size? If Cohen's d (for example) is .2
then the effect is small and you know it would take a larger sample to
detect it. However, if an effect size of .2 is meaningful in a
particular area of research, the researcher should know this and make
sure he/she has a reasonable sample to detect such effect sizes.
Paul R. Swank, Ph.D. Professor
Director of Reseach
Children's Learning Institute
University of Texas Health Science Center-Houston
-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of
Pardee, Roy
Sent: Wednesday, April 25, 2007 11:38 AM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: Computing power post-hoc
Huh--interesting--thanks.
My vague (and fading!) recollection of my stats courses nominated power
analysis as a means for assessing the type-2 error. So--I fail to
reject the null--no publication for me, boo hoo, wail/gnash, etc. ;-)
But power analysis was suggested as useful for assessing whether there
was actually anything to be said positively, in favor of the null
hypothesis.
So if my data are such that I could have detected an effect all the way
down to size X, and we know that X is never going to be real-world
significant in a million years, then maybe we can say that--hey, whether
or not X is really zero "in the eyes of God", it is almost certainly not
going to be larger than X, and so it's not worth studying any further.
Is that plausible at all, or have I just totally screwed the logic?
Thanks!
-Roy
-----Original Message-----
From: Peter Flom [mailto:peterflomconsulting@mindspring.com]
Sent: Wednesday, April 25, 2007 9:09 AM
To: Pardee, Roy; SAS-L@LISTSERV.UGA.EDU
Subject: Re: Computing power post-hoc
"Pardee, Roy" <pardee.r@GHC.ORG> asked
>
>Can you answer the question "how big would my effect size had to have
>been in order for it to be statistically distinguishable from zero?"
>w/a post-hoc power analysis? I would think that could be useful to
know...
>
>Thanks!
Well...... it depends
that is exactly the sort of question power analysis is designed to
answer.
HOWEVER, what we stats-geeks have been discussing is whether those
answers are good answers. The problem is that the answers depend on the
assumptions you make.
So.....
How big would my effect size have to be?
given what? The same variances? Hmmmm.....in lots of situations, a
bigger effect size would mean different variances. How different? Who
knows?
The whole question gets into the philosophy of statistics, as well, and
to the question of whether p-values are really worth anything in most of
the situations they are used in. My answer is "usually, they are not
worth much". Why not?
Well, what's a p-value? The chance of a type-1 error. What's a type-1
error? It's saying the null is false WHEN IT'S TRUE. The null
hypothesis is usually something like "nothing is happening". This is
NEVER literally true. There is always SOME difference between two
populations. If you weighed everyone with an odd social security number
(all 150 million people) and everyone with an even SSN (another 150
million) they would NOT be exactly the same. In addition, in most
research, we strongly suspect that there is something really going on.
In this case, the chance of a type 1 error is 0, because something IS
going on, so you cannot be wrong if you say something is going on.
The p-value is (in gross terms) a measure of how much shit will stick on
a wall when you throw a lot of it. :-)
Also, the p-value measures the WRONG thing. That is, the p-value
measures:
If nothing were going on, how likely are these data?
that is not usually what we want to know. Rather, we usually want to
know
Given these data, how likely is it that nothing is going on?
or, even better,
Given these data, what is our best guess as to what is going on, and how
good is that guess?
This question cannot be answered by p values
to quote David Cassell
I am HTCT
Peter
