Date:         Mon, 3 Aug 2009 14:42:45 -0700
Reply-To:     jasonm@UCLA.EDU
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         J M <jasonm@UCLA.EDU>
Subject:      Re: Question on Comparing Two Averages
Comments: To: Peter Flom <peterflomconsulting@mindspring.com>
In-Reply-To:  <26320529.1249325584783.JavaMail.root@elwamui-little.atl.sa.earthlink.net>
Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed"

>
> HOWEVER, with n = 4,500, a tiny difference will be statistically
> significant. Will it be of any importance?

So, if I have 20% respond "yes" in 2008 and then 22% respond "yes" in
2009 I can say that due to the large sample the difference would most
likely be statistically significant if we had the correct data to
actually test this claim?


Quoting Peter Flom <peterflomconsulting@mindspring.com>:

> J M <jasonm@UCLA.EDU> wrote
>> I'm wondering what people's thoughts are on this:
>> A group of people were tested in 2008 and asked to reply yes or no to
>> a series of questions (n=5,000).
>> They were tested again in 2009 and asked to reply yes or no to the
>> same series of questions (n=4,500).
>> There is no raw data. The response rates for each of the questions in
>> 2008 and 2009 is unknown. We just know how many people answered yes to
>> each of the questions in 2008 and 2009.
>> Although this would be a very crude analysis, would the results of a
>> dependent t-test comparing the two averages on each of the questions
>> to see if the difference is significantly different from zero between
>> 2008 and 2009 mean anything?
>
> How would you do a dependent t-test?  To do that you would need to
> know who got how many answers
> right, and you say you don't know that.  You'll have to do the much
> less powerful independent sample t-test.
>
> HOWEVER, with n = 4,500, a tiny difference will be statistically
> significant.  Will it be of any importance?
>
> You don't say how many questions were asked, or what the
> distributions are, so it's hard to even give an example
> that would be useful.
>
> You also don't say why you are doing this.  You don't say what the
> questions are, or why you are comparing them.
>
> PLUS, the bit about "response rates are unknown" makes it sound like
> you did some kind of survey.  If you don't know the
> response rates, then nothing you do will be of much value (sorry).
>
> Peter
>
>
> Peter L. Flom, PhD
> Statistical Consultant
> www DOT peterflomconsulting DOT com
> http://www.associatedcontent.com/user/582880/peter_flom.html
>