Date: Mon, 7 Nov 2005 16:30:45 +0100
Reply-To: Marta García-Granero
<biostatistics@terra.es>
Sender: "SPSSX(r) Discussion" <SPSSX-L@LISTSERV.UGA.EDU>
From: Marta García-Granero
<biostatistics@terra.es>
Organization: Asesoría Bioestadística
Subject: Re: Samplesize?
In-Reply-To: <10049.1131374952@www56.gmx.net>
Content-Type: text/plain; charset=ISO-8859-15
Hello Karl,
KK> Thank you again for your comprehensive and well explained answer.
You're wellcome.
>> Anyway, I see another complication in your design: it is a factorial
>> design (Factor A: system; factor B: task). Do you foresee an
>> interaction between system and task? (like: system A is faster for
>> task 1, but slower for task 2). In that case, you should consider the
>> sample size required to detect that interaction (*), and the fact that
>> the statistical analysis for system A vs system B would be more
>> complicated than a simple repeated measures ANOVA.
KK> My hypothesis is that system A serves the user better than system B. System
KK> A uses an approach I have developed, system B does implements an existing
KK> approach which serves as a baseline. I measure the performance along several
KK> measures. First the time users need to perform a task, second the average
KK> user ratings (these ratings are given by the user on results the system
KK> returns to the user during the task), and third the amount of user actions
KK> necessary to complete the task. I use three measures since they provide a
KK> more comprehensive view about the effect of the new approach.
KK> I am mainly interested in the difference between these two systems. I see
KK> the tasks as some realistic problem for a potential user of such a system.
KK> I therefore think I have two IVs in this case, system and task. However, I
KK> am not particularly interested in finding out about the nature of each task
KK> in relation to the system.
Even if you are not interested in the nature of each task, it's
another IV in your model. You have 4 groups, therefore you have 3 df
that can be further split into:
- 1 df: "System" main effect
- 1 df: "Task" main effect
- 1 df: Interaction between task and system
Lack of interaction is needed to be able to talk about a general
"system" main effect (is system A better than system B for any task?).
If interaction is present, then you can only answer questions related
to "simple effects":
(a) for Task 1: is system A better than system B?
(b) for Task 2: is system A better than system B?
The presence of interaction might be due to the "system" effect being
restricted to only one task, or to the "system" effect being much
stronger for one task than for the other, or even for reverse effects
(while system A is better than system B for task 1, for task 2 the
effect is reversed and system B is better than system A...)
KK> It is more about that the task represents the
KK> general use of the two systems. I know that I have the danger, that if the
KK> user tasks do not fulfill that, I might get bad results because of that and
KK> not because of the system that was used. I think this is what you refer to
KK> as interaction. However, I am more interested in the general outcome and the
KK> average result rather than a factorial division between system and task. I
KK> have counterbalanced the order in which people perform these tasks according
KK> to the following table:
KK> 1st Task 2nd Task 3rd Task 4th Task
KK> ------------------------------------------------------------------------
KK> User 1 | Task A@SystemA Task A@ SystemB Task B@ SystemA Task B@ SystemB
KK> User 2 | Task A@ SystemB Task A@ SystemA Task B@ SystemB Task B@ SystemA
KK> User 3 | Task B@ SystemA Task B@ SystemB Task A@ SystemA Task A@ SystemB
KK> User 4 | Task B@ SystemB Task B@ SystemA Task A@ SystemB Task A@ SystemA
KK> ------------------------------------------------------------------------
KK> However, to come back to your suggestion, I would have the data to do both,
KK> a factorial 2x2 full factorial ANOVA between results from 2 tasks on 2
KK> systems (within subject) or a simple, collapsed view on two systems and
KK> their average performance on the tasks provided. The second, simpler
KK> analysis would be a confounded version of the factorial analysis. I would be
KK> served well with the simpler analysis if there would a strong effect (which
KK> I expect to happen).
KK> What do you think about that?
Unless you have sound "a priori" reasons to believe that an
interaction effect is not to be expected, then your model should take
it into account. Do you know that statistical joke that says that if 2
people find a chicken, one of them eats it while the other watches,
then statistics says that everyone ate half a chicken? An average of
heterogeneous effects is not really representative, in my opinion.
--
Best regards,
Marta mailto:biostatistics@terra.es
