Date:         Wed, 31 Jan 1996 09:20:44 -0500
Reply-To:     Phil Gallagher <pgallagh@MAIL.CATO.COM>
Sender:       "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From:         Phil Gallagher <pgallagh@MAIL.CATO.COM>
Organization: Cato Research Ltd.
Subject:      Unexpected answers to survey questions
In-Reply-To:  <C30D0D6102D2B4D1>

Dear gurus-in-the-trenches,
 
   I hope you can help me compile a compendium of examples of (totally)
unexpected answers to survey questions - and who better than those of us
grunts in the data management and analysis trenches who labor trying to
clean up after the mistakes of our "betters" who design the questions?
:-)
 
    I plan to give a talk in which one of the messages will be "Be
absolutely certain to do an honest pre-test of your survey instrument
with subjects similar to those who will participate in the primary study"
so that you can identify most of the questions whose meanings will not be
interpreted the way your high-faluting graduate student buddies would
interpret them and whose answers may well include categories that none of
the aforementioned graduate students would ever have thought of.  (I was
once asked to help with a study that asked respondents what prohibited
drugs they used and how they used them - you would not believe and I
ain't a-gonna tell you some of the things they used and how they got them
into their bodies!  but we didn't know that until the instrument was used
in the study communities.)
 
    One of the classic examples from the Forties was on the Standard
Achievement tests given every year in each school grade (at least in
Florida).  On the front page you filled in your demographic information;
there was generalized hilarity when one fourth grader (approx. 9 years
old) filled in the blank for SEX where "Boy" or "Girl" was expected with
"Yes".  (In those simpler days I doubt that anyone who laughed at this
was even thinking of the interpretation you were just giggling about - we
just thought it was funny that the kid was so thick that he didn't
realize that he should have said "Boy" - how times have changed!)
 
    Anyway, here is what I hope will happen:  some of you will send
examples of questions with "off-the-wall" answers, suitably disguised so
that confidentiality is preserved, with a note saying that it is OK for
SAS-L readers to reproduce and even (!) publish (!) them (fat chance,
right?).  I will compile them into one not-too-long message, removing the
names of the contributors to a single acknowledgment, with the intent of
further preserving confidentiality.  Then I'll post it back to SAS-L and
use it in my talk.  :-)
 
    It wouldn't hurt a bit if some of you were able to include clever
solutions to the illustrated problems, right?  I guess it would be
overkill to include statements (even if true!) to the effect of "Because
of the problems consequent to the unreliability of the answers to this
important question, this $16.3 million study was abandoned in the
analysis phase."  (If any students from the U of Michigan-U of Maryland
Joint Program on Survey Methodology (JPSM) contribute, I will try to
remember to send them red, blue, or gold stars to paste upon their
foreheads.)  (They may have to be Virtual Red, Blue, or Gold Stars if the
sending is by email, but the thought is there.)  Please respond before 3
February, if possible.
 
                                    Phil Gallagher
 
                                    pgallaghe@mail.cato.com
 
                                    pngallagher@sopnia.sph.unc.edu