Date:         Thu, 20 Jun 1996 15:00:00 EST
Reply-To:     "Alderton, David" <dla5@CIDHIV1.EM.CDC.GOV>
Sender:       "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From:         "Alderton, David" <dla5@CIDHIV1.EM.CDC.GOV>
Subject:      Re: Scoring Standardized Exams

> Rob Morrison wrote:
> >
> > What is done in the statistical community to deal with this?  Having 46
> missing
> > answers seems different from 46 wrong answers yet it seems they should
both
> > get 0/46.
Traditional test scores are a function of the number correct and the number
wrong, blanks are ignored.  Psychometricians (the people who study the
mathematical properties of tests) have long used (since the 1930s) the
following scoring methodology:
     Score = #correct - #wrong*(1/[k-1]), with k=# forced choice
alternatives
Take a 10 question 5 alternative test.  If a person BLINDLY guesses, she or
he should get 2 questions correct, i.e., 1/5*10 (a 1/5 chance on each item).
 This person's score will be zero using the above formula, since:
     0 = 2 - 8*1/[5-1] = 2 - 8 * 1/4 = 2 - 8 * .25 = 2 - 2
This scoring can obviously lead to negative scores, however, most large norm
based test scores are transformed to another metric (some function of a
cumulative probability metric) so that negative scores are either 1st
percentile, 0 "points", or some positive minimum value.
 
> In the Norwegian Mathematical Olympiad - http://www.math.uio.no/~abelk -
we
> give multiple choice questions in the first two rounds. Here, there are
five
> alternatives, one of which is correct. We score the answers:
>
>    Wrong: 0 points
>    Correct: 5 points
>    Blank: 1 point
Einar Andreas Rodland described the scoring of the Norwegian Mathematical
Olympiad which is simply a variant of the traditional correction of guessing
formulation above.   Take a 10 item 5 alternative test again.  If a person
leaves everything blank she or he gets 10 points.  If a person blindly
guesses on all 10 items, she or he will get 2 items correct at 5 points each
and thus get a score of 10 points which is equivalent to the person who left
them all blank, which is the goal; however, getting 10 points for doing
nothing is as odd as getting negative points for guessing badly.
 
The key theoretical formulation is that you want someone who knows nothing
but blindly guesses to get the same score as someone who knows nothing and
"honestly" answers by not responding!
 
Two additional points related to this.  For those of use who have taken
large normative tests that use these corrections for guessing we have all
been "told" about it in the directions when instructed NOT to guess unless
you can eliminate one or more alternatives.  Wonder why?  Take a 5
alternative question and look at what happens to the guessing probability
based on the number of alternatives you can eliminate...
 
       Correct by
Eliminate   Guessing
0       1/5=.20 -- you break even
1       1/4=.25 -- you win on average
2       1/3=.33
3       1/2=.50
4       1/1=1.0  (know the right answer even if you don't!)
 
Last point.  Empirical studies show that the traditional correction for
guessing penalty (#wrong * 1/[k-1]) is too conservative ... so GUESS!
 
Regards, David.
David L. Alderton, Ph.D.
Centers for Disease Control and Prevention
Division of HIV/AIDS
Surveillance Branch
Atlanta, GA -- DLA5@cidHIV1.em.CDC.gov