Date: Mon, 19 Feb 2007 14:59:33 -0500
Reply-To: Sigurd Hermansen <HERMANS1@WESTAT.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Sigurd Hermansen <HERMANS1@WESTAT.COM>
Subject: Re: Credit Bureaus--Legal Liability and Model Validation
In-Reply-To: <5.1.0.14.2.20070219151608.07aa0060@pop3.powernet.co.uk>
Content-Type: text/plain; charset="us-ascii"
>Perhaps the most important thing to say is that these statistical
models are not (as seems to be being suggested) generally based on
hypothesis but, rather, on historical data. If (perhaps by the
mechanism I've described) past experiences are that a sudden change in
credit information is of more concern than a long-term sub-optimal
credit history, then the history-based models will reflect that
experience.
>
A time series amounts to a sample drawn from part of a targeted
population. That part of the population that lies in the future may have
a different distribution of outcomes than the historical part. As Peter
Whittle wrote over forty years ago in his pithy book, Prediction and
Regulation, predictive models are trying to forecast "... a probability
distribution of future values, conditioned by the knowledge of past
values." Even if the relations between predictors and outcomes remain
"stable to specification", that is, relatively constant, a time series
models have to include predictions of predictors as well as estimates of
model parameters. Moreover, and all too critical in this situation,
close fit of a predictive model to historical data does not mean that a
predictive model will hold up well in the future, any more than a day of
smooth sailing through uncharted waters in a dense fog means that a ship
won't hit an iceberg during the next hour. A theoretically good model
may predict very poorly during a short interval, and a bad model may
predict the past exceptionally well.
>I therefore think that the short answer to the sort of questions you
keep asking is this: if a reputable credit risk assessment model says
that a certain (whatever) feature of credit history warrants a
substantial change in credit rating then, yes, that probably IS
justified - however surprised you, I or anyone else may be by that. Of
course, it's only justified _statistically_ (i.e. probabilistically),
and therefore may not be correct in terms of any given individual, but
'statistical' is, I'm afraid, the name of the game.
>
Perhaps we have a better picture of predictive models in the credit
industry if we stand your justification on its head. An unexpectedly
adverse observation in a series of favorable observations suggests that
the model does not have enough information about a credit applicant for
a good prediction of future payments. That new information makes the
individual look far less predictable than before and greatly reduces the
confidence of the lender in the applicant's ability and willingness to
pay his or her debts. The original model may predict future behavior of
the applicant reasonably well, or not. (Say the applicant actually has a
99% chance of paying his or her debts fully and on time, and the model
predicts 99% of individuals in the applicants class of borrowers will
repay, while a substantial group of other applicants actually have only
a 90% chance of repayment.) New found uncertainty about whether the
applicant fits the 99% repayment class as determined by the model may
decrease a lender's confidence in the applicant. That alone could
justify a lower credit rating. A predictive model does not have to be
valid so long as lenders have confidence in it. In essence, the credit
scoring model may become invalid for an applicant.
S
-----Original Message-----
From: owner-sas-l@listserv.uga.edu [mailto:owner-sas-l@listserv.uga.edu]
On Behalf Of John Whittington
Sent: Monday, February 19, 2007 10:28 AM
To: Tom White; SAS-L@LISTSERV.UGA.EDU
Subject: Re: Credit Bureaus--Legal Liability and Model Validation
At 10:01 19/02/07 -0500, Tom White wrote (in small part):
>We are talking now about Credit Bureau scores which presumably assess
>customer's behavior and likelihood of default. If a person has a
>stellar past behavior in terms of paying bills, why would this one bad
>thing cause such a precipitous drop to one's score from say, 720 to 610
>or so?
At risk of being accused of wasting SAS-L bandwidth, and certainly
without any expert knowledge in this area, I can certainly lend my
support to those who have expressed the view that recent information
should, at least in some situations, be given considerable weighting in
credit-scoring models.
People with persistently poor credit ratings are probably relatively
rare - and, in any event, are easy to identify, with or without
statistical models. I would suggest that, particularly in recent times,
a far more common picture is for someone to have a long history of a
good credit rating and then, at some point in time, and for whatever
reason, run into financial difficulties and hence suddenly, maybe for
the first time, develop an unfavourable entry on his credit rating. If
I were a potential lender, I think I'd probably be more concerned about
someone who had just exhibit his/her first 'credit problem', after years
of having a pristine record, than I would be concerned about someone who
had more-or-less successfully 'muddled along' with a mediocre credit
rating for some considerable time - on the basis that the recent change
could herald the start of a 'downward slide'.
Perhaps the most important thing to say is that these statistical models
are not (as seems to be being suggested) generally based on hypothesis
but, rather, on historical data. If (perhaps by the mechanism I've
described) past experiences are that a sudden change in credit
information is of more concern than a long-term sub-optimal credit
history, then the history-based models will reflect that experience.
... my few, rather 'OT', thoughts!
Kind Regards,
John
----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: John.W@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------
| 
