Date:         Fri, 27 Sep 2002 16:00:32 +0100
Reply-To:     "Femminella, Oliver" <Oliver.Femminella@HALIFAXCETELEM.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         "Femminella, Oliver" <Oliver.Femminella@HALIFAXCETELEM.COM>
Subject:      FW: Regression with Class Variables and Stepwise Selection
Content-Type: text/plain; charset="iso-8859-1"

Paige,

If I remember correctly I still have a few papers on neural networks
which discuss collinearity, however I'm sure where I've stored them
(Jerome Friedman at Stanford may have written something in one of his
publications) !!

However, here are two extremely useful links on the state-of-art
neural network developments during the '90s.

Essentially 'regularised' neural networks apply 2nd order regularisation
(smoothing),
whereas ridge regression is termed 0th order regularisation (smoothing).

A single value of global smoothing parameter may be suboptimal for a number
variables in a model.
In ridge regression one can apply a separate smoothing parameter for
each variable. However the model remains liner in it's parameters as opposed
to non-linear models.

1).
FAQ on Neural Networks maintained by Warren Sarle:
ftp://ftp.sas.com/pub/neural/FAQ.html

particularly Warren's discussion 'Ill-Conditioning in Neural Networks':

ftp://ftp.sas.com/pub/neural/illcond/illcond.html

2).
You can find excellent papers by David MacKay on
'Bayesian learning for Neural Networks' at
http://www.inference.phy.cam.ac.uk/mackay/BayesNets.html

-----Original Message-----
From: Paige Miller [mailto:paige.miller@KODAK.COM]
Sent: 27 September 2002 15:27
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: Regression with Class Variables and Stepwise Selection


Femminella, Oliver wrote:
> Paige,
>         you mention Ridge Regression...does ANYBODY know
> of (or has) any SAS implementation (macro) of this
> relatively old regression model...even though there
> are more 'practical' modelling approaches used nowadays
> for dealing with multicollienearity - (e.g. neural networks,
> CART, logistic, support vector machines, etc.) -
> (in Ridge Regression one has to choose Lambda, the
> 'smoothing' or 'regularisation' parameter(s)).

I hope your not suggesting that newer 'practical' methods be
automatically preferred to older methods. It would be nice to see
comparisons of the different modelling approaches on different types of
data to see which performs best, but I am aware of only a few such methods.

How does CART or logistic regression or neural networks deal with
multicollinearity any better than other procedures?

Ridge Regression in SAS is in PROC REG. Partial Least Squares is in PROC
PLS. Both specifically deal with multicollinearity.

--
Paige Miller
paige.miller@kodak.com
http://www.kodak.com

"It's nothing until I call it!" -- Bill Klem, NL Umpire
"When you get the choice to sit it out or dance, I hope you dance" --
Lee Ann Womack