Date: Mon, 8 May 2006 15:31:58 -0400
Reply-To: Peter Flom <Flom@NDRI.ORG>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Peter Flom <Flom@NDRI.ORG>
Subject: Re: comparing linear vs nonlinear model
Content-Type: text/plain; charset=US-ASCII
>>> "Sigurd Hermansen" <HERMANS1@WESTAT.com> 5/8/2006 3:13 pm >>> wrote
I wonder if loss functions should be incorporated into a model choice
process, or should loss estimates drive choices among alternative,
If you know enough about the costs of errors to make such functions, then I think this makes sense; indeed, the decision theoretic approach has much to recommend it, I think. But coming up with good numeric approximations can be hard, again depending on substantive area. I think this would be easiest where all the costs and returns are based on money.
But in my example (the ordinal logistic regression of drug use) we weren't going to use that model for prediction. What is the cost of a wrong explanation? This seems intractable. And what is the cost of a more complicated explanation? Although df is an obvious choice here, I am not sure it is a true measure of cost of explanation. Some variables are easy to include and others are not.
This once again gets into what Robert Abelson talks about in Statistics as Principled Argument (a GREAT book): The MAGIC criteria.
So, e.g., if you found that hair color was related to drug use, you would have a lot to do with regard to both interestingness and credibility. Degrees of freedom doesn't capture this.
And THIS gets back to my point: Trying to use formal methods to make these decisions pushes the thought-process from the analyst to the computer program: But computers are stupid. And SAS, as great as it is, has no idea what the variables you put into a model mean.
Looked at the other way, it may be hard to explain leaving a variable OUT of a model, regardless of effect size, p, or anything else, if that variable has a lot of research backing it up. If you found, for instance, that sex was not related to height among adults, you would have some explaining to do.