Date:         Wed, 21 Jul 2010 11:19:05 +0000
Reply-To:     goladin@gmail.com
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         "goladin@gmail.com" <goladin@GMAIL.COM>
Subject:      Re: In search of a more efficient program
Comments: To: Andy Arnold <awasas@COX.NET>
Content-Type: text/plain; charset="utf-8"

Hi,

Reducing the length of numeric can cause trouble in terms of precision.

Regards,
Murphy Choy
Sent from my Nokia phone
-----Original Message-----
From: Andy Arnold
Sent:  21/07/2010 7:05:30 PM
Subject:  In search of a more efficient program

Background:
I've inherited a large, complex SAS program.  Most files are quite small;
however, some are extremely large and have become a problem.
The files in question have 12-24 fields that are mostly numeric with a few
short (1-4character) fields.  The files that are killing me have 20M records
and one has 160M records.
The files don't use SAS compression because the records are too short to
make compression cost & time effective.

Problem 1:
What are the trade-offs if I force numeric length to 4 or 2?  Does SAS
always use a NUM 8 format internally?  If I force short numeric field
lengths, will SAS have to convert them up to length=8 and back down again in
order to use the data?

Problem 2:
What are the trade-offs if I use an index instead of a sort?  The problem
sort takes an hour, uses a single numeric key with about 10 distinct values,
sorts 20M records that are about 100 bytes long.  I've been successful with
indexing before; in that situation, I needed to sort the file in 15
different sequences.

Thanks for your input and advice.

--Andy