Date: Wed, 10 Oct 2001 03:22:00 GMT
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Bruce Erlichman <berlichman@NYC.RR.COM>
Organization: B. Erlichman, Inc.
Subject: Re: Sort a 8GB data set
Content-Type: text/plain; charset=us-ascii
Divide the dataset into smaller pieces: a b c ... Then sort
each of these individually. Then combine: set a b c ...This set
statement will not append, it will take the sort into account.
On 9 Oct 01 18:36:32 GMT, Cassell.David@EPAMAIL.EPA.GOV (David L.
>> I am working with a 8GB dataset under SAS 8.0 on a Windows NT under
>> NTFS sytem. I need to sort the data set according to 2 variables. I
>> have 22GB free space in the hard drive. During the sorting, a
>> temporary system utility file with size 18GB is created. Then the
>> process stopped because of "OUT OF RESOURCE".
>> What should I do with it? Thanks a lot.
>First of all, don't sort it unless you really have to.
>Second, for a standard PROC SORT, SAS needs several times the size of the
>original dataset for its scratch space. So you can see why you ran out of
>resources. You can avoid this problem in several ways.
>You may find that the TAGSORT option in PROC SORT will solve your problem
>for you, with minimal coding [just add the word TAGSORT in the PROC SORT
>statement]. This will take longer, but will use less space. It's a
>If you'll need to access the entire dataset multiple times afterward, this
>may be your best bet. Also, the fatter your dataset [moving more toward a
>very wide record with a smaller number of records], the better TAGSORT will
>look. A very tall, thin dataset will not benefit from TAGSORT as much.
>Or you may find that using the indexing capability of PROC DATASETS is your
>best option. This is typically a lot faster than a full sort. If you will
>need several different sort orders, you can create all of them in the same
>PROC DATASETS call. Furthermore, if you will then need to access only
>subsets of the data, you may get a big win here. OTOH, if you will want to
>access the full dataset many times once you are done with this part of the
>process, you will find a serious slowdown. SAS is far faster at sequential
>access than at indexed access - it is a matter of the I/O off the hard
>So, part of the answer is another question: Why are you sorting this data
>and what will you have to do with it afterward? The answers will dictate