Date:         Tue, 9 Oct 2001 12:47:38 -0700
Reply-To:     "Karsten M. Self" <kmself@IX.NETCOM.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         "Karsten M. Self" <kmself@IX.NETCOM.COM>
Subject:      Re: Sort a 8GB data set
In-Reply-To:  <OF3862AA0B.5D33B56E-ON88256AE0.00653208@rtp.epa.gov>; from
              Cassell.David@EPAMAIL.EPA.GOV on Tue, Oct 09,
              2001 at 11:36:32AM -0700
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature";

on Tue, Oct 09, 2001 at 11:36:32AM -0700, David L. Cassell (Cassell.David@EPAMAIL.EPA.GOV) wrote:
> Rich wrote:
> > I am working with a 8GB dataset under SAS 8.0 on a Windows NT under
> > NTFS sytem. I need to sort the data set according to 2 variables. I
> > have 22GB free space in the hard drive. During the sorting, a
> > temporary system utility file with size 18GB is created. Then the
> > process stopped because of "OUT OF RESOURCE".
> >
> > What should I do with it? Thanks a lot.
> 
> First of all, don't sort it unless you really have to.

Strong second to this.

> So, part of the answer is another question: Why are you sorting this
> data set, and what will you have to do with it afterward?  The answers
> will dictate your best strategy.

Several years back, we had a similar problem posted, for a mainframe
user.  The processing consisted of an initial sort, some dataset
manipulations, summarizations, and a report.  Loop truncation was also
utilized to good effect (e.g.:  15 element arrays were searched, but a
null value indicated the remaining elements were also empty, allowing
early exit).

This was reordered as:

   - A subsetting SAS view w/ KEEP statements, utilizing NOTSORTED, and
     outputting summarized data on LAST. processing.
   - A sort of the output.
   - A summarization of the sorted output.

The overall speed improvement was about ninefold.

Sorts should, in general, be avoided, and deferred until substantial
data reduction have occured.

Post details of your processing for further assistance.

Peace.

-- 
Karsten M. Self <kmself@ix.netcom.com>        http://kmself.home.netcom.com/
 What part of "Gestalt" don't you understand?              Home of the brave
  http://gestalt-system.sourceforge.net/                    Land of the free
   Free Dmitry! Boycott Adobe! Repeal the DMCA!  http://www.freesklyarov.org
Geek for Hire                      http://kmself.home.netcom.com/resume.html

[application/pgp-signature]