Date:         Wed, 15 May 2002 15:12:06 +0200
Reply-To:     Ulrike Grömping <ugroempi@FORD.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         Ulrike Grömping <ugroempi@FORD.COM>
Organization: Ford Motor Company
Subject:      Re: storage space problems
Content-Type: text/plain; charset=iso-8859-1

Roger,

thanks. This clarifies it. I was thinking that the second file of the
same name would be generated in the work library, but that seems not to
be the case.

By the way, watching the files grow doesn't work in my Windows
environment, which was what confused me. I can watch the space on the
drive shrink, but I cannot find a file that grows (utility files in the
work directory do grow during processing, but apparently not the SAS
datasets).

Regards, Ulrike

Roger Lustig wrote:
>
> Ulrike:
>
> If you write out a file with the same name as your source file, that
> means you need space equal to the source file plus the output file.
> Why?  Because SAS won't delete the source file and rename the output
> file until the very end of the proc or data step.   Until then, the
> output file will have a different, temporary name.  (In Windows, open up
> an explorer window and watch the relevant folders as the program runs!)
>
> In a DATA step merge, you're unlikely to use the WORK library to a
> significant extent.
>
> In a SQL merge, you'll be sorting all the files you join, and will thus
> need something on the order of 3x what you include from the source
> file(s).  In general, expect to use 3x the file size when sorting, and
> expect to use it in the WORK library.
>
> Oh, and by the way: IDE/ATA storage space now costs on the order of
> $2/GB.  If you're running Windows, have your boss spring for a couple of
> 60GB drives, one for WORK, one for keepers.  And then design your
> programs to use multiple drives at once, minimizing the steps that
> involve reading from, and writing to, the same drive at once.  If your
> source files are on one drive, and your output on another, you generally
> save time.  If you're merging a big file to a small one, it may actually
> be economical to add a step to copy the smaller file to a drive other
> than the output drive.
>
> Best,
>
> Roger
>
> Ulrike Grömping wrote:
> >
> > Dear all,
> >
> > I am confused about SAS' usage of storage: When do I need space in the
> > work library, and when do I need it in the permanent library ?
> >
> >  I will have to handle some very large data sets, and have just started
> > with a roughly 800MB/3.5Mio observation dataset (PC). I have two
> > partitions on my computer, and had assigned the permanent library to the
> > partition with less free space (D:), assuming that I would need just
> > enough space for accomodating the data set.
> >
> > However, even though I could not find a physical file causing the
> > problem, a data step with a merge, that did not change the size of the
> > permanent data set, resulted drive D: running out of space. When I
> > reallocated the permanent library to the larger partition, all went
> > well. Does anybody know, how SAS handles space in the data step, i.e.:
> >
> > - When does it use the work library ?
> > - When does it use the permanent library instead?
> > - Where does it write onto the disk, if the disk gets fuller and fuller,
> > while I can't find the file that causes this ?
> >
> > Regards, Ulrike