Date: Fri, 23 Feb 2001 07:42:46 +1300
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
Organization: Internet ProLink NZ Limited, New Zealand
Subject: Re: PROC SORT - working files vanish
Content-Type: text/plain; charset=ISO-8859-1
Fri, 23 Feb 2001 07:12:11 +1300 (NZST)
01h15m44s ago ...
On Thu, 22 Feb 2001 16:56:27 +0000 (GMT), Smith, Curtis, Mr, DCAA wrote:
> Are you sorting the SAS file with or without an out=?
With an out= to a file on a non-WORK directory. I can watch both the
WORK and this non-WORK directory durng the 30 mins or so of sorting.
Nothing appears in the out= until the very last stage when the file is
created at a steady rate. But for some input files 2 or three apparent
passes are made and the intermediate files seem to vanish.
> Can you show us the code and sort options you are using?
Sure I can - it is at work. Basically just PROC SORT then BY then RUN -
3 lines. I tried the TAGSORT option but that took 5 times as long.
What do you mean by sort options - options in the SAS code or in the
Would something here possibly explain why I can't easily keep track of
resource usage - i.e. those files seem to vanish? They can't be being
copied anywhere as it all happens too quickly.
I think what I am missing here is unix knowledge - (of which I have
very little) - can a file be hidden easily from "ls" for example? Maybe
this is a severe form of "locking"? The SORT itself works fine.
> -----Original Message-----
> From: th@IPROLINK.CO.NZ [mailto:th@IPROLINK.CO.NZ]
> Sent: Wednesday, February 21, 2001 8:20 PM
> To: SAS-L@LISTSERV.UGA.EDU
> Subject: PROC SORT - working files vanish
> I am sorting large SAS datasets using PROC SORT. Each is 3 to 4
> Gigabytes in size. PROC SORT takes 15-30 mins. This is fine... but I am
> just confused by the following:
> Keeping an eye on disk space usage, I watch the temporary intermediate
> sort file build up and then vanish *after* the output file is written.
> In my last 2 tests though PROC SORT seems to have switched to a
> different algorithm requiring more than one *pass* - I see it create a
> 3-4 Gig intermediate file and then suddenly "poof" it vanishes - and so
> does the source dataset and there is no output dataset visible at that
> stage either (I store the output in a non-work temporary directory -
> everything else is in the SAS "work" directory).
> Then I see a second pass build up another big file which vanishes - the
> last run required 4 passes and the last pass finished with a small
> output file -- but then I saw the true output sorted dataset being
> faithfully written - but nowhere could I find the work file it was
> working from!!!!
> SAS was running on a unix machine. I was checking files using ftp under
> "Marathon" and also, unable to believe my eyes I tried "ls" on the
> Is there some way that SAS can "hide" these intermediate files from
> view? That's my only question actually!!! It could be something I don't
> understand about unix file structure.
> Any ideas or pointers much appreciated.