Date: Tue, 15 May 2001 03:28:47 -0000
Reply-To: sashole@bellsouth.net
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Paul Dorfman <paul_dorfman@HOTMAIL.COM>
Subject: Re: out of memory error while processing large data
Content-Type: text/plain; format=flowed
>From: Gerhard Hellriegel <ghellrieg@T-ONLINE.DE>
>
>YES!! A lot of problems with memory and SAS V8!! We run it under >OS/390
>and we need up to 5 times more memory as in V6!
Gerhard,
Hmmmm... I thought I've tested many memory-demanding features of V8.1 under
OS/390 and W2000P pretty much beyond the limit intended by the software and
did not find it especially troublesome. I have even reported that V8's
handle on memory usage with character temp arrays has become better than in
V6 by a sizeable measure. 5 times? Wow! I would like to see an instance of
that happening... when it matters. I mean, yes,
data _null_; run;
takes about twice as much memory to run in V8 than in V6, but due to the
brevity of execution, the amount of data processed, and that of total amount
of memory annihilated (< 10 MB) it did not seem overly disturbing.
>Especially with procs like FREQ, SUMMARY, TABULATE, ... Also SQL seems >to
>have some problems.
Again, I have not seen many more problems with these in V8 than in V6, but
maybe I just did not twist them hard enough. Normally, when CLASS procs
start running out of memory, it is a fair indication that the cardinality of
the CLASS variable(s) is too high to be handled by a binary tree, and other
approaches (SORTing/BYing, hashing, etc.) might be knocking on the door.
>I think most of the statistical procs which work with a kind of tables
> >(clusters, memory blocks, whatever) to keep some data in memory will
> >also have problems.
That may very well be true; hard to say without running one. David and/or
Dale would almost certainly report their grievances.
>We have one application with a PROC FREQ which runs in V6.09 with >about
>80M region size and failed in V8 with 400M!
There is a good reason for it, which does not mean, however, that in V8,
FREQ uses much more than in V6. In V6 under MVS, SAS handles memory quite
differently from V8. In particular, the REGION specification in V6 is
irrelevant with MEMSIZE=0 because SAS has a design feature to circumvent the
safeguard imposed by REGION and grab as much memory as it needs. Not
surprisingly, one would not find lots of love for that V6 feature in the
humble camp of MVS system programmers. (They *have* found a way to have Big
Brother cut the job off if it becomes too memory greedy.) In V8, SI have
changed that behavior, and MEMSIZE=0 no longer can go to the sky, but only
up to REGION specification. It seems more logical, and should help alleviate
some understandable animosity between professional SAS programmers and their
friends from the Central Command Unit. All the more that as far as I can
judge by having run the same jobs using V6 and V8 in the same environment,
the dreaded message "NO MKLEs FOUND" is gone for good, without the necessity
to apply a V6 zap.
>We got it to run with 500M!
Sure, but again, for the reasons cited, it means *not* that in V6 and V8,
the same step actually used 80MB and 500MB correspondigly. The log might be
able to attest to it quite eloquently.
>You can refer to the SAS website for a big list with memory problems. >Some
>of them should be fixed with 8.2, some in V9 (???)
Hard to argue with that; but how many of them have happened on the real
computer?
Kind regards,
========================
Paul M. Dorfman
Jacksonville, Fl
========================
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com
