Date: Thu, 24 Aug 2006 22:40:10 -0700
Reply-To: David L Cassell <davidlcassell@MSN.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: David L Cassell <davidlcassell@MSN.COM>
Subject: Re: Data Step Views and Memory limitations
In-Reply-To: <200608242307.k7OMV7U1025562@mailgw.cc.uga.edu>
Content-Type: text/plain; format=flowed
abe_anon@HOTMAIL.COM wrote:
>Scenario#1 (problematic): A SAS Dataset is SET within a Data Step and
>processing involving the use of two very large (memory-intensive) formats
>for table look-up is applied. The Data Step gets a run-time error: not
>enough memory.
>
>Scenario #2 (a solution?): A View is defined, consisting of SETting the
>same
>input SAS dataset as in Scenario#1 and then applying one of the
>memory-intensive format lookups. This View is then SET within a Data Step
>and the second memory-intensive format is applied. This works; there is no
>problem with memory.
>
>Question: Since no memory problem occurred in the second scenario, it
>appears that the memory space for the View and the memory space for the
>Data
>Step are independent of each other. This suggets that the Data Step runs in
>a separate process from the View, and each process has their own memory
>space. Does anyone know if this is indeed how SAS operates "behind the
>scenes"? Is this true for all pl;atforms or just some? Specifically:IBM
>mainframe MVS or Z/OS vs Unix vs Windows.
>
>Or was the successful run in Scenario#2 just a fluke, and this method of
>using a View to get around memory limitations will not really work?
>
>Thanks for whatever insight you can bring,
>Abe
I don't think there is any way of telling whether this is a fluke or not,
without a lot more testing on your part.
However, if your formats are so large that they are consuming your RAM,
then they are also probably too large to be efficient. Once you get past
about 32,000 unique values in your format, it is losing efficiency when
compared to alternatives.
So wouldn't your process be happier if you used something else in this
situation? A nice hash or two might be just what you need to get
around the problem you found.
HTH,
David
--
David L. Cassell
mathematical statistician
Design Pathways
3115 NW Norwood Pl.
Corvallis OR 97330
_________________________________________________________________
Call friends with PC-to-PC calling -- FREE
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline
