Date:         Thu, 31 Aug 2006 22:35:31 -0700
Reply-To:     David L Cassell <davidlcassell@MSN.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         David L Cassell <davidlcassell@MSN.COM>
Subject:      Re: Slow Data Operations, is SAS using all Memory?
In-Reply-To:  <200608311405.k7VE3VpT019516@mailgw.cc.uga.edu>
Content-Type: text/plain; format=flowed

frotty22@YAHOO.COM wrote back:
>
> > frotty22@YAHOO.COM wrote:
> > >
> > >Hi,
> > >
> > >I have Windows XP SP2 PC equipped with 2.5 GB RAM.
> > >But when I look at the Task Manager Performance window while SAS is
> > >working on data operations, it shows that there is still half of the
> > >physical memory available but the page file size increases and the
> > >harddrive LED on the computer shows the drive working all the time.
> > >After finishing the operation, the CPU time is always only a fraction
> > >of the overall time.
> > >
> > >Is it possible that SAS is not using all of the physical RAM? Do I have
> > >to set some options for that?
> > >
> > >Thanks for any hints!
> >
> > [1]  Your OS will *never* let SAS have all the physical RAM.
> > Even Windows is too smart for that.  You have to have lots of
> > RAM available for all the other tasks your OS has to do, and you
> > need to have RAM available for other processes.
> >
> > [2]  Your RAM is not the problem.  You say:
> >
> > >After finishing the operation, the CPU time is always only a fraction
> > >of the overall time.
> >
> > This is almost always an indication that your disk I/O is the
> > real bottleneck.  Your RAM sits around, and your CPU chip sits
> > idle a lot of the time, while huge amounts of disk I/O get
> > done.  You do not say what you are doing, but you may not
> > be using the features of SAS as efficiently as you could.
> >
> > [3]  There are things you can do to tune your system.  But it
> > sounds so far like you need to tune your code a lot first, so
> > I would not even bother going in that direction.  Focus on
> > what you are doing that makes your disk drive work so darn hard
> > that your CPU ends up sitting around taking a nap.
> >
> > HTH,
> > David
> > --
> > David L. Cassell
> > mathematical statistician
> > Design Pathways
> > 3115 NW Norwood Pl.
> > Corvallis OR 97330
> >
> > _________________________________________________________________
> > Call friends with PC-to-PC calling -- FREE
> >
>
>Thanks for you answers!
>
>I still think it is a bit strange that from 2.5 GB there is half of it
>sitting idle. SAS itself uses only a bit more than 100 MB. If the
>system would require 1.25 GB for other stuff, why did my computer work
>before the upgrade when it only had .5GB ?
>
>I am aware that joins with databases of 1.5 or more GB are likely to
>need some virtual memory. But the swapfile is too small during
>processing. It seems like SAS is directly working on the harddrive
>without trying to use the memory.
>
>Thanks again,
>   Bob

I think that the problem is I/O rather than memory usage.

Since I don't know what code you are using to do this large
join, I'll have to speculate that you are (either implicitly or
explicitly) doing a sort-sort-merge, and probably doing it in
PROC SQL.  So you are suffering from massive sort-o-mania.
This is often best addressed by re-design of the process.
It may be that you would do better designing this as a hash,
or perhaps avoiding the current process flow completely.
But I can't tell that from way over here.

If you really believe that you need to allocate more RAM,
then look into the REALMEMSIZE system option.  Be careful
with it.  Don't allocate more RAM than will actually be available
to SAS at invocation, or you'll end up causing performance
problems with thrashing of your virtual memory.

Once you have that, you can muck around with the
SORTSIZE system option.  Perhaps a simple -SORTSIZE MAX
given your *careful* choice of REALMEMSIZE .

You can mess with MEMSIZE too, but optimal choices
for that vary according to your OS, so be careful.  How
you assess a good choice for MEMSIZE is dependent on
your available system monitoring tools.

There are other things you can do to mess up your SAS
system (:-) but I always recommend coding over system
options, unless the system options have been messed up
already and need to be fixed.

HTH,
David
--
David L. Cassell
mathematical statistician
Design Pathways
3115 NW Norwood Pl.
Corvallis OR 97330

_________________________________________________________________
All-in-one security and maintenance for your PC.  Get a free 90-day trial!
http://www.windowsonecare.com/trial.aspx?sc_cid=msn_hotmail