Date: Sat, 9 Jul 2005 12:57:47 -0400
Reply-To: Sigurd Hermansen <HERMANS1@WESTAT.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Sigurd Hermansen <HERMANS1@WESTAT.COM>
Subject: Re: SAS 9.1 XP not using RAM
Content-Type: text/plain; charset="iso-8859-1"
MC:
Assuming that David's right (as usual) and you have a network data transfer bottleneck, you'll have to reduce the number of bytes being transferred to improve performance. A larger cache on the destination side won't help.
Francis Harvey {http://www.nesug.org/html/Proceedings/nesug04/io/io07.pdf) discusses a number of methods that we used to improve transfers of data over intranets and the Internet. When network transfer really did take forever, we had to resort to careful selection of data items and values, summarization down to the minimally acceptable levels of precision, and file compression methods. We discovered that almost all very large datasets reduce with little or no loss of information to much smaller datasets. Once we made values of selected columns somewhat more granular and summarized data into distinct rows with counts, we would obtain results very close or identical to results that we had obtained from the original dataset.
SAS offers a wide variety of tools that one can use to remove noise and repetition from data. Consider the principles of sufficient statistics when extracting and loading large volumes of data. In much of the data warehousing and data mining work that I have done, voluminous detail of transaction and event databases often washes out in analyses.
Database experts have an annoying habit of ignoring pragmatics in data transfers and storage. All that wasted bandwidth drains resources and reduces information to noise ratios. I enjoy as much as anyone the technical challenge in managing very large volumes of data, but, as in music, when volume of data takes on more than a supporting role, it can distract more than it contributes.
Sig
________________________________
From: owner-sas-l@listserv.uga.edu on behalf of MC
Sent: Fri 7/8/2005 5:06 PM
To: SAS-L@LISTSERV.UGA.EDU; arnold.g@GHC.ORG
Cc: MC
Subject: Re: SAS 9.1 XP not using RAM
Arnold,
Thanks for your reply.
It actually has a gigabit connection running at about 20% utilization.
The server has 4 different drives (not only letters) and the work space is
one of those local drives.
I don't think it is paging as orgininally posted as the disk utilization is
relatively low, the page file utilization is low and as per this from microsoft:
"When you use System Monitor, the values that are returned by the Pages/sec
counter for the Memory performance object may be more than you expect. These
values may not be related to either paging file activity or cache activity.
Instead, these values may be caused by an application that is sequentially
reading a memory-mapped file."
I am thinking that it has to do with the network now.
However, should it be using more RAM with a 20+GB dataset?
This file is not indexed.
Thanks,
MC
On Fri, 8 Jul 2005 12:18:05 -0700, Arnold, Garth <arnold.g@GHC.ORG> wrote:
>Hello MC -
>
>The 32-bit version of WinXP is limited to a gross of 4GB RAM exclusive
>of swap file, ~3.6GB in reality, IIRC. A 64-bit version of WinXP could
>use more, but your app/SAS would need to be 64-bit as well in order to
>benefit.
>
>Neither SAS nor any other app will 'use' the swap file in the way you
>may be thinking - SAS will not read ahead and use large chunks of that
>space as an i/o buffer, although Windows itself may do some of that.
>
>You definitely need to have a swap file in order for Windows to run, so
>don't delete both files. Your current configuration meets Microsoft's
>recommendation for size and allocation (ref MSKB article 308417).
>
>I'd wonder about these issues:
>
> - the speed of your network connection: if you're pulling all of that
>20GB file across a 100Mb connection, it will take several hours to
>complete.
>
> - indexing on the file: if you're pulling a subset of the file across
>the network, are you taking advantage of an indexed file?
>
> - use of disk on the WinXP workstation: if you're creating a 20GB file
>locally (or a large subset of 20GB), do you have SAS work allocated to
>one drive and SAS permanent storage on an/the other drive? Ideally, you
>have even more drives (physical drives, not just drive letter volumes of
>the same physical drive) that you can use to additionally isolate the
>Windows swap file and even the \windows directory.
>
> - file fragmentation on either the server or workstation end.
>Defragmentation software can provide significant improvements in disk
>i/o speed.
>
>My $.02,
>
>Garth
>
>****************************************
>Garth Arnold
>IS Supervisor, CHS - Infrastructure
>Center for Health Studies
>Group Health Cooperative
>Seattle, WA USA
>arnold.g@ghc.org
>****************************************
>
>-----Original Message-----
>From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.VT.EDU] On Behalf Of Phil
>Rack
>Sent: Friday, July 08, 2005 11:30 AM
>To: SAS-L@LISTSERV.VT.EDU
>Subject: Re: [SAS-L] SAS 9.1 XP not using RAM
>
>
>My understanding of XP is that it only sees 4GB of RAM. That includes
>system RAM and Page Files. Try turning off your page file and reboot
>and see if this handles your problem.
>
>
>Philip Rack
>
>MineQuest, LLC
>
>SAS Consulting and Contract Programming
>
>1939 Queensbridge Dr.
>
>Columbus, OH 43235
>
>Tel: (614) 457-3714
>
>Fax: (614) 737-3419
>
>
>
>HYPERLINK "http://www.MineQuest.com"http://www.MineQuest.com
>
>
>
>
>
>
>
>-----Original Message-----
>
>From: MC [HYPERLINK
>"mailto:supermario@MAC.COM"mailto:supermario@MAC.COM]
>
>Sent: Friday, July 08, 2005 2:00 PM
>
>To: SAS-L@LISTSERV.UGA.EDU
>
>Subject: SAS 9.1 XP not using RAM
>
>
>
>I was wondering if anyone has seen this before or could give me some
>ideas on how to fix it.
>
>
>
>The server has:
>
>Dual Xeon at 2.6GHZ
>
>4GB of RAM
>
>
>
>1 page file on the C drive 2GB
>
>1 page file on the E drive 4GB
>
>
>
>Running XP SP1.
>
>
>
>We are pulling a 20GB dataset from a network location using a data step.
>
>
>
>SAS, instead of using RAM, is doing everything by paging. No RAM is
>being used except for system stuff (by looking at the Task Manager, only
>252MB is being used).
>
>
>
>Looking at the "performance" app, the pc is doing about 2920 pages/sec.
>
>
>
>Available Mbyes is 3253MB.
>
>
>
>% usage of page file on C drive is 0.7
>
>% usage of page file on e drive is 0.4
>
>
>
>Any ideas what needs to be changed? Any options?
>
>
>
>The dat set page size is 16384
>
>number of data set pages is 1041784
>
>Observations is 8334252
>
>Variables is 294
>
>Observation length is 1824.
>
>
>
>I have tried setting the buffsize to 81920 and bufno to 10 but nothing
>seems to change in the performance.
>
>
>
>Thanks in advance,
>
>MC
>
>
>
>
>
>
>
>
>--
>No virus found in this outgoing message.
>Checked by AVG Anti-Virus.
>Version: 7.0.323 / Virus Database: 267.8.10/43 - Release Date: 7/6/2005
| 
