Hi Joe,

You certainly could do the traditional process of elimination to help locate where the problem is for sure. i.e. Most PC's nowadays have multi-Gig harddrives. If I write 40M obs with 3vars to a local drive it completes in about 1m13s. Your incoming dataset buffer and your datastep processing may be more involved, but you certainly could do a test and repoint your program to output to your local drive. If, indeed, it runs faster, then you have determined that the slow-down is remote from your local box, which would confirm that your processing is ruled out of the equation of contributing factors.

You could also reconfirm the SAN issues by writing a very stripped down test with the volume of records necessary and see if the with/without SAN shows the difference.

Hope this is helpful.

Mark Terjeson Investment Business Intelligence Investment Management & Research Russell Investments 253-439-2367

Russell Global Leaders in Multi-Manager Investing

-----Original Message----- From: SAS(r) Discussion [mailto:SAS-L@LISTSERV.UGA.EDU] On Behalf Of Nordlund, Dan (DSHS/RDA) Sent: Friday, January 29, 2010 9:48 AM To: SAS-L@LISTSERV.UGA.EDU Subject: Re: puzzling SAS I/O question

Yes it is on a Windows Server and the files are on an attached high speed storage area network. The usual explanation for slow processing that I see is in fact I/O contention with other large jobs that are running concurrently. But I have never experienced this magnitude of slow down.

Dan

Daniel J. Nordlund Washington State Department of Social and Health Services Planning, Performance, and Accountability Research and Data Analysis Division Olympia, WA 98504-5204

From: Joe Matise [mailto:snoopy369@gmail.com] Sent: Friday, January 29, 2010 9:32 AM To: Nordlund, Dan (DSHS/RDA) Cc: SAS-L@listserv.uga.edu Subject: Re: puzzling SAS I/O question

Is it on a server? If so I'd blame the server, probably has use issues (more people hitting it now). Or perhaps you have (nearly) run out of storage space and it's scrambling to find space for the last bit? Can you find the temporary work location and see how big the file is that's being created? Should give you an idea of how far it is, at least.

I can't think of a sas i/o reason but I'm not an expert on that side either.

-Joe

On Fri, Jan 29, 2010 at 11:26 AM, Nordlund, Dan (DSHS/RDA) <NordlDJ@dshs.wa.gov> wrote:

I have a 12 GB (SAS compressed) SAS file that I need to process and add some computed variables to it. The resulting file is 15 GB (SAS compressed, 40 million) records). For a variety of reasons, I had to re-run the process. Both times the job finished in approximately 1/2 hour.

Then I wanted to read just 3 variables (about 150 bytes per record) from the 15 GB file using a simple data step.

data svclib. service_codes (compress=yes); set svclib.svc_span_transactions(keep=service_code tie_breaker source_system_id); run;

The resulting file will be about 1 GB (SAS compressed). The job has been running for nearly 3 hours and hasn't finished yet.

Does anyone have any ideas about what might be going on? It seems to me that reading 3 variables from the 15 GB file shouldn't take 6+ times longer than creating the file in the first place. I am about to contact my IT people to see about disk problems, I/O contention, etc. but wanted to verify that there is no SAS file I/O reasons for the above results. I would be happy to provide any other info that might be helpful.

Puzzled near Seattle,

Dan

Daniel J. Nordlund Washington State Department of Social and Health Services Planning, Performance, and Accountability Research and Data Analysis Division Olympia, WA 98504-5204