Dan posted the following clarification:
> Yes it is on a Windows Server and the files are on an attached high
> speed storage area network. The usual explanation for slow processing
> that I see is in fact I/O contention with other large jobs that are
> running concurrently. But I have never experienced this magnitude of
> slow down.
Dan, I am going to jump onto the dogpile of responses from Joe, Art, and Mark and also opine that it is a network I/O issue. Arf!
As you know, one of the unsung benefits one gets by compressing SAS data sets is fewer overall I/O's transferring data between disk storage and computer memory due to SAS transferring more observations per I/O. So, compression results in fewer overall I/O's when performing a sequential read of the compressed SAS data set than if the same data set were uncompressed. Fewer I/O's result in fewer seconds, minutes, hours, days, spent waiting for a program to run. Since your I/O times went precipitously upward doing simple DATA step tasks, it is unlikely that SAS's uncompressing of incoming observations and compressing outgoing observations is the culprit. It is more in line with the type of server I/O delays that all of us have experienced at one time or another. That is, the trip to and from the server was during a computer rush hour.
You know, your systems folks should be able to run a monitor on the server and been able to tell you that there was heavy I/O activity, or some other marker of poor performance. Hopefully, they are approachable enough and open enough to share that information with a SAS programmer!
Dan, best of luck in all of your SAS endeavors!
I hope that this suggestion proves helpful now, and in the future!
Of course, all of these opinions and insights are my own, and do not reflect those of my organization or my associates. All SAS code and/or methodologies specified in this posting are for illustrative purposes only and no warranty is stated or implied as to their accuracy or applicability. People deciding to use information in this posting do so at their own risk.
Michael A. Raithel
"The man who wrote the book on performance"
Author: Tuning SAS Applications in the MVS Environment
Author: Tuning SAS Applications in the OS/390 and z/OS Environments, Second Edition
Author: The Complete Guide to SAS Indexes
Never insult an alligator until after you have crossed the river. - Cordell Hull