Date: Mon, 18 Aug 2008 14:55:58 -0700
Reply-To: Reeza <fkhurshed@HOTMAIL.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: Reeza <fkhurshed@HOTMAIL.COM>
Subject: Re: File I/O error--large dataset problems?
Content-Type: text/plain; charset=ISO-8859-1
On Aug 18, 10:50 am, "alan.ill...@gmail.com" <alan.ill...@gmail.com>
> I'm analyzing data with about 235 million rows & 31 variables. It's
> about 55 GB.
> I have it stored on a 64-bit server with 1.6 TB free but SAS itself is
> running a different, local machine. The client has the drive maped.
> I'm starting at the outset with a very simple filter, designed to
> reduce the size of it to something more manageable:
> *lgwork is a library referring to the server;
> data lgwork.omadv;
> set lgwork.om_orig;
> if exdate - date <= 60 and year >= 2005;
> If I take out the 'if' statement and just copy the data through a data
> step, SAS handles it fine. The temporary .lck file keeps going up and
> information is copied. When I use this filter though, the .lck file
> sits at 1 kb and a temporary, empty dataset called lgwork.omadv is
> This procedure was run overnight in a batch process & I got a File I/O
> error in the morning.
> Does anyone know what the problem might be? I'm running SAS 9.1.3 SP 4
> on windows XP.
> Any help or advice you can offer is greatly appreciated! Thanks!
You may want to try a where condition as well, instead of if, I think
its faster. Not sure with the calculation included though.