Date: Thu, 5 Apr 2007 15:30:43 +0000
Reply-To: toby dunn <tobydunn@HOTMAIL.COM>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: toby dunn <tobydunn@HOTMAIL.COM>
Subject: Re: Unix Pipe Unzipped Files Through a Data Step
In-Reply-To: <CA8F89971ADA9F47A6C915BA2397844203714059@MAILBE2.westat.com>
Content-Type: text/plain; format=flowed
Sig ,
Thanks to you and every one else.... I needed to pipe the file names in and
select based off a value in the name. Then run that list through a Data
Step to read the dat ainto SAS. Clifford pinted me in the right direction
with Rick Langston's example. I tried something almost identical yesterday
but was a little off. A little reading of Ricksters code and a little
tweeking and I gotter to work like a charm.
Toby Dunn
To sensible men, every day is a day of reckoning. ~John W. Gardner
The important thing is this: To be able at any moment to sacrifice that
which we are for what we could become. ~Charles DuBois
Don't get your knickers in a knot. Nothing is solved and it just makes you
walk funny. ~Kathryn Carpenter
From: Sigurd Hermansen <HERMANS1@WESTAT.COM>
Reply-To: Sigurd Hermansen <HERMANS1@WESTAT.COM>
To: SAS-L@LISTSERV.UGA.EDU
Subject: Re: Unix Pipe Unzipped Files Through a Data Step
Date: Thu, 5 Apr 2007 11:17:06 -0400
Toby:
The excellent SAS archives have an entry from '02:
http://www.listserv.uga.edu/cgi-bin/wa?A2=ind9912D&L=sas-l&P=R7250&D=1&H
=0&O=D&T=1
Also, the zcat command in Unix/Linux accepts a parameter containing a
'wildcard' file name. Whatever zcat generates when executed by a Unix
shell command will pipe thru to a dataset in a FILENAME ... PIPE ....;
statement. Take subsets of your text files, gzip them into a directory,
and then zcat using a wildcard file definition to see the text stream
that it produces. Seems that the INFILE statement has to match up to the
file organization. If I recall correctly, the zip files that we were
streaming into a Data step view had no line breaks.
I found that pipe'd input thru a Data step view and into a SAS SQL query
with a WHERE clause worked much faster than a typical SET input from a
SAS dataset. The fetch rate from disk greatly exceeded fetch rates from
unzipped files, and the WHERE clause acted as a filter on the contents
of a data cache.
S
-----Original Message-----
From: owner-sas-l@listserv.uga.edu [mailto:owner-sas-l@listserv.uga.edu]
On Behalf Of toby dunn
Sent: Wednesday, April 04, 2007 4:37 PM
To: SAS-L@LISTSERV.UGA.EDU
Subject: Unix Pipe Unzipped Files Through a Data Step
I was wondering if anyone else has any thoughts on this.
I have a directory on my Unix Box with a bunch of Ziped txt files. I
wanted to read each of these into one Data set and Id which records came
from which file. The kicker is they are ziped up and I dont want to
unzip all of them and then read them in I want SAS to do this for me.
So how to get SAS to Pipe the File Names into a data step and have it
unzip them (On The Fly) so I can use the FileVar option and the
FileName= option?
Toby Dunn
To sensible men, every day is a day of reckoning. ~John W. Gardner
The important thing is this: To be able at any moment to sacrifice that
which we are for what we could become. ~Charles DuBois
Don't get your knickers in a knot. Nothing is solved and it just makes
you walk funny. ~Kathryn Carpenter
_________________________________________________________________
The average US Credit Score is 675. The cost to see yours: $0 by
Experian.
http://www.freecreditreport.com/pm/default.aspx?sc=660600&bcd=EMAILFOOTE
RAVERAGE
_________________________________________________________________
The average US Credit Score is 675. The cost to see yours: $0 by Experian.
http://www.freecreditreport.com/pm/default.aspx?sc=660600&bcd=EMAILFOOTERAVERAGE
