Date:         Thu, 6 May 1999 10:29:18 -0700
Reply-To:     "Karsten M. Self" <kmself@IX.NETCOM.COM>
Sender:       "SAS(r) Discussion" <SAS-L@UGA.CC.UGA.EDU>
From:         "Karsten M. Self" <kmself@IX.NETCOM.COM>
Organization: Self Analysis
Subject:      Re: reading data from unix pipe
Comments: To: Fanchon Finucane <fanchon.finucane@ABACUS-DIRECT.COM>
Content-Type: text/plain; charset=us-ascii

> Date:    Wed, 5 May 1999 22:36:46 -0600
> From:    Fanchon Finucane <fanchon.finucane@ABACUS-DIRECT.COM>
> Subject: reading data from unix pipe
>
> Hello SAS-L,
>
> We are trying to read data in from a piped file in unix but are having
> problems when we want to read a block at a time rather than 1 observation
> at a time. Right now, a block of data reads in 50 observations (3 vars per
> observation).  &modlname is the piped text.
 
I'm not quite sure what you mean by reading "a block at a time".  SAS is
uses record-oriented input methods, and reads data one record at a time,
delimited by a newline, unless the binary INFILE RCDFMT option is used.
 
If you need to specifically block or unblock data, you can use the Unix
'dd' utility.  'man dd' for more information.  A grizzled veteran is a
valuable resource if you're trying to work things out.
 
SAS uses an input buffer of 256 bytes unless a longer record length
(LRECL INFILE option) is used.
 
Please re-post your code with log, using an explicit statement of your
FILENAME PIPE statement (no macrovariables), any and all log NOTE,
WARNING, and ERROR messages, and writing to log the _INFILE_ record for
the first ten or so records of your data:
 
  if _n_ le 10 then put _INFILE_;
 
 
Finally, when working with a FILENAME PIPE, it's often useful to first
try dumping the piped output to a file when developing code, then
re-writing your code to use a pipe once you've worked out details such
as record structure.  It's also very helpful to be using a Bourne or
derived shell (sh, ksh, bash) rather than a C-sh or derivative (csh,
tcsh) due to stdin/sterr redirection capabilities.  Your shell is
specified by the Unix environment variable $SHELL.  On some systems,
/bin/sh is actually a Posix-compliant shell (derived from ksh) rather
than the Bourne shell.  'man sh' should provide more information.
 
> We tried 2 different methods:
>
> Method #1 SAS log
> -----------------
>           libname output '/prod/sas_tmp/sasdata';
> NOTE: Libref OUTPUT was successfully assigned as follows:
>       Engine:        V612
>       Physical Name: /prod/sas_tmp/sasdata
> 2
> 3          %let modlname = %sysfunc(scan(&sysparm,1,' '));
> 4          %let numrows = 50;
> 5
> 6          data output.&modlname (drop = i);
> 7             infile "&modlname";
> 8             do i = 1 to &numrows;
> 9                 input ID $ 1-20 VAR_NAME $ 21-40  VALUE 41-60 @;
> 10                output;
> 11            end;
> 12         run;
>
> NOTE: The infile "MYTEST" is:
>       File Name=/vol/......./MYTEST
>
> NOTE: 1658 records were read from the infile "MYTEST".
>       The minimum record length was 256.
>       The maximum record length was 256.
>       One or more lines were truncated.
> NOTE: The data set OUTPUT.MYTEST has 82900 observations and 3 variables.
>
> I'm confused because each piped line is a length of 3150 (61 * 50).  Yet
> the SAS log above says 256.  Do we need to explicitly put an LRECL in?
> Also, I'm concerned because of the "one or more lines were truncated" note.
> A PROC PRINT shows that only the first of the 50 observations appears to be
> read per piped line.  Yet, when I execute this program using a regular file
> and not a pipe, everything works out fine.
>
> Method #2 sas program (don't have log at the moment)
> ----------------------------------------------------
>           libname output '/prod/sas_tmp/sasdata';
>
>           %let modlname = %sysfunc(scan(&sysparm,1,' '));
>
>           data output.&modlname;
>              infile "&modlname";
>              input ID $ 1-20 VAR_NAME $ 21-40  VALUE 41-60 @@;
>              end;
>           run;
>
> This ends with a "Severe I/O" error.  The pipe does not seem to close.
>
>
> I would really appreciate it if you have any suggestions about this that
> you could share with me.  I found some old SAS-L postings that talked about
> using the "mknod" statement.  Should we be using that?
>
> TIA,
> Fanchon Finucane
 
--
Karsten M. Self (kmself@ix.netcom.com)
 
    What part of "Gestalt" don't you understand?
    Welchen Teil von "Gestalt" verstehen Sie nicht?
 
  Web: http://www.netcom.com/~kmself/
  SAS for Linux: http://www.netcom.com/~kmself/SAS/SAS4Linux.html