I think it's a DOS problem, and 16K is I believe an improvement over what it
used to be.
How are the batches of files delivered to you? If they arrive in a ZIP
archive, you might look at the possibility of filtering into multiple
directories as the files are unzipped, perhap via a script.
In SAS, you should be able to code something like
FILENAME stuff "c:\lib\sub1\*.dat,c:\lib\sub1\*.dat, ... ";
Then you could still process all in one step.
On Thu, 5 Jul 2001 04:37:32 GMT, Roger Lustig <julierog@IX.NETCOM.COM>
>There's nothing like testing...
>The other day I asked about a FILENAME statement that would
>run all the files in a directory/folder through a DATA step.
>Ian Whitlock and Charles Patridge helped out. I settled on
> FILENAME stuff "c:\lib\*.dat";
>I neglected to mention that I have many, many files. 25,000 of
>them in this batch; tomorrow I get 150,000 to be processed together.
>Evidently, SAS plus the DOS/Win98 OS (I was on NT before; but I
>took the problem home) must step through the entire directory
>from the top in order to find the next file. The result: ever-
>increasing execution time. The first 1000 records/files took 6 seconds
>to process; the next, 11; and so on, more or less, each additional
>1000 requiring another 5 seconds or so.
>First attempt at a solution: modified wildcards.
> FILENAME stuff "c:\lib\1234*.dat";
>This would run about 15% of the total number of records; with
>6 or 7 of these, plus some appending, I'd get the same results.
>No dice. In fact, the first few thousand records ran as slowly
>as the last few had when I used the global wildcard.
>Second attempt: split the folder up into multiple folders. This
>worked nicely--execution time dropped from 28 min. for 25,000
>records to 7. Of course, moving all the files from place to place
>wasn't fast either...
>But it did produce one observation--not that that's much of a
>sample. I found that the first few moves of a few thousand records
>took a long time; but once the number of records in the big folder
>dropped to 16K or so, things sped up dramatically and immediately.
>Was this just a coincidence? Or is there something about DOS/Win
>file structure (FAT32) that slows down when there are more than
>16K files in a folder?