>From: ciro baldi <baldi@ISTAT.IT>
>this time i need an help on a performance issue.
>I'am processing huge amount of data and my programs do a lot of data
>steps and sorting.
Without even reading any further, I suspect this is where your main problem
might lie. Almost without exception, a stream-of-the-conciousness process
like this can be rethought and recoded so as to reduce the number of steps
and particularly sort steps. Stream-of-the-conciousness tend to be
business-analyst-friendly, but machine-hostile. Show us at least a part of
the process, and I am sure you will have performance-improving suggestions
Additionally, a program where a large file has to be viewed as grouped by a
number of different sets of keys at different points in the program can
usually get a huge benefit from indexing the file beforehand in all the
necessary ways instead of physically regrouping the file every time a
different by-processing is required. David Cassell has written and spoken
quite eloquently and convincingly on the subject a number of times starting
(AFAIK) at PNWSUG'99 and at following SUGI(s), and also on this very forum
(look it up in the archives).
>I run sas8.1 on win2000. this pc has 1GB ram and i set in the sasv8.cfg
>the following two parameters:
I would set memsize=0 to make SAS at liberty to use all the memory it
pleases, although I do not think it will do anything to solve this
>but watching at the task manager indicators it seems that my programs
>use only a little amount of memory (up to 120 mb when the program is
>sorting), causing the processing to be dramatically slow.
I am not sure that the low memory utilization is the culprit here...
>how can it be possible?
Sort programs (including one SAS uses) are traditionally designed to
minimize, not maximize memory usage. The primary reason behind it is the
lack of memory in the early machines, which is why files could always never
be matched by keeping one or more files as lookup tables in memory. The only
way was sequential matching (merging in SAS terms) after sorting, which,
although relatively slow, could be devised to use very little memory and
hence was at least doable. But even sort algorithms designed to improve
sorting performance by deliberately inflating memory usage (such as
distribution counting sort) are successful only as long as the satellite
record tail is not too long, otherwise I/O becomes a major bottleneck, for
which there is little remedy.
>maybe the programs could be changed in a way to better use the memory
>that is available, and i would like to learn how to do it,
Very well! Most often it can be done by reversing the thinking based on lack
of memory, i.e. replacing sorting by table lookup wherever applicable.
Indexing mentioned above is basically an implementation if the same idea,
except that the searching is done on disk and does not have to be coded
>but my primary interest is: letting the programs unchanged is there any
>other >configuration parameter that should be set to improve the
A magic trick like one you mention is a dream of anyone having to fix a
bunch of programs like yours written by someone else at an unknown point in
time. Would not we all like things to be fixable that easily! Unfortunately,
it works very rarely, if ever...
Paul M. Dorfman
MSN 8 with e-mail virus protection service: 2 months FREE*