>From: ciro baldi <baldi@ISTAT.IT> > >dear l-ers, >this time i need an help on a performance issue. >I'am processing huge amount of data and my programs do a lot of data >steps and sorting.

Ciro,

Without even reading any further, I suspect this is where your main problem might lie. Almost without exception, a stream-of-the-conciousness process like this can be rethought and recoded so as to reduce the number of steps and particularly sort steps. Stream-of-the-conciousness tend to be business-analyst-friendly, but machine-hostile. Show us at least a part of the process, and I am sure you will have performance-improving suggestions in abundance.

Additionally, a program where a large file has to be viewed as grouped by a number of different sets of keys at different points in the program can usually get a huge benefit from indexing the file beforehand in all the necessary ways instead of physically regrouping the file every time a different by-processing is required. David Cassell has written and spoken quite eloquently and convincingly on the subject a number of times starting (AFAIK) at PNWSUG'99 and at following SUGI(s), and also on this very forum (look it up in the archives).

>I run sas8.1 on win2000. this pc has 1GB ram and i set in the sasv8.cfg >the following two parameters: >memsize 512m

I would set memsize=0 to make SAS at liberty to use all the memory it pleases, although I do not think it will do anything to solve this particular problem.

>sortsize 256m >. >but watching at the task manager indicators it seems that my programs >use only a little amount of memory (up to 120 mb when the program is >sorting), causing the processing to be dramatically slow.

I am not sure that the low memory utilization is the culprit here...

>how can it be possible?

Sort programs (including one SAS uses) are traditionally designed to minimize, not maximize memory usage. The primary reason behind it is the lack of memory in the early machines, which is why files could always never be matched by keeping one or more files as lookup tables in memory. The only way was sequential matching (merging in SAS terms) after sorting, which, although relatively slow, could be devised to use very little memory and hence was at least doable. But even sort algorithms designed to improve sorting performance by deliberately inflating memory usage (such as distribution counting sort) are successful only as long as the satellite record tail is not too long, otherwise I/O becomes a major bottleneck, for which there is little remedy.

>maybe the programs could be changed in a way to better use the memory >that is available, and i would like to learn how to do it,

Very well! Most often it can be done by reversing the thinking based on lack of memory, i.e. replacing sorting by table lookup wherever applicable. Indexing mentioned above is basically an implementation if the same idea, except that the searching is done on disk and does not have to be coded explicitly.

>but my primary interest is: letting the programs unchanged is there any >other >configuration parameter that should be set to improve the >performance?

A magic trick like one you mention is a dream of anyone having to fix a bunch of programs like yours written by someone else at an unknown point in time. Would not we all like things to be fixable that easily! Unfortunately, it works very rarely, if ever...

Kind regards, ------------------ Paul M. Dorfman Jacksonville, FL ------------------

_________________________________________________________________ MSN 8 with e-mail virus protection service: 2 months FREE* http://join.msn.com/?page=features/virus