Date: Wed, 20 Sep 2000 11:19:39 GMT
Reply-To: simon_pickles@MY-DEJA.COM
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: simon_pickles@MY-DEJA.COM
Organization: Deja.com - Before you buy.
Subject: Re: Improving SAS performance under Unix
I read your notes with great interest as we're reaching a position
where on our unix where performance is becoming an issue.
We're trying to run a SAS Share Server that is getting hit by loads of
big queries at once and can't seem to handle it. We're getting the
following message and the Share Server closes down. SAS Inst say
it's 'probably' a memory problem (makes sense). Have you by chance ever
seen this and can you say what it is and is there a solution ?
ERROR: System is Terminating Due to Hang Up Signal (SIGHUP)
and then
ERROR: Expecting page 8302, got page 8301 instead.
I've tried various MEMSIZE settings from 128MB to 0 (machine has only
512MB).
thanks
Simon
In article <20000919164342.K15539@ix.netcom.com>,
kmself@IX.NETCOM.COM wrote:
> --OQhbRXNHSL5w/5po
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> Content-Transfer-Encoding: quoted-printable
>
> I had an offline request for assistance in expediting processing under
> Unix from a SAS-L participant. Thought I'd share my thoughts on the
> topic. A week or two old, but still largely relevant <g>.
>
> ----- Forwarded message from karsten -----
>
> > Karsten:
> >=20
> > Sorry to bother you but I figure if anybody knows the answer to this
> > question, you do.
> >=20
> > We have a drop dead multi-million dollar project that need to
> > implement as quickly as possible. What I would like our Unix
> > administrator to do is set up me and one of my staff so that any
> > jobs we submit automatically receive the highest possible priority
> > (ideally, by userid.) He does not know how to do this. All he can
> > do is renice the process once it is submitted. Unfortunately, he
> > probably won't be here at 3AM to help.
>
> First, it's very helpful to find out what's blocking or bottlenecking
> your system.
>
> You can get an idea as to what's running by doing:
>
> $ top=20
>
> =2E..under SunOS, this provides some useful information including a
stat
> called wait -- this is the number of jobs which *have* access to CPU
but
> are blocked by something else. Usually I/O. Finding SAS jobs in this
> state is typically an idication of I/O contention.
>
> Other useful commands which can point to other system utilization or
> limiting factors (the memory stats commands vary by vendor):
>
> $ uname -a # system name/type
> $ ulimit -a # user limits
> $ vmstat # virtual memory stats
> $ free # free memory stats
>
> In a single-system (shared-server) environment, you can get more
> performance for your own tasks by:
>
> - Running at a higher priority than other jobs.
>
> - Running without interference from other jobs.
>
> Both solutions suggest you have the means to block other processes
from
> running. These might be other SAS processes, or (more likely at 3am)
> system administrative tasks -- backups, various system housecleaning
> operations such as 'locate' database, man page cleanup, etc.
>
> You can increase your own process priority through the 'nice' command.
> A 'nice' job runs with lower priority than an unniced one. The higher
> the nice value, the lower the priority. Conversely, the lower the
nice
> value, the higher the priority. Range is typically -20 - 20 or 1 - 40
> (varies by system), with default being 0 or 20 respectively. Nicing a
> SAS process to (<normal> - 1) should be sufficient to enhance it over
> all other normal processes. This *will* impact overall system
> performance, including interactive user shells. A sufficiently low
nice
> value may interfere with fundamental system performance --=20
>
> Longer term, you might be interested in the following:
>
> My suggestions for disk organization are pretty standard, across *all*
> platforms. This isn't something you can implement overnight, but it
may
> be something to pursue for longer-term system optimization. If you
can
> swing folks into supporting this, you'll see some pretty hefty
> performance enhancements. Essentially, you want:
>
> - Programs, list, and log files: RAID5 (high redundancy)
partitions,
> seperate from data, as code and output tend to be much less
> volatile.
>
> - Long-term data: RAID5, multiple spindles. You're concerned
about
> data integrity first, I/O second. RAID buys you integrity,
> spindles buy you performance.
>
> - Scratch: RAID0 or RAID1/0. Aka "stripe" sets, possibly with
> mirroring, though in almost all instances, striping is far and
away
> the only thing you want. As many possible spindles aggregated
into
> the largest possible storage. Because striping *reduces* the
> reliability of storage (it's a weakest link phenomenon,
essentially
> you get reliability =3D 1/n, where n is the number of spindles
> (disks) striped together), you'll want alternative scratch space
> available either as a hot-swap replacement or shared storage in
> another location.
>
> Increasing system memory may also help disk peformance to the extent
> that caching reduces the need for disk read/write activities.
>
> If you're in frequent contention with other users and are finding you
> don't get the love you need from MIS, I'd strongly recommend you look
> into getting SAS on workstations. I've got the Linux demo, can
> generally recommend it -- it works. PCs offer single-user performance
> roughly on par with high-end servers, better when you allow for the
fact
> that users aren't in contention with one another. OTOH, you don't get
> the aggregate power of a really big server for large jobs. =20
>
> The usual bottleneck becomes networked data transfer. This can be
> mitigated in a data warehousing environment by cutting data
> distributions to CDROM or DVD (650 MB and 18 GB respective storage),
or
> at least minimize networked data transfer by localizing traffic to a
> subnet.
>
> Again, my suggestions for SAS job optimization:
>
> 1 Minimize the number of program steps you use.
>
> 2 Sort late and infrequently (inverse Chicago voting school).
>
> 3 Select only required fields
>
> 4 Subset early -- discard unneded records
>
> 5 Summarize early -- use NOTSORTED to take advantage of existing
> collation without breaking when it does. PROC SUMMARY and
PROC
> FREQ can be used for data reduction.
>
> 6 Use SQL and data step views to reduce data I/O.
>
> 7 Use SQL to accomplish in one step what requires several in
> PROC/Data step.
>
> 8 Use indices rather than sort for multiple collation
processing.
>
> 9 Use system memory to reduce disk I/O -- allocate large amounts
> of memory -64MB - multiple GB where available.
>
> 10 Use SAS formats to eliminate match-merging (but be aware that
> format memory may not be freed after usage).
>
> 11 Where indicated, use advanced processing techniques, including
> array processing and hash tables, to reduce data I/O and need
> for sorting and/or collation of very large data sets. Be
aware
> that programmer time is traded for system efficiencies.
>
> Combinations of these techniques can achieve 5 - 10 -fold reductions
in
> processing time.
>
> Again, good luck.
>
> ----- End forwarded message -----
>
> --=20
> Karsten M. Self <kmself@ix.netcom.com>
http://www.netcom.com/~kmself
> Evangelist, Opensales, Inc.
http://www.opensales.org
> What part of "Gestalt" don't you understand? Debian GNU/Linux
rocks!
> http://gestalt-system.sourceforge.net/ K5:
http://www.kuro5hin.org
> GPG fingerprint: F932 8B25 5FDD 2528 D595 DC61 3847 889F 55F2 B9B0
>
> --OQhbRXNHSL5w/5po
> Content-Type: application/pgp-signature
> Content-Disposition: inline
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.2 (GNU/Linux)
> Comment: For info see http://www.gnupg.org
>
> iD8DBQE5x/ouOEeIn1XyubARAkDOAJ94ldv3/liqnPDWY+5jVqPv4XK2YQCfUWb6
> nI6j/LN3KHYmiwx+VxmFNW4=
> =R7Xj
> -----END PGP SIGNATURE-----
>
> --OQhbRXNHSL5w/5po--
>
Sent via Deja.com http://www.deja.com/
Before you buy.
| 
