On Mon, 10 Nov 2008 10:48:25 +0000, Tracy Clegg <tracy.clegg@UCD.IE> wrote:
>Thanks to Peter and Bruce for your thoughts on this.
>To shed a little more light on what we are trying to do: The measurements
>are hourly concentration of a gas detected on a farm. The only thing it
>would correlate with would be weather patterns which we don't have
>information on. As Peter suggested identifying a 'high cluster' could be
>done by identifying where a number of points occurred in a row above/below a
>given value (eg. 2 standard deviations above the mean). Something like this
>might work, but it would have to somehow allow for low values occurring
>within the cluster. Some periods of time show very high values interspersed
>with very low values. That's the whole problem. We'd like to know if a (eg)
>7 day period had significantly high overall values even though it contained
>low values. We thought of using a temporal sliding window that would move
>through time summing at given time-lengths. Then identify windows of high
>concentrations of the gas in total. The problem is how to do this with
>varying sizes of the window and does anyone know of any software that we can
>use to do this?
Do you have SAS/ETS? If so, look at PROC EXPAND. It's good for computing
moving averages and the like.
>Thanks again for your help
>From: Peter Flom [mailto:firstname.lastname@example.org]
>Sent: 06 November 2008 17:35
>To: Tracy Clegg; SAS-L@LISTSERV.UGA.EDU
>Subject: Re: temporal cluster
>Tracy Clegg <tracy.clegg@UCD.IE> wrote
>>I have hourly measurements of a continuous variable (concentration of a
>>gas) measured over 5 years. Could anyone tell me how to identify
>>clusters where the continuous variable was unusually high over a certain
>>period of time?
>I think cluster is the wrong term here, and may put people in mind of
>cluster analysis, which isn't what you want.
>What do you mean by "unusually high"?
>Does the value change over time, other than randomly, and if so, how?
>How are the values distributed over time?
>Is there autocorrelation?
>Some ways that you *might* want to do this:
> Find the mean value over the 5 years. Define 'unusual' as more than 2 sd
> Fit a loess curve, define 'unusual' as some distance above the predicted
> Fit fsome other curve, define 'ususual' as some distance above that.
>but you also want 'clusters'. If there is a lot of autocorrelation, then
>identifying one very high point will likely identify others that are almost
>Or you could say that a 'high cluster' is XXXX points in a row that are XXXX
>above a curve based on XXX
>If there values, over time, fit some distribution well, then you could find
>some definition of outlier based on the number of points and the nature of
>HTH or at least gives you some ideas
>Peter L. Flom, PhD
>www DOT peterflom DOT com