THE USE OF STATISTICAL TECHNIQUES TO JUSTIFY THE RECOVERY OF CORRUPTED POLLUTION DATA

ABSTRACT:

Data from continuous pollution monitoring can be corrupted due to many reasons, for example, by instrumentation failure, calibration faults or operator error. Although distorted, this data often exhibits an underlying relationship to the correct measurements. Operators intuitively feel that such data can be recovered by minor manipulation, and thus included in their compliance reports. However this can be hard to justify, and regulatory authorities have understandably preferred to regard data as either “good” or “bad”.

Borrowing terminology from self validating (SEVA) instruments, data may usefully beregarded as “blurred” where the underlying measurements can still be recovered. The justification of such data manipulation has been aided by three developments:

i) Much wider adoption of statistical techniques in industry (e.g. six sigma) has
improved the skills base;
ii) There is greater availability of software tools; and,
iii) An increase in logger memory capacity means more system and analyser
health diagnostics are now recorded.

As a contribution to the debate about “best practice” two detailed industrial case studies are presented to illustrate how several months of blurred data have been recovered, and how statistical techniques and diagnostic logs have been used to justify the data manipulation to the regulatory authorities. Case study one concerns calibration error. Case study two concerns instrument malfunction. To allow this paper to be used as a training aid, all techniques used are explained with examples.