Methodologies for uncertainty assessment

NUSAP

Description

The NUSAP system for multidimensional uncertainty assessment (Funtowicz and Ravetz, 1990) aims to provide an analysis and diagnosis of uncertainty in science for policy. The basic idea is to qualify quantities using the five qualifiers of the NUSAP acronym: Numeral, Unit, Spread, Assessment, and Pedigree. NUSAP complements quantitative analysis with expert judgement of reliability (Assessment) and systematic multi-criteria evaluation of the different phases of production of a given knowledge base (Pedigree). Pedigree criteria can be: proxy representation, empirical basis, methodological rigor, theoretical understanding, and degree of validation. Pedigree assessment can be further extended to also address societal dimensions of uncertainty, using criteria addressing different types of value weighting, quality of problem frames etc. NUSAP provides insight on two independent properties related to uncertainty in numbers, namely spread and strength. Spread expresses inexactness whereas strength expresses the methodological and epistemological limitations of the underlying knowledge base. The two metrics can be combined in a Diagnostic Diagram mapping strength of for instance model parameters and sensitivity of model outcome to spread in these model parameters. Neither spread alone nor strength alone is a sufficient measure for quality. Robustness of model output to parameter strength could be good even if parameter strength is low, if the spread in that parameter has a negligible effect on model outputs. In this situation our ignorance of the true value of the parameter has no immediate consequences. Alternatively, model outputs can be robust against parameter spread even if its relative contribution to the total spread in the model is high provided that parameter strength is also high. In the latter case, the uncertainty in the model outcome adequately reflects the inherent irreducible uncertainty in the system represented by the model. Uncertainty then is a property of the modelled system and does not stem from imperfect knowledge on that system. Mapping components of the knowledge base in a diagnostic diagram thus reveals the weakest spots and helps in the setting of priorities for improvement.

Resources required

Resources required for assessing the Spread qualifier depend on the method chosen (some form of Sensitivity Analysis or Monte Carlo analysis usually in combination with expert elicitation will be needed).

      For the assessment of Pedigree, many resources (pedigree matrices, pedigree calculator, kite diagram maker, elicitation protocol and questionnaires) are freely available from http://www.nusap.net. Basic skills of Expert Elicitation are required.

      Basic skills for facilitating structured group discussions are needed if one uses an expert workshop. In addition, skills are needed to arrive at a balanced composition of the workshop audience to minimise biases.

      Time required per expert elicitation in a one to one interview depends on the number of parameters and the complexity of the case. Typically, it may vary between 1 and 5 hours. A substantial amount of time may be needed for a good preparation of the elicitation interviews.

      Recommended length for a NUSAP expert elicitation workshop is between one and one and a half days

Strengths and limitations

+         Identifies both quantitative and qualitative uncertainty in quantitative information and enables them to be displayed in a standardised and self-explanatory way

+         Promotes criticism by clients and users of all sorts, both expert and lay, and will thereby support extended peer review processes

+         It is flexible in its use and can be used on different levels of comprehensiveness: from a "back of the envelope" sketch based on self elicitation to a comprehensive and sophisticated procedure involving structured, informed, in-depth group discussions on a parameter by parameter format

-         The scoring of pedigree criteria is to a large extent based on subjective judgements. Therefore, outcomes may be sensitive to the selection of experts involved in the scoring.

-         It is hard to apply the method to complex models with large numbers of parameters.

References

Www.nusap.net

Funtowicz SO and Ravetz,JR (1990) Uncertainty and Quality in Science for Policy. Dordrecht: Kluwer.

Craye M, van der Sluijs JP and Funtowicz SO (2004) A reflexive approach to dealing with uncertainties in environmental health risk science and policy, International Journal for Risk Assessment and Management 5 (2).

Van der Sluijs JP, Craye M, Funtowicz SO, Kloprogge P, Ravetz JR and Risbey J (2005) Combining Quantitative and Qualitative Measures of Uncertainty in Model based Environmental Assessment: the NUSAP System, Risk Analysis, 25 (2).

Van der Sluijs JP, Risbey J and Ravetz JR (in press) Uncertainty Assessment of VOC emissions from Paint in the Netherlands, Environmental Monitoring and Assessment.