Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling | |
Elshall, AS (Elshall, Ahmed S.)1,8,9; Ye, M (Ye, Ming)1,2,3; Pei, YZ (Pei, Yongzhen)3; Zhang, F (Zhang, Fan)4; Niu, GY (Niu, Guo-Yue)5,6; Barron-Gafford, GA (Barron-Gafford, Greg A.)5,7 | |
Source Publication | STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT |
2018-12-01 | |
Volume | 32Issue:10Pages:2809-2819 |
DOI | 10.1007/s00477-018-1592-3 |
Abstract | This paper defines a new scoring rule, namely relative model score (RMS), for evaluating ensemble simulations of environmental models. RMS implicitly incorporates the measures of ensemble mean accuracy, prediction interval precision, and prediction interval reliability for evaluating the overall model predictive performance. RMS is numerically evaluated from the probability density functions of ensemble simulations given by individual models or several models via model averaging. We demonstrate the advantages of using RMS through an example of soil respiration modeling. The example considers two alternative models with different fidelity, and for each model Bayesian inverse modeling is conducted using two different likelihood functions. This gives four single-model ensembles of model simulations. For each likelihood function, Bayesian model averaging is applied to the ensemble simulations of the two models, resulting in two multi-model prediction ensembles. Predictive performance for these ensembles is evaluated using various scoring rules. Results show that RMS outperforms the commonly used scoring rules of log-score, pseudo Bayes factor based on Bayesian model evidence (BME), and continuous ranked probability score (CRPS). RMS avoids the problem of rounding error specific to log-score. Being applicable to any likelihood functions, RMS has broader applicability than BME that is only applicable to the same likelihood function of multiple models. By directly considering the relative score of candidate models at each cross-validation datum, RMS results in more plausible model ranking than CRPS. Therefore, RMS is considered as a robust scoring rule for evaluating predictive performance of single-model and multi-model prediction ensembles. |
Subject Area | 地理学 |
WOS ID | WOS:000446064000003 |
Language | 英语 |
Indexed By | SCIE |
Keyword | Thermodynamic Integration Marginal Likelihood Decomposition Uncertainty Multimodel Prediction Distributions Reliability Mechanisms Selection |
WOS Research Area | Engineering ; Environmental Sciences & Ecology ; Mathematics ; Water Resources |
WOS Subject | Engineering, Environmental ; Engineering, Civil ; Environmental Sciences ; Statistics & Probability ; Water Resources |
Cooperation Status | 国际 |
ISSN | 1436-3240 |
Department | 环境变化与地表过程重点实验室 |
Publisher | SPRINGER |
Citation statistics | |
Document Type | 期刊论文 |
Identifier | http://ir.itpcas.ac.cn/handle/131C11/8535 |
Collection | 图书馆 |
Corresponding Author | Ye, M (Ye, Ming) |
Affiliation | 1.Florida State Univ, Dept Comp Sci, Tallahassee, FL 32306 USA; 2.Florida State Univ, Dept Earth Ocean & Atmospher Sci, Tallahassee, FL 32306 USA; 3.Tianjin Polytech Univ, Sch Comp Sci & Software Engn, Tianjin 300387, Peoples R China; 4.Chinese Acad Sci, Inst Tibetan Plateau Res, Key Lab Tibetan Environm Changes & Land Surface P, Beijing 100101, Peoples R China; 5.Univ Arizona, Biosphere 2, Tucson, AZ USA; 6.Univ Arizona, Dept Hydrol & Water Resources, Tucson, AZ 85721 USA; 7.Univ Arizona, Sch Geog & Dev, Tucson, AZ USA; 8.Univ Hawaii Manoa, Dept Geol & Geophys, Honolulu, HI 96822 USA; 9.Univ Hawaii Manoa, Water Resources Res Ctr, Honolulu, HI 96822 USA. |
Recommended Citation GB/T 7714 | Elshall, AS ,Ye, M ,Pei, YZ ,et al. Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling[J]. STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT,2018,32(10):2809-2819. |
APA | Elshall, AS ,Ye, M ,Pei, YZ ,Zhang, F ,Niu, GY ,&Barron-Gafford, GA .(2018).Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling.STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT,32(10),2809-2819. |
MLA | Elshall, AS ,et al."Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling".STOCHASTIC ENVIRONMENTAL RESEARCH AND RISK ASSESSMENT 32.10(2018):2809-2819. |
Files in This Item: | ||||||
File Name/Size | DocType | Version | Access | License | ||
2018071.pdf(1345KB) | 期刊论文 | 出版稿 | 开放获取 | CC BY-NC-SA | View Application Full Text |
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment