Interrelationship between evaluation metrics to assess agro-ecological models
Abstract
When evaluating the performances of simulation models, the perception of the quality of the outputs may depend on the statistics used to compare simulated and observed data. In order to have a comprehensive understanding of model performance, the use of a variety of metrics is generally advocated. However, since they may be correlated, the use of two or more metrics may convey the same information, leading to redundancy. This study intends to investigate the interrelationship between evaluation metrics, with the aim of identifying the most useful set of indicators, for assessing simulation performance. Our focus is on agro-ecological modelling. Twenty-three performance indicators were selected to compare simulated and observed data of four agronomic and meteorological variables: above-ground biomass, leaf area index, hourly air relative humidity and daily solar radiation . Indicators were calculated on large data sets, collected to effectively apply correlation analysis techniques. For each variable, the interrelationship between each pair of indicators was evaluated, by computing the Spearman’s rank correlation coefficient. A definition of “stable correlation” was proposed, based on the test of heterogeneity, allowing to assess whether two or more correlation coefficients are equal. An optimal subset of indicators was identified, striking a balance between number of indicators, amount of provided information and information redundancy. They are: Index of Agreement, Squared Bias, Root Mean Squared Relative Error, Pattern Index, Persistence Model Efficiency and Spearman’s Correlation Coefficient. The present study was carried out in the context of CropM-LiveM cross-cutting activities of MACSUR knowledge hub.
Full Text:
Short paper (PDF)References
Alexandrov, G. A. et al., 2011. Technical assessment and evaluation of environmental models and software: Letter to the Editor. Environmental Modelling & Software, 26(3), pp. 328–336.
Bellocchi, G., Acutis, M., Fila, G. & Donatelli, M., 2002. An indicator of solar radiation model performance based on a fuzzy expert system. Agronomy Journal, 94, pp. 1222-1233.
Bellocchi, G., Rivington, M., Donatelli, M. & Matthews, K., 2010. Validation of biophysical models: issues and methodologies. Agronomy for Sustainable Development, 30, pp. 109-130.
Bregaglio, S. et al., 2010. An integrated evaluation of thirteen modelling solutions for the generation of hourly values of air relative humidity. Theoretical and Applied Climatology, 102, pp. 429–438.
Bristow, K. L. & Campbell, G. S., 1984. On the relationship between incoming solar radiation and daily maximum and minimum temperature. Agricultural and Forest Meteorology, 31(2), pp. 159–166.
Campbell, G. S. & Donatelli, M., 1998. A simple model to estimate global solar radiation. Proceedings of the 5th European Society of Agronomy Congress, Zima M., Bartosova M. eds., Nitra (Slovak), pp. 133-134.
Cochran, W. G., 1954. The combination of estimates from different experiments. Biometrics, 10, pp. 101-129.
Confalonieri, R. et al., 2005b. WARM: a scientific group on rice modelling. Italian Journal of Agrometeorology, Volume 2, pp. 54-60.
Confalonieri, R. & Bocchi, S., 2005a. Evaluation of CropSyst for simulating the yield of flooded rice in northern Italy. European Journal of Agronomy, 23, pp. 315-326.
Confalonieri, R., Gusberti, D., Bocchi, S. & Acutis, M., 2006. The CropSyst model to simulate the N balance of rice for alternative management. Agronomy for Sustainable Development, 26, pp. 241-249.
Diodato, N. & Bellocchi, G., 2007. Modelling solar radiation over complex terrains using monthly climatological data. Agricultural and Forest Meteorology, 144, pp. 111-126.
Hargreaves, G. H. & Samani, Z. A., 1982. Estimating potential evapotranspiration. Journal of Irrigation and Drainage Engineering, 108(IR3), pp. 223-230.
Huth, N. & Holzworth, D., 2005. Common sense in model testing. Proceedings of MODSIM 2005: "International Congress on Modelling and Simulation". Modelling and Simulation Society of Australia and New Zealand Inc, December 2005, Melbourne (Australia), pp. 2804-2809.
Rivington, M., Bellocchi, G., Matthews, K. B. & Buchan, K., 2005. Evaluation of three model estimations of solar radiation at 24 UK stations. Agricultural and Forest Meteorology, 135, pp. 228-243.
Stöckle, C. O., Donatelli, M. & Nelson, R., 2003. CropSyst, a cropping systems simulation model. European Journal of Agronomy, 18, pp. 289-307.
Van Keulen, H. & Wolf, J., 1986. Modelling of agricultural production: weather, soils and crops. Wageningen: Pudoc.
Weaver, B. & Wuensch, K. L., 2013. SPSS and SAS programs for comparing Pearson correlations and OLS regression coefficients. Behavior Research Methods, 45, pp. 880-895.
Previous issues and volumes can be found in the 'Archives' section.
You can refer to a paper published in this series in the following format Author (2013) Title. FACCE MACSUR Reports 2: D-C1.3, where "D-C1.3" is the article ID en lieu of page range.