Protocol for model evaluation

Gianni Bellocchi, Mike Rivington, Marco Acutis

Abstract


This deliverable focuses on the development of methods for model evaluation in order to have unambiguous indications derived from the use of several evaluation metrics. The information about model quality is aggregated into a single indicator using a fuzzy expert system that can be applied to a wide range of model estimates where suitable test data are available. This is a cross-cutting activity between CropM (C1.4) and LiveM (L2.2).


Keywords


model evaluation, fuzzy logic

Full Text:

PDF

References


Acutis, M., Trevisiol, P., Confalonieri, R., Bellocchi, G., Grazioli, E., van den Eede, G., Paoletti, C., 2007. AMPE: a software tool for analytical method validation. Journal of AOAC International 90, 1432-1438.

Alexandrov, G.A, Ames, D., Bellocchi, G., Bruen, M., Crout, N., Erechtchoukova, M., Hildebrandt, A., Hoffman, F., Jackisch, C., Khaiter, P., Mannina, G., Matsunaga, T., Purucker, S.T., Rivington, M., Samaniego, L., 2011. Technical assessment and evaluation of environmental models and software. Environmental Modelling & Software 26, 328-336.

Bellocchi, G., Acutis, M., Fila, G., Donatelli, M., 2002a. An indicator of solar radiation model performance based on a fuzzy expert system. Agronomy Journal 94, 1222-1233.

Bellocchi, G., Acutis, M., Paoletti, C., Confalonieri, R., Trevisiol, P., Grazioli, E., Delobel, C., Savini, C., Mazzara, M., van den Eede, G., 2008. Expanding horizons in the validation of GMO analytical methods: fuzzy-based expert systems. Food Analytical Methods 2, 126–135.

Bellocchi, G., Fila, G., Donatelli, M., 2002b. Integrated evaluation of cropping systems models by fuzzy-based procedure. 7th European Society for Agronomy Congress, 15-18 July, Cordoba, Spain, 241-242.

Bellocchi, G., Rivington, M., Donatelli, M., Matthews, K.B., 2010. Validation of biophysical models: issues and methodologies. A review. Agronomy for Sustainable Development 30, 109-130.

Bennett, N.D., Croke, B.F.W., Guariso, G., Guillaume, J.H.A., Hamilton, S.H., Jakeman, A.J., Marsili-Libelli, S., Newham, L.T.H., Norton, J.P., Perrin, C., Pierce, S.A., Robson, B., Seppelt, R., Voinov, A.A., Fath, B.A., Andreassian, V., 2013. Characterising performance of environmental models. Environmental Modelling & Software 40, 1-20.

Bregaglio, S., Donatelli, M., Confalonieri, R., Acutis, M., Orlandini, S., 2010. An integrated evaluation of thirteen modelling solutions for the generation of hourly values of air relative humidity. Theoretical and Applied Climatology 102, 429-438.

Bregaglio, S., Donatelli, M., Confalonieri, R., Acutis, M., Orlandini, S., 2011. Multi metric evaluation of leaf wetness models for large-area application of plant disease models. Agricultural and Forest Meteorology 151, 1163-1172.

Carozzi, M., Bregaglio, S., Scaglia, B., Bernardoni, E., Acutis, M., Confalonieri, R., 2013. The development of a methodology using fuzzy logic to assess the performance of cropping systems based on a case study of maize in the Po Valley. Soil Use and Management 29, 576-585.

Confalonieri, R., Acutis, M., Bellocchi, G., Donatelli, M., 2009a. Multi-metric evaluation of the models WARM, CropSyst, and WOFOST for rice. Ecological Modelling 220, 1395-1410.

Confalonieri, R., Bellocchi, G., Boschetti, M., Acutis, M., 2009b. Evaluation of parameterization strategies for rice modelling. Spanish Journal of Agricultural Research 7, 680-686.

Confalonieri, R., Bregaglio, S., Acutis, M., 2010a. A proposal of an indicator for quantifying model robustness based on the relationship between variability of errors and of explored conditions. Ecological Modelling 221, 960-964.

Confalonieri, R., Bregaglio, S., Acutis, M., 2012. Quantifying plasticity in simulation models. Ecological Modelling 221, 159-166.

Confalonieri, R., Bregaglio, S., Bocchi, S., Acutis, M., 2010b. An integrated procedure to evaluate hydrological models. Hydrological Processes 24, 2762-2770.

Confalonieri, R., Gusberti, D., Acutis, M., 2006. Comparison of WOFOST, CropSyst and WARM for simulating rice growth (Japonica type – short cycle varieties). Italian Journal of Agrometeorology 3, 7-16.

Criscuolo, L., Donatelli, M., Bellocchi, G., Acutis, M., 2007. Component and software application for model output evaluation. Farming Systems Design 2007: an international symposium on Methodologies for Integrated Analysis of Farm Production Systems, September 10-12, Catania, Italy, 2, 211-212.

Diodato, N., Bellocchi G., 2007a. Modelling reference evapotranspiration over complex terrains from minimum climatological data. Water Resources Research 43, doi:10.1029/2006WR005405.

Diodato, N., Bellocchi, G., 2007b. Modelling solar radiation over complex terrains using monthly climatological data. Agricultural and Forest Meteorology 144, 111-126.

Donatelli, M., Acutis, M., Bellocchi, G., Fila, G., 2004. New indices to quantify patterns of residuals produced by model estimates. Agronomy Journal 96, 631-645.

Donatelli, M., Rizzoli, A.E., 2008. A design for framework-independent model components of biophysical systems. In: Sànchez-Marrè, M., Béjar, J., Comas, J., Rizzoli, A., Guariso, G. (Eds.) iEMSs2008 International Congress on Environmental Modelling and Software, July 7-10, Barcelona, Spain, 2, 727-734.

Fila, G., Bellocchi, G., Acutis, M., Donatelli, M., 2003a. IRENE: a software to evaluate model performance. European Journal of Agronomy 18, 369-372.

Fila G., Bellocchi G., Donatelli M., Acutis M., 2003b. IRENE_DLL: a class library for evaluating numerical estimates. Agronomy Journal 95, 1330-1333.

Greenwood, D.J., Neeteson, J.J., Draycott, A., 1985. Response of potatoes to N fertilizer: dynamic model. Plant and Soil 85, 185–203.

Jain, S., Sudheer, K., 2008. Fitting of hydrologic models: a close look at the Nash–Sutcliffe index. Journal of Hydrologic Engineering 13, 981-986.

Laniak, G.F., Olchin, G., Goodall, J., Voinov, A., Hill, M., Glynn, P., Whelan, G., Geller, G., Quinn, N., Blind, M., Peckham, S., Reaney, S., Gaber, N., Kennedy, R., Hughes, A., 2013. Integrated environmental modelling: A vision and roadmap for the future. Environmental Modelling & Software 39, 3-23.

Legates, D.R., McCabe, G.J., 2013. A refined index of model performance: a rejoinder. International Journal of Climatology 33, 1053-1056.

Loague, K., Green, R.E., 1991. Statistical and graphical methods for evaluating solute transport models: overview and application. Journal of Contaminant Hydrology 7, 51-73.

Matthews, K.B., Rivington, M., Blackstock, K., McCrum, G., Buchan, K., Miller, D.G., 2011. Raising the bar? - The challenges of evaluating the outcomes of environmental modelling and software. Environmental Modelling and Software 26, 247-257.

Moriasi, D.N., Arnold, J.G., van Liew, M.W., Bingner, R.L., Harmel, R.D., Veith, T.L., 2007. Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Transactions of the ASABE 50, 885−900.

Nash, J.E., Sutcliffe, J.V., 1970. River flow forecasting through conceptual models, Part I - A discussion of principles. Journal of Hydrology 10, 282–290.

Olesen, H.R., Chang, J.C., 2010. Consolidating tools for model evaluation. International Journal of Environment and Pollution 40, 175-183.

Richter, K., Atzberger, C., Hank, T.B., Mauser, W., 2012. Derivation of biophysical variables from Earth Observation data: validation and statistical measures. Journal of Applied Remote Sensing 6, 063557.

Ritter, A., Muñoz-Carpena, R., 2013. Performance evaluation of hydrological models: Statistical significance for reducing subjectivity in goodness-of-fit assessments. Journal of Hydrology 480, 33-45.

Rivington, M., Bellocchi, G., Matthews, K.B., Buchan, K., 2005. Evaluation of three model estimations of solar radiation at 24 UK stations. Agricultural and Forest Meteorology 135, 228-243.

Stöckle, C.O., Kjelgaard, J., Bellocchi, G., 2004. Evaluation of estimated weather data for calculating Penman-Monteith reference crop evapotranspiration. Irrigation Science 1, 39–46.

Taylor, K.E., 2001. Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research 106, 7183-7192.

Tedeschi, L.O., 2006. Assessment of the adequacy of mathematical models. Agricultural Systems 89, 225-247.

Wallach, D., 2006. Evaluating crop models. In: Wallach, D., Makowski, D., Jones, J.W. (Eds.) Working with dynamic crop models: evaluation, analysis, parameterization, and applications. Elsevier, Amsterdam, 11-50.

Willmott, C.J., 1981. On the validation of models. Physical Geography 2, 184-194.

Willmott, C.J., Robeson, S.M., Matsuura, K., 2012. A refined index of model performance. International Journal of Climatology 32, 2088-2094.




Previous issues and volumes can be found in the 'Archives' section.

You can refer to a paper published in this series in the following format Author (2013) Title. FACCE MACSUR Reports 2: D-C1.3, where "D-C1.3" is the article ID en lieu of page range.