Evaluating Density Forecasts: Forecast Combinations, Model Mixtures, Calibration and Sharpness

Publication date: 26 Aug 2008 | Publication type: NIESR Discussion Paper | Theme: Macroeconomics | NIESR Author(s): Mitchell, J | External Author(s): Wallis, K.F. | JEL Classification: C22, C53 | NIESR Discussion Paper Number: 320

In a recent article Gneiting, Balabdaoui and Raftery (JRSSB, 2007) propose the criterion of sharpness for the evaluation of predictive distributions or density forecasts. They motivate their proposal by an example in which standard evaluation procedures based on probability integral transforms cannot distinguish between the ideal forecast and several competing forecasts. In this paper we show that their example has some unrealistic features from the perspective of the time-series forecasting literature, hence it is an insecure foundation for their argument that existing calibration procedures are inadequate in practice. We present an alternative, more realistic example in which relevant statistical methods, including information-based methods, provide the required discrimination between competing forecasts. We conclude that there is no need for a subsidiary criterion of sharpness.

Keyword tags: 
calibration, density forecast, predictive distribution, probability integral transform

Research programmes