531: Goodness Index

The goodness of fit index can be interpreted as the average correlation or information criterion between the cross-validated forecasts and observations for all series. (If the Transform Y Data is on then the cross-validated correlation / information criterion is between the transformed forecasts and observations.) However, if the Pearson's or Spearman's correlation coefficient is used, rather than simply calculating an arithmetic average of the correlations, the correlations are first transformed to the Fisher z-scale, averaged, and then transformed back to the correlation scale (except in the case where any of the correlations are +/-1.0, in which case a simple average is calculated). The Fisher z-scale transform is used for Pearson's and Spearman's correlations because a simple average of these coefficients does not make mathematical sense for a variety of reasons ( Silver and Dunlap 1987 ).

The Goodness Index menu permits the user to change the score used to calculate the index. Since version 11, the default coefficient has been Kendall's tau, but in prior versions Pearson's correlation was used. Since version 17, various information criteria have been available; these criteria may be preferable when using one of the Generalised Linear Model (GLM) options in CPT. (See Regression Options for further details on GLMs.

Kendall's tau is set as the default because CPT then attempts to maximise the discriminatory power of the forecasts and is insensitive to the distribution of the data. Kendall's tau-c is a correction to Kendall's tau for ties. If there are ties in the Y data (which may happen if the data are counts, for example, or if there are multiple zeroes) then Kendall's tau cannot reach ! or -1, and tau-c is a more suitable alternative.

If it is considered important to minimise the squared errors in the forecasts then Pearson's correlation may be a preferred option. Note that selecting Pearson's correlation when the Transform Y Data is on is not generally to be recommended because it can result in Pearson's skill scores that are noticeably poorer than the goodness index might suggest. Similarly, if the Y data are not normally distributed and the Transform Y Data option is off and a GLM is not being used the use of Kendall's tau or Spearman's correlation is recommended.

If you elect to use one of the information criteria, in most cases the Bayesian Information Criterion (BIC) is preferred over the Akaike (AIC), but the Mutual Information Criterion (MIC) may be most suitable when using CCA or PCR ( DelSole and Tippett 2021 ).