1. Methodology
  2. Spatial microsimulation
Download icon

Validation of spatial microsimulation models: A proposal to adopt the Bland-Altman method

  1. Kate A Timmins  Is a corresponding author
  2. Kimberley L Edwards  Is a corresponding author
  1. School of Sport and Exercise Science, University of Lincoln, United Kingdom
  2. Arthritis Research UK Centre for Sport Exercise and Osteoarthritis, University of Nottingham, United Kingdom
Research article
Volume 9(2) Summer 2016
Cite this article as: K. A Timmins, K. L Edwards; 2016; Validation of spatial microsimulation models: A proposal to adopt the Bland-Altman method; International Journal of Microsimulation; 9(2); 106-122. doi: 10.34196/ijm.00137
3 figures and 3 tables

Figures

Figure 1
Download asset Open asset
Hypothetical illustrations of Bland Altman plots.
Figure 2
Download asset Open asset
Scatter plots for age and sex categories, simulated counts versus census ward totals.
Figure 3
Download asset Open asset
Bland Altman plots for age and sex categories, simulated versus census ward totals.

Tables

Table 1
Methods to assess goodness-of-fit of spatial microsimulation models.
Method Description
Correlation and R2 Plots simulated area count against actual area count. The R2, or coefficient of determination, is the square of the Pearson correlation coefficient and gives an indication of fit.
Standard Error about Identity (SEI) In contrast to R2, SEI reflects the error around the line of identity (x=y; also known as the line of equality), rather than the line of best fit.
Total Absolute Error (TAE) A sum of the error (the difference between simulated and actual population counts for each area) across a category.
Standardized Absolute Error (SAE) TAE is divided by the total expected count. The standardization allows comparison between tables.
Root Mean Squared Error (RMSE) The square root of the mean squared error. Gives an indication of error dispersion (if error follows a normal distribution).
z-score, modified z-score, and zm2 The z-score is a cell level statistic that reflects the difference in the relative size of the category between actual and simulated populations. The modified score deals better with small cell counts, though an adjustment still needs to be made where area counts are 0. The zm2 is an overall measure, summing the squared z-scores.
E5 A count of the number of areas where error is greater than 5%.
Table 2
Validation metrics for the comparison of simulated and census counts in each constraint category.
Category R2 SEI TAE SAE RMSE zm2 E5
Age group 1 1 1 2 0.0000006 0.016128 0.003318 0
Age group 2 1 0.999999 −522 −0.0001742 0.260556 2.175607 0
Age group 3 1 0.999991 −62 −0.0000195 0.688421 13.26152 6
Age group 4 1 0.999988 −1365 −0.0005442 0.626817 12.09789 6
Age group 5 1 0.999999 −221 −0.0001081 0.169536 2.005676 1
Age group 6 1 0.999988 −1591 −0.0009531 0.455455 14.59673 3
Age group 7 1 0.999989 −251 −0.0001029 0.689271 22.73108 22
Male 1 0.999998 −1703 −0.0001987 0.809217 6.331774 0
Female 1 0.999999 −2307 −0.0002389 0.85425 7.099424 4
Total 1 1 −4010 −0.0002200 0.722166 2.450981 0
Table 3
Summary of the strengths and weaknesses of validation methods presented.
R2 SEI TAE SAE RMSE zm Zm2 E5 BA plot
Indication of individual area error X X X X X √ X √ √
Indication of systematic (overall) bias X √ √ √ X X √ X √
Identify outliers √ X X X X √ X √ √
Indication of error distribution (by area size) X X X X √ X X X √
Handles empty cells (0 counts) √ √ √ √ √ X X X √
Comparable across models (of different sizes) √ √ X √ X √ X √ √
Single overall metric (for each category) √ √ √ √ √ X √ √ X

Download links