Validation of spatial microsimulation models: A proposal to adopt the Bland-Altman method
Kate A Timmins, Kimberley L Edwards
School of Sport and Exercise Science, University of Lincoln, United Kingdom; Arthritis Research UK Centre for Sport Exercise and Osteoarthritis, University of Nottingham, United Kingdom
Cite as: K. Timmins, K. Edwards; 2016; Validation of spatial microsimulation models: A proposal to adopt the Bland-Altman method; International Journal of Microsimulation; 9(2); 106–122. doi: 10.34196/ijm.00137
Abstract
Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies.
The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison.
The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods.
This article contains 3 figures and 3 tables.
Figures
Figure 1. Hypothetical illustrations of Bland Altman plots.Figure 2. Scatter plots for age and sex categories, simulated counts versus census ward totals.Figure 3. Bland Altman plots for age and sex categories, simulated versus census ward totals.
Tables
Table 1. Methods to assess goodness-of-fit of spatial microsimulation models.
Method
Description
Correlation and R2
Plots simulated area count against actual area count. The R2, or coefficient of determination, is the square of the Pearson correlation coefficient and gives an indication of fit.
Standard Error about Identity (SEI)
In contrast to R2, SEI reflects the error around the line of identity (x=y; also known as the line of equality), rather than the line of best fit.
Total Absolute Error (TAE)
A sum of the error (the difference between simulated and actual population counts for each area) across a category.
Standardized Absolute Error (SAE)
TAE is divided by the total expected count. The standardization allows comparison between tables.
Root Mean Squared Error (RMSE)
The square root of the mean squared error. Gives an indication of error dispersion (if error follows a normal distribution).
z-score, modified z-score, and zm2
The z-score is a cell level statistic that reflects the difference in the relative size of the category between actual and simulated populations. The modified score deals better with small cell counts, though an adjustment still needs to be made where area counts are 0. The zm2 is an overall measure, summing the squared z-scores.
E5
A count of the number of areas where error is greater than 5%.
Table 2. Validation metrics for the comparison of simulated and census counts in each constraint category.
Category
R2
SEI
TAE
SAE
RMSE
zm2
E5
Age group 1
1
1
2
0.0000006
0.016128
0.003318
0
Age group 2
1
0.999999
−522
−0.0001742
0.260556
2.175607
0
Age group 3
1
0.999991
−62
−0.0000195
0.688421
13.26152
6
Age group 4
1
0.999988
−1365
−0.0005442
0.626817
12.09789
6
Age group 5
1
0.999999
−221
−0.0001081
0.169536
2.005676
1
Age group 6
1
0.999988
−1591
−0.0009531
0.455455
14.59673
3
Age group 7
1
0.999989
−251
−0.0001029
0.689271
22.73108
22
Male
1
0.999998
−1703
−0.0001987
0.809217
6.331774
0
Female
1
0.999999
−2307
−0.0002389
0.85425
7.099424
4
Total
1
1
−4010
−0.0002200
0.722166
2.450981
0
Table 3. Summary of the strengths and weaknesses of validation methods presented.
