AboutSubmit my research

Methodology

Research articles · Dec 31, 2013

Testing the Statistical Significance of Microsimulation Results: A Plea

Tim Goedemé, Karel Van den Bosch, Lina Salanauskaite, Gerlinde Verbist

Herman Deleeck Centre for Social Policy, University of Antwerp, Belgium; Belgian Federal Planning Bureau, Belgium

Volume 6(3) Winter 2013

Cite as: T. Goedemé, K. Bosch, L. Salanauskaite, G. Verbist; 2013; Testing the Statistical Significance of Microsimulation Results: A Plea; International Journal of Microsimulation; 6(3); 50–77. doi: 10.34196/ijm.00088

Abstract

In the microsimulation literature, it is still uncommon to test the statistical significance of results. In this article we argue that this situation is both undesirable and unnecessary. Provided the parameters used in the microsimulation are exogenous, as is often the case in static microsimulation of the first-order effects of policy changes, simple statistical tests can be sufficient. Moreover, standard routines have been developed which enable applied researchers to calculate the sampling variance of microsimulation results, while taking the sample design into account, even of relatively complex statistics such as relative poverty, inequality measures and indicators of polarization, with relative ease and a limited time investment. We stress that when comparing simulated and baseline variables, as well as when comparing two simulated variables, it is crucial to take account of the covariance between those variables. Due to this covariance, the mean difference between the variables can generally (though not always) be estimated with much greater precision than the means of the separate variables.

This article contains 8 figures and 1 table.

Figures

Tables

Table 1. The effect of family transfers on equivalent disposable household income in Lithuania, EU-SILC 2006.
Panel / Outcome 95% confidence interval
Scenario Estimate Standard-error Lower bound Upper bound
(A) Mean equivalent income Baseline (1) 1,519.70 24.42 1,471.82 1,567.58
Without family transfers (2) 1,493.29 24.36 1,445.54 1,541.04
Estonian family transfers (3) 1,520.23 24.38 1,472.43 1,568.02
Difference (2)–(1) −26.41 0.86 −28.11 −24.71
Difference (3)–(1) 0.53 0.45 −0.35 1.41
Difference (3)–(2) 26.94 0.99 25.00 28.88
(B) Percentage poor (fixed poverty line) Baseline (1) 20.25 0.91 18.47 22.03
Without family transfers (2) 21.57 0.93 19.75 23.39
Estonian family transfers (3) 20.08 0.90 18.30 21.85
Difference (2)–(1) 1.32 0.26 0.81 1.83
Difference (3)–(1) −0.18 0.17 −0.51 0.16
Difference (3)–(2) −1.49 0.30 −2.08 −0.91
(C) Percentage poor (floating poverty line) Baseline (1) 20.25 0.78 18.72 21.78
Without family transfers (2) 20.79 0.78 19.26 22.32
Estonian family transfers (3) 20.11 0.78 18.59 21.64
Difference (2)–(1) 0.54 0.28 −0.00 1.08
Difference (3)–(1) −0.14 0.18 −0.50 0.22
Difference (3)–(2) −0.68 0.32 −1.30 −0.06
(D) Gini coefficient Baseline (1) 34.95 0.59 33.79 36.11
Without family transfers (2) 35.49 0.60 34.32 36.66
Estonian family transfers (3) 34.90 0.59 33.75 36.05
Difference (2)–(1) 0.54 0.04 0.47 0.61
Difference (3)–(1) −0.05 0.03 −0.10 0.00
Difference (3)–(2) −0.59 0.05 −0.69 −0.49
(E) Decile ratio Baseline (1) 19.52 0.70 18.14 20.89
Without family transfers (2) 18.97 0.71 17.57 20.36
Estonian family transfers (3) 19.41 0.71 18.01 20.81
Difference (2)–(1) −0.55 0.24 −1.02 −0.08
Difference (3)–(1) −0.11 0.21 −0.51 0.30
Difference (3)–(2) 0.44 0.30 −0.14 1.03

Notes: The poverty line is calculated as 60 per cent of the median equivalent disposable household income. In the case of a fixed poverty line, the poverty line is kept constant for incomes with and without family transfers. In the case of a floating poverty line, the poverty line is equal to 60 per cent of the median equivalent disposable household income, with the median income recalculated in every reform scenario.

Source: EU-SILC 2006 UDB, EUROMOD, own calculations. Results in panels A and B are calculated with standard Stata commands, those in panels C, D and E with the DASP module. See the Appendix for details.