Comparing two methods of reweighting a survey file to small area data
Robert Tanton, Paul Williamson, Ann Harding
National Centre for Social and Economic Modelling (NATSEM), Institute for Governance and Policy Analysis, University of Canberra, Australia; Department of Geography and Planning, University of Liverpool, United Kingdom
Cite as: R. Tanton, P. Williamson, A. Harding; 2014; Comparing two methods of reweighting a survey file to small area data; International Journal of Microsimulation; 7(1); 76–99. doi: 10.34196/ijm.00094
Abstract
One method of calculating small area estimates using survey data involves deriving new weights for each respondent in the survey. These new weights are derived so that the survey data sums to some known totals for a small area (from either a Census or administrative data). There are different methods for calculating these weights, and this paper analyses the results from two different methods – a generalised regression method and combinatorial optimisation. The weights derived from each method are compared, and advantages and disadvantages of each method are assessed. Estimates of housing stress at a Statistical Local Area in Australia from each method are then calculated, and these estimates are then validated against a third reliable source, Australian Census data from 2001.
This article contains 5 figures and 8 tables.
Figures
Figure 1. SLA level housing stress estimates (%).Figure 2. SLA level housing stress estimates (%).Figure 3. SLA level housing stress estimates (%): CO-Min Absolute.Figure 4. Distribution of GREGWT weights for NSW.Figure 5. Distribution of GREGWT weights for the Australian Capital Territory.
Tables
Table 1. Benchmarks used for creating small-area weights.
Residents in different types of non-private dwelling
Person
Single
Yes
8
Household Type
Household
Single
No
1
Household size – number usual residents
Household
Single
Yes
7
Dwelling tenure by weekly household rent
Household
Multi
No
7
Dwelling tenure by household type
Household
Multi
Yes
15
Dwelling tenure by weekly household income
Household
Multi
No
16
Monthly household mortgage by weekly household income
Household
Multi
Yes
12
Weekly household rental by weekly household income
Household
Multi
Yes
20
Dwelling structure by household family composition
Household
Multi
No
12
Total number of benchmark tabulations
10
Total number of benchmarks
130
Multi-dimensional means cross-tabulations of variables.
Not fully specified means that one or more of the cells in a benchmark tabulation were not used for weight production. For example, for the benchmark table of ‘Household Type’, the count of ‘Private households’ was extracted for use as a benchmark, whilst the count of ‘Non-private dwellings’ was excluded from the reweighting process.
Table 2. Comparison of methods in summary.
GREGWT
CO
Approach
National household weights from a national survey dataset are reweighted to household weights for SLAs by constraining to small-area census counts
Selection of a combination of households from a national survey microdata set that best fit small-area census counts
Weights
In fractional numbers
In integer numbers
Preparation of census data
Needs to address re-allocation of ‘not-stated’ and ‘not applicable’ counts
Needs to address re-allocation of ‘not-stated’ and ‘not applicable’ counts
Conflicting benchmark counts due to statistical disclosure measures
Causes non-convergence because no set of weights can be found that simultaneously satisfies all benchmarks
Seeks to minimise the difference between the final weights and the target benchmarks which typically results in weights that matchthe average of any discrepant benchmarks
Optimisation strategy
Algorithm reaches an optimised solution when residual (i.e. difference between an synthetic estimate and the benchmark count) approaches zero
Minimise absolute or proportional error
‘Convergent’ & ‘non-convergent’ SLAs
In some cases no convergent solution may be found; Average Household Absolute Sum of Residuals is >1 provides a proxy indicator for this non-convergence.
No convergence issues, although final ‘optimal’ estimates may still fail to fit all user-supplied benchmarks
Source: NATSEM (GREGWT) and Williamson (CO).
Table 3. Summary measures of goodness of fit.
Measure
Description
Overall Total Absolute Error (OTAE)
Absolute Sum of Residuals summed across all benchmark counts
Overall Total Absolute Error per household (OTAE/HH)
Absolute sum of residuals per household across all benchmark counts
Overall Total Absolute Proportional Error (OTAPE)
Absolute difference between benchmark counts when expressed as fraction of the table total
Overall relative sum of Z-square scores (ORSumZ2)
For each benchmark table, the Z-score of each benchmark count squared, and summed for the table; then divide by chi-square critical value for table (--> RSumZ2), then sum across all tables (--> ORSumZ2). For a given table, RSumZ2 > 1 shows it is not fitting.
Table 4. Results for constrained variables, Australian Capital Territory.
Measure
GREGWT
CO (Min Proportion)
CO (Min Absolute)
OTAE
139.6
133.4
92.2
OTAE/HH
0.1
0.1
0.1
OTAPE
0.4
0.2
0.2
ORSumZ2
48.4
0.5
27.8
Note: Lower numbers signify greater accuracy.
Table 5. Results for constrained variables, New South Wales.
Measure
GREGWT
CO (Min Proportion)
CO (Min Absolute)
OTAE
602.9
483.1
979.3
OTAE/HH
0.1
0.1
0.1
OTAPE
0.2
0.1
0.2
ORSumZ2
60.5
1.9
29.2
Note: Lower numbers signify greater accuracy.
Table 6. Results for housing stress, Australian Capital Territory and New South Wales.
