Cross-validating administrative and survey datasets through microsimulation

For this paper, the sample survey we use is PSELL version 3/2004 covering income reference year 2003 (PSELL3). This sample survey provides a representative but stratified random sample of approximately 3,600 private households (9,800 individuals) resident in Luxembourg (‘international civil servants’ included). Institutional households (mainly elderly people residing in institutions) are not covered by the survey. The unit of analysis is the ‘resident household’ (people living in the same house). The data collection method is face-to-face interview. Information about all types of gross earnings are collected through the survey, including labour income, investment and property income, social benefits in cash, private transfers, etc.

PSELL3 survey weights are determined taking into account non-response patterns (c.f. Section 3.4) and calibrating them to external control distributions. Non-response is controlled using the variables gender, age, citizenship, activity status, type of health insurance, and marital status of the ‘reference person’ of the household. Calibration relies on ‘updated information’ from the last census (2001). At the household level, the variables concerned are the household type (a typology based on the age of members of the household and size of the household), the tenure status of the head, and a geographical criterion. At the individual level, the variables include gender, age and citizenship.

Our source of administrative data is the Social Security Data Warehouse (SSDW), recently set up by the Inspection Générale de la Sécurité Sociale (IGSS) administration in Luxembourg. The main objective of the Data Warehouse is to compose a normalized and exhaustive basis for the generation of statistics serving diversified purposes (general reports, OECD, etc). The SSDW gathers data from operational files belonging to various administrations such as Social Security and the National Population Registry that are of interest for social protection analysis: monthly and yearly information on affiliation to social security, social contributions, and benefits like pensions or family allowances, etc. The basic unit is the individual. Administrative data, exhaustive in their universe of definition, are neither related to a sampling process nor to high non-response rates that require weighting and imputation on the survey data side. However, they are not error free, a point we return to in Section 3.3.

One omission in the SSDW is a lack of data from the fiscal administration. Instead the IGSS administration provides the information on labour earnings required to calculate the social contributions paid either by the employer, or by the earner when self-employed or socially insured on a voluntary basis. This results in three limitations concerning SSDW income data. First, in Luxembourg wages ‘declared’ to the social security administration are allowed to be truncated when greater than seven times the Minimum Social Wage⁴. As a result labour earnings may be truncated for high wages. Second, the declared earnings of persons paying social contributions on a voluntary basis may be far from their real level. Third, on the basis of the data available, farmers’ income cannot be properly determined. In addition to those limitations, the SSDW contains no information concerning capital income and private transfers.

Within the SSDW, ‘families’ are constructed on a ‘fiscal basis’. ‘Resident households’, which are the unit of analysis in PSELL3, cannot be identified. Instead, an alternative form of ‘fiscal household’ must be constructed. First, spouses⁵ are identified as a foundation for the household. This means that unmarried cohabitants do not appear as linked in the database (they belong to different fiscal households); this conforms to fiscal rules, which are briefly described in the Appendix. Second, a link is created between parents and their ‘children’ (in essence, those who are unmarried and either younger than 21; or older but still a student or disabled) through the family benefits raised by the former during the year⁶

For the purposes of this paper, only persons recorded as having positive earnings (income or allowance), plus the voluntarily insured or coinsured, have been extracted from the SSDW. One implication is that ‘international civil servants’ residing in Luxembourg may not appear in the EUROMOD input database (because they usually neither contribute to, nor benefit from, in monetary terms, the social security system in Luxembourg). In addition, in conformity with the PSELL3 database, only residents are included⁷. We thereby exclude all non-resident cross-border workers, despite the fact that they represent as much as 37% of total employment in 2003⁸, a level which is a particularity of Luxembourg (hence their importance in relation to the tax-benefit system). Finally, because it is impossible to identify people living in institutional households in the SSDW, they are included in SSDW data extract (but not in PSELL3). The net result is that the administrative data extracted from the SSDW for the year 2003 contains observations on 449,000 residents.

To permit cross-validation of the two input datasets it is important to eliminate identifiable dissimilarities between them with regard to their respective populations and the lack of precision in some important (income-related) variables. Table 1 summarizes the problem and provides insight about complementary adaptations that are needed for an ex ante better comparability of the survey and administrative datasets. We can see, for example, that capital income has to be dropped from the survey-based data because no information is available about such an income in the administrative-based data. Keeping capital income on one side only would bias our results and weaken comparability of outcomes.

Individuals receiving an income from agriculture also dropped from both datasets, again to enhance comparability, because in the administrative-based dataset there is an imperfect link between the contents of the income variable and the reality of earnings.

In all cases, when individuals are dropped, all members of the household are dropped as well in order to avoid bias due to a change in the structure of the household, a bias that might be transferred downstream.

When comparing monetary characteristics, the ‘equivalised disposable income’ of households will play a crucial role. As is well known, the equivalised disposable income⁹ is the ratio of total disposable income¹⁰ to the equivalent weight of the household. Following the ‘OECD-modified scale’, we assign a value (weight) of 1 to the household head, a value of 0.5 to each additional adult member, and 0.3 to each child (younger than 14). The idea is to allow a comparison of ‘well-being’ among families whose compositions differ while taking into account the economies of scale a family of several persons is benefiting from, compared to a single person. The equivalised disposable income (which from now on will be ‘called’ ‘equivalised income’ for short) is evaluated at the household level. Each member of the household is then attributed this (common) value of equivalised income.

Usually in the literature, the ‘resident’ household matters rather than the ‘fiscal’ one. Departing from this, we work with fiscal households, whether they are in survey-based or administrative-based data. This induces three effects that may generate some discrepancies between our results and the results based on (as they usually are) resident households.

First, the disparity in income is affected. Table 2 gives an illustration of a resident household composed of 2 unmarried parents and 2 dependent children. Under the ‘resident’ framework, the total income (3,910) is divided by the total equivalent weight (2.3) to determine the equivalised income of each member of the household (1,700). Under the fiscal framework, the father, unmarried, is fiscally separated from his partner and the children. The father’s (fiscal) household is associated with an equivalised income of 2,110 whereas the equivalised income attributed to the rest of the family is 1,000. Splitting households from resident into fiscal units therefore generates some disparity, and affects the conclusions that may be drawn about income heterogeneity within the whole population.

Second, the first moments of equivalised income (i.e. the ‘median’ from which the poverty line is calculated) differ from those evaluated on a resident household basis. From the illustration shown in Table 2, one can see that the mean equivalised income is 1,700 (median 1,700) if resident households are considered and 1,277.5 (median 1,000) when fiscal households are considered. This outcome stems from the definition of equivalised income. As also illustrated in Table 2, although both the household disposable income (to be attributed to each member within the household) and the individual equivalent weight are unambiguously lower in a ‘fiscal’ framework than a ‘resident’ one, the impact on the individual ratio is qualitatively unknown ex ante, as is the average evolution of the equivalised income throughout the population.

Finally, the move to a fiscal basis for households has implications for policy rules regarding the scheme known as ‘Minimum Guaranteed Income’. In Luxembourg this scheme is organized on the basis of resident household characteristics, most notably total household income. To reflect the move to a fiscal household basis, we instead apply the Minimum Guaranteed Income scheme based on fiscal household characteristics, via a change of the relevant rules with the EUROMOD parameter files. This approach is applied whether analysis is based on survey or administrative data, in order to eliminate differences attributable simply to definitional differences when cross-validating the two datasets. The implication is a slightly higher number of beneficiaries and, on average, a more generous complementary social allowance.

Table 3 compares selected measures for which the household is the unit of analysis. Differences attributable to the switch from resident to fiscal households are identified by presenting results from the survey data on both a resident and a fiscal household basis.

As one can see in Table 3, the weighted results from the harmonised survey dataset are ‘representative’ of a population of 169,620 resident households or, through the splitting procedure, 205,802 fiscal households. The higher number of fiscal households is to be expected, as nineteen percent of resident households contain two or more fiscal households. For the same reason a higher percentage of fiscal households comprise only one person (47%) than resident households (30%).

More generally, when considering fiscal households, Table 3 shows how close the survey-based data are to administrative-based data, despite the ex ante difference in source data. Of course, a partial reason for the similarity in results is the adaptation/selection procedure described in Section 2. Moreover, the weighting process of the survey data is itself based on administrative data sources partially overlapping our administrative-based dataset (see Section 2.1). Even so, this is not a priori a guarantee for comparability at the level of fiscal households, bearing in mind issues for which full harmonisation was not possible and the relatively small survey sample size. For this reason the level of agreement between survey-based and administrative-based data sources is, in this case, reassuring.

Tables 3 and 4 compare the harmonised input datasets when the individual, rather than the household, is the unit of analysis. The administrative dataset comprises 418,749 persons, whilst the weighted survey-based dataset represents 419,030 – a difference of only 0.07%. Once again, one observes strong similarities between the non-monetary characteristics given in Table 4 for the two datasets. The maximum difference between any of the measures reported is only two percentage points; all but two measures differ by a maximum of only one percentage point. This provides further evidence to support the view that the main sampling and harmonisation issues have all been adequately dealt with.

To an extent the similarity in non-monetary survey and administrative measures presented in Tables 2 to 4, whilst reassuring, is perhaps relatively unsurprising, given that harmonisation focused upon comparability of population coverage and the weighting of survey data to known (administratively recorded) sociodemographic totals. More challenging, and of more interest, is the comparison and cross-validation of monetary measures, none of which have been directly controlled as part of the harmonisation process.

As a starting point for this cross-validation, Table 5 focuses on differences in the mean and median values of the main income components. This comparison reveals that, at an individual level, the mean ‘primary income’ (see notes to Table 5) recorded in the administrative dataset is 7.3% lower than that recorded in the survey dataset. A search for further explanations reveals that this difference appears to be mainly due to a discrepancy in recorded employment income (about 90% of primary income, excluding capital income), which is 9% higher in the survey than in the administrative records. Interestingly, Nordberg (2003), using Finnish data, finds a recorded level of ‘earned income’ (conceptually similar to our ‘primary income’) lower for register data in 1995 but higher in 1999.

Possible sources of the discrepancy in the primary incomes reported in Table 5 include sampling error and a range of non-sampling errors (coverage dissimilarity, concept error, non-response, reporting and processing errors). Each of these sources is now considered in turn. As will be seen, it is difficult, if not impossible, to estimate the non sampling errors, hence a fortiori the need to evaluate the impact of them through cross-validation. Following a review of possible sources of error, it is to this cross-validation that attention is then turned.

Section 3.3 focused on estimates of average monetary values. We now turn our attention to the distribution of equivalised income. From Table 6 (columns B and D) it is clear that, unlike average values, synthetic measurements of inequalities do not differ too much between the datasets. Balanced indicators, like the Gini coefficient¹¹ and the interquartile or interdecile ratios, when derived from the survey-based data, are statistically compatible with those resulting from administrative-based data. The value of the more targeted Atkinson index¹² is also close to being statistically compatible¹³ when the aversion to inequality is low (with a coefficient of 0.5) but severely diverges if a stronger aversion to inequality is considered (Atkinson index with a coefficient of 2). This point is clarified below. Consideration of the other results presented in Table 6 (columns A and C) is deferred to Section 3.5.

In general terms, we might conclude that if averages of equivalised income differ, the shapes of the distributions (as determined on a ‘fiscal household’ basis following microsimulation of tax and benefit rules using EUROMOD) roughly coincide, except for discrepancies observed at the extremes of the curves.

This is confirmed through an analysis of the at-risk-of-poverty rates and a more detailed description of the distributions of income. The ‘at-risk-of-poverty rate’ is conventionally defined as the proportion of individuals whose equivalised income is below the so-called ‘poverty line,’ which is 60% of the median equivalised income. Tables 7 and 8 show at-risk-of-poverty rates for various population typologies and all categories within each of them¹⁴.

It is worth noting that the usual basis for analysis of poverty is the resident household and not the fiscal one, which makes a difference regarding the household disposable income, the equivalised income of the members, and hence the poverty line and the at-risk-of poverty rates (see Table 5). In contrast, we are focusing here on indicators relating to fiscal households. Indeed, we are constrained by the administrative-based data where no information is available about resident households. It would therefore make no sense to compare our results with others published at the European or national levels, and based on resident households. Rather, one must remember that our main objective is simply the comparison of our two input datasets for cross-validation purposes.

The global survey-based at-risk-of-poverty rate of 11. 5% is higher than that derived from our administrative data¹⁵ (9.6%) (Table 7). This holds true as well for most categories: only singles more than 65 years old and households with 2 workers or more are signalled as less at risk of poverty through survey-based than administrative-based data¹⁶. Meanwhile the poverty line is lower than the first income decile (€1,189) in the administrative-based data but higher (€1,243 EUR for the first decile) in survey-based data. More generally, the usual findings follow: younger people, singles younger than 65, singles with dependent(s) (most often lone parents) and the members of households where either nobody or only one person is working are more at risk of poverty than the other categories within the population, whichever dataset is under consideration.

One can also see that these populations are more concentrated in the lower end of the income distributions. Singles with dependent(s) and the households with no worker also experience less equivalised income, on average, than the members of the other associated categories (Table 8). Nevertheless, no systematic link can be observed between the mean level of equivalised income within a category and the at-risk-of-poverty rate. Finally, the number of dependents in two-parent households is also shown to play a role in raising the risk of poverty.

A higher average for the survey-based at-risk-of-poverty rate could be seen as somewhat contradictory compared to the lower degree of inequality as measured via the variant of the Atkinson index focussed on poorer individuals (inequality aversion coefficient = 2). However, a larger share of the population below the poverty line does not say too much about the distribution of the ‘poor’ along the income line. For example, it can be shown that the intensity of poverty, measured by the ‘income gap ratio’, is lower through survey-based data¹⁷ : the poor, relatively more numerous, are nevertheless benefiting from equivalised incomes closer to the poverty line, on average. This ‘concentration’ effect is also broadly visible through the Gini coefficient when computed within the population under the poverty line. The coefficient is lower for survey-based data (0.0847) than for administrative-based data (0.0884). Moreover, it is well known that the Atkinson index with a high aversion to inequality is very sensitive to the lower end of the income distribution. For example, dropping only the first percentile of the equivalised income distributions from both data sources results in indices that that are not only closer, but that also become statistically compatible¹⁸. This result is perhaps not overly surprising, given that panel data like the PSELL3/EU-SILC, although well suited for income distribution studies, often lack precision regarding the lower end of the income distribution due, for example, to low response rates in specific categories of the population, including ‘poorer’ households (see Section 3.3).

Concerning the other extreme of the distribution of equivalised income, a few striking discrepancies between the datasets are once again noticeable (Table 8). In particular, mean income within the upper decile, as expressed in terms of the population mean, is found to be highly variable. For example, the upper decile mean for the elderly is 233% higher than the global mean in the administrative-based dataset (211% higher in the survey-based dataset). In similar fashion the upper decile mean for singles with dependents is lower, compared to the population mean, in the administrative data (203%) than in the survey data (214%). However, such deviations were expected, given the known distortions regarding the collection of higher incomes reported in Sections 2.1 and 2.2.

Between those tail ends of the distributions, Table 3.6 shows that the profiles of the distributions are remarkably similar. This result, together with the general gap between the mean incomes as constructed from the two datasets, partly explains why the survey data indicate a higher share of individuals belong to tax-paying households. Regarding individual characteristics, the usual tendencies prevail. For example, the members of households with dependents are less often taxpayers than the population on average. This correctly reflects the fact, noted by Berger et al. (2002), that the Luxembourg tax-benefit system exhibits a clear ‘family advantage’.

So far in our cross-validation we have emphasized the similarities and discrepancies observed between our survey-based and administrative-based datasets. In doing so, all of our results have referred to the tax system as implemented and applied to earnings in 2003. Such an approach is static, focusing on the fiscal benchmark of 2003. In so far as differences have been observed, the unanswered question remains: how sensitive will the microsimulation of alternative tax systems be to these observed differences? To address this question, we now compare the outcomes resulting from the two datasets after changing the fiscal rules. Similarities consistent with the previously observed proximities between the distributions of income might be expected. This expectation is now briefly confirmed.

For the purposes of our analysis, we consider and compare three alternatives for the year 2003: the tax-system then in force (results already presented); the fiscal system as it would have been in 2003 if a significant tax reform, actually spread over the years 2001–2002, had not been implemented; and finally a hypothetical fiscal system involving no tax on income. The main characteristics of the tax system in Luxembourg and of the 2001–2002 tax reform are described in the Appendix. By applying all three fiscal systems to the observed 2003 population, we avoid the problem of observing changes that are not directly the result of changes in the tax system itself, but of changes in population, the economy, inflation, and other non-fiscal policies. (For further elaboration of this point, see Liégeois et al., 2010; Fuchs and Lietz, 2007; Immervoll, 2000; see Callan and Walsh, 2006, for a proposal of alternative benchmarks, including a ‘distributional neutral policy’, mainly appropriate when comparing countries).

To assess the distributional effects of our three fiscal policy alternatives we used the static microsimulation model EUROMOD. EUROMOD was chosen for this purpose as its in-built flexibility readily enabled us to research the first-round effects of policy reforms that have an impact on earnings (through social contributions, taxes and cash benefits), and hence on poverty and inequality (Sutherland, 2007).

It should be noted, however, that EUROMOD does not currently implement feedback effects arising from price, budget constraint or behavioural responses.

In Table 6, the Gini coefficient, the Atkinson inequality indices and the percentile ratios are reported for administrative-based and survey-based data, with and without the tax reform, before and after income tax.

Figure 1

Download asset Open asset

First we explore the impact of taxation. Estimates of the before-tax Gini index based on survey and administrative data are very similar (0.297 and 0299 respectively). Estimates of the post-tax Gini index are also highly similar, regardless of the tax system in force. In all cases it decreases, varying significantly more between fiscal systems (0.23 without tax reform; 0.25 with tax reform) than between datasets (where the measurable difference is of the order of only 0.002-0.003). This drop in the inequality coefficient is mainly due to vertical redistribution¹⁹ of the tax system (Reynolds-Smolensky index). The horizontal redistribution²⁰ appears to be negligible. Here again, results stemming from the two datasets are similar.

Next we examine post-tax differences in outcome for the fiscal system alternatives under consideration. Table 6 clearly shows that the values of the inequality coefficients are increased due to the 2001–2 tax reform, meaning that the inequalities in the distribution of equivalised income are amplified. This is illustrated by Figure 1, which shows the change in mean equivalised income by decile. On the whole, the 2001–2 tax reform increases equivalised income, with the size of the increase dwarfing the difference between datasets (5.6% according to our survey-based data; 5.8% according to our administrative-based data). Moreover, the higher the decile, the higher the relative change. A 9%-increase of mean equivalised income is observed when considering the highest decile compared with less than 1% for the lowest decile, whatever the dataset. Again, regardless of decile, the difference in estimated increase between datasets is far smaller than the size of the estimated size of that increase.

The reinforced income inequality is largely explained through a reduction of the Reynolds-Smolensky index of vertical redistribution. This can be further decomposed into ‘progressivity’ (Kakwani index) and ‘magnitude’ (a coefficient depending on the average rate of taxation), both factors playing a positive role in the vertical redistribution. Yet again differences in these indicators attributable to changes in fiscal systems are far larger than those due to data source (survey or administrative data).

Regardless of data source, the decomposition helps in understanding what is at stake in the tax reform. Clearly, the reduction in vertical equity due to the reform results from a drop in the rate of taxation of 5.0-5.1% and not from the progressivity, which increases from 0.342/0.357 to 0.411/0.430 as measured by the Kakwani index²¹.

In summary, what is most striking in Figure 1 and Table 6 is the consistency between effects whether changes are measured using administrative-based or survey-based data. In fact only one element, found in Table 6, shows a greater difference between datasets than between fiscal systems. This difference, already discussed in Section 3.4, is the Atkinson index calculated with an inequality version of 2.

Panel Socio-Economique Liewen zu Lёtzebuerg; (http://www.ceps.lu).

EU-SILC is an instrument aiming at collecting timely and comparable cross-sectional and longitudinal multidimensional microdata on income, poverty, social exclusion and living conditions (see http://epp.eurostat.ec.europa.eu/portal/page/portal/microdata/eu silc).

http://www.iser.essex.ac.uk/msu/emod/

Minimum Social Wage = €1368.74 per month as of 1 January 2003.

Either married all through the year or married during the (civil) year, or divorced during the year.

If unmarried parents, the child goes to his mother’s household, unless there is an explicit demand from the mother to link the child with his father concerning the family benefits. If born during the year or when family benefits come to an end during the (civil) year, a child is still linked to the household of his parent(s).

Information for non-residents is partially available in the Data Warehouse.

Source: STATEC – National statistical institute of Luxembourg (http://www.statistiques.public.lu).

For a detailed presentation of social indicators, see Atkinson et al. (2002) and Marlier et al. (2007).

Total disposable income = (earnings – social contributions – taxes + social benefits) summed up for all members of the household.

The Gini coefficient takes a value between 0 (minimum inequality) and 1. If we define the social welfare as $W(x)=\frac{1}{n^2}{{{\displaystyle \sum }}^{\text{}}}_i{{{\displaystyle \sum }}^{\text{}}}_j\min \{x_i,\text{}{x}_j\}$, then it can be shown that W(x) = μ * ( 1 – G), where n is the number of individuals, x_i/_j is the income level, μ is the average income, and G is the Gini inequality index. See Essama-Nssah (2000) and Lambert (1993).

The Atkinson inequality index can be expressed as $\left(\epsilon \right)=1-{\left[\frac{1}{n}*{{\sum }^{\text{}}}_i{\left(\frac{x_i}{\mu }\right)}^{1-\epsilon }\right]}^{\frac{1}{1-\epsilon }}$, where n is the number of individuals, x_i is the income level, μ is the average income, and is the inequality aversion coefficient. It takes a value between 0 (minimum inequality) and 1 and can be interpreted in terms of social welfare; it shows that part of total income which might be saved, while keeping the social welfare (associated to the Atkinson index) unchanged and distributing the remaining disposable income equally. The higher the value of ε, the stronger the impact of the left side of the distribution on the index. See Essama-Nssah (2000) and Lambert (1993).

This would be statistically compatible if a 99% confidence interval.

A decomposition of inequality indices by population sub-group could also enlighten the question.

As this difference falls outside the 95% confidence interval for the survey-based poverty rate (10.5% – 12.5%), it also appears to be statistically significant.

However, the difference is not statistically significant at the 1% level for the former category.

The Foster-Greer-Thorbecke poverty index with parameter 1, which is the product of the poverty rate and the income gap ratio, is shown to be 0.015 through survey-based data (respectively 0.016 through administrative- based data), leading to an income gap ratio of 0.015/0.115 = 13% (resp. 17%). The income gap ratio = 1 – (Mean income of the “poor”/Poverty line); it refers to the extent to which the incomes of the poor lie below the poverty line.

If the first percentile of the equivalised income distribution is left out, the Atkinson index with an inequality aversion of 2 drops from 0.226 to 0.159 for administrative-based data. It is much more slightly modified for survey-based data, from 0.168 down to 0.156 with a 95% confidence interval, which becomes [0.1490.162]. For a more general analysis of such tendencies, see Van Kerm (2007).

Vertical redistribution consists of reducing inequalities of equivalised income between households who have the same structure but a different income level.

Horizontal redistribution consists of reducing inequalities of equivalised income between households who have the same income level but a different structure.

The increase in progressivity can be explained by an enlargement of the first tax bracket (tax rate = 0%), which overcomes, regarding the measurement of progressivity through the Kakwani index, the effect of reducing the marginal tax rates for higher income levels.

The research presented in this paper is part of the REDIS project (Coherence of Social Transfer Policies in Luxembourg through the use of microsimulation models) funded by the Luxembourg National Research Fund under Grant FNR/06/28/19. We are grateful to Isabelle Debourges (IGSS) for her helpful assistance with the data and two anonymous referees, together with the editor of the International Journal of Microsimulation, for many stimulating suggestions. The paper was first presented at the Conference on Tax-benefit Microsimulation in the Enlarged Europe: Results from the I-CUE Project and Perspectives for the Future, Vienna, 3–4 April 2008. We are grateful to participants for helpful comments. Of course, we are solely responsible for any remaining errors.

1. Version of Record published: April 30, 2011 (version 1)

© 2011, Liégeois et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.