Evaluating the results of a social benefit simulation using individual administrative data on benefit receipt

The simulation of social benefits is an important application of tax-benefit microsimulation models in social policy research. Simulation outcomes inform about the effects of social policies on macroeconomic variables, such as the income distribution or public budgets. Analogously, the potential effects of policy reforms can be simulated and examined. Furthermore, tax-benefit simulations allow for an evaluation of the interactions of different benefits in often complex social benefit systems. In this way, they contribute to the evaluation of the effectiveness and efficiency of social policies. However, the quality of the simulation outcomes has consequences for the assessment of the effectiveness of the benefit system. For example, an increasing number of recent studies on benefit non-take-up as one measure for the ineffectiveness of social policies explicitly address the difficulties in determining benefit entitlements using tax-benefit microsimulation models, as the level of measured benefit non-take-up also depends on the quality of the data (Tasseva, 2016; Bruckmeier et al., 2021; Doorley and Kakoulidou, 2024), and on the quality of the simulation (Bargain et al., 2012; Harnisch, 2019).

Consequently, the validation of the simulation outcomes to ensure reliable and relevant results is an important step in the application of tax-benefit simulations. In a macrovalidation simulation outcomes could be validated against aggregated figures from official statistics. In addition, the distribution of the simulated net earnings or disposable income can be compared with external statistics on the income distribution (Sutherland and Figari, 2009; Richiardi and Collado, 2021). Other validation approaches could focus on individual deviations, the quality of the underlying data or compare the outcomes of different simulation models for a common set of households (Jara and Oliva, 2018; Hufkens et al., 2016; Siebertova et al., 2016).

Especially when simulating social benefit entitlements in complex social policy systems, there are numerous potential reasons for inaccurate simulation outcomes. Typically, the information necessary for a comprehensive means test is not available in survey data originally collected for various purposes. Missing or inaccurate information in the data as well as complex regulations make it necessary to rely on assumptions for the simulation of eligibility. Even when using unbiased data with minimized measurement errors, benefit simulation results may be inaccurate. For example, design-related features of the benefit programme such as the interplay with other benefit programmes can lead to incorrectly simulated entitlements as well as incorrectly simulated missing entitlements. Hence, the quality of the simulation not only depends on the extent of measurement errors in the data, but also on modelling issues.

With this paper we contribute to the literature on validating the results of tax-benefit simulation models. We examine how well the results of an open-source tax-benefit microsimulation model for Germany (GETTSIM) on means-tested minimum income (UBII) entitlements match the benefits contained in administrative data on UBII. Our analysis has two objectives: First, the results should provide an assessment of the validity of the UBII simulation results using GETTSIM. Second, generalized conclusions for policy and non-take-up analyses based on tax-benefit microsimulation models will be drawn.

The administrative data in our study are collected from the local job agencies responsible for the administration of UBII. In contrast to most benefit simulation studies, the input variables on socio-demographic characteristics as well as financial information on income for the simulation come from the administrative processes. The administrative data is provided in the UBII application process and used by the local job agencies to calculate the UBII entitlement. On the basis of this accurate data, the typical measurement errors of survey data, such as the over- or underreporting of income, can be avoided. Furthermore, the data reference period for the variables in our data refers to months, which is also the basis for calculating the benefit entitlement. This means that time-period issues that lead to simulation errors should also be minimized. Remaining inaccuracies in the benefit simulation could occur for different reasons. Examples are discretionary decisions in the means test, information not considered in the simulation, or specific complex life circumstances that cannot be represented in the syntax of the simulation model. Furthermore, since GETTSIM also simulates entitlements to other social benefits, simulation errors that arise from the complexity of the benefit system and the interaction or overlap of different benefits also become visible.

Our analysis considers three concepts of validation. First, we examine how often the simulation correctly simulates UBII entitlements in general. The proportion of observations without simulated entitlements among all households corresponds to the beta error reported in the literature on benefit non-take-up and is usually used as a measure of simulation quality. Second, we compare the distribution of simulated and recorded claims in a macrovalidation, which is particularly important for policy analyses. Third, we examine deviations from the recorded entitlements at the individual level.

The paper is structured as follows: section 2 provides a brief description of the UBII benefit system. In section 3, we describe our data and the sample selection for the analysis. Section 4 introduces the tax-benefit microsimulation model GETTSIM and documents the used variables and data cleaning steps required for the simulation with GETTSIM. We then examine the simulation results with regard to deviations from recorded data in eligibility, the distribution of simulated UBII benefits and deviations in individual entitlements in section 5. Finally, we discuss the implications for policy analysis using GETTSIM and benefit simulations in general in section 6.

Our analysis focuses on the simulation outcomes for the entitlement to unemployment benefit II (UBII), introduced in Germany in 2005. In 2023 it was renamed “Citizen’s Income”, but since the data used in this study covers a period before 2023, the term “UBII” is used throughout. UBII is the most important means-tested social benefit in Germany with around 5.5 million recipients in 2024. It ensures the minimum income for employable persons and their households. The benefit is means-tested and assets-tested, with virtually all types of income and assets being considered when assessing eligibility. In addition, the benefit is granted conditionally on job-search and take up of employment. Some groups are exempted from work obligations, for example because they care for children or are in the education system. Note that unemployment is not a necessary requirement and recipients may live in very different circumstances, for example as low-wage earners, long-term unemployed, carers of relatives, or self-employed.

The total cash UBII benefit always relates to the household and consists of several components: standard benefits to cover living costs for each family member and individually calculated benefits for housing and heating costs. In addition, UBII recipients are covered by health insurance. Finally, UBII may grant additional cash benefits for special needs. For example, single parents can receive an extra monthly benefit up to 60 percent of the standard benefit to cover living costs. Additionally, other benefits for special life circumstances and needs exist (for example during pregnancy). In 2024, the nationwide uniform benefit to cover the cost of living for a single person amounted to 563 Euros per month. Benefits for partners and children are lower and age-dependent. The costs of accommodation and heating costs are fully covered, but only if they are appropriate in comparison to average costs in the region. The net income of all household members is deducted from total needs, with an individual allowance for earned income.

One advantage of tax-benefit microsimulation models is that the entire system of social benefits is considered in the analysis. This is particularly important in the case of the UBII simulation as there are interdependencies with two other important benefits, Housing Benefit (HB) and the means-tested Child Benefit (CB). The receipt of HB and the CB excludes the simultaneous receipt of UBII, but HB and CB can be claimed simultaneously. These two benefits target households with low (wage) income that is, however, significantly above the UBII minimum income. The aim is to enable low-income earners to avoid receiving UBII by receiving these two upstream benefits. Figure 1 illustrates the interaction of these three benefits for the example of a family with a minor child. The transition between the different benefit systems is determined by individual conditions, for example the amount of housing costs and the composition of the household. Households cannot claim UBII if the combined entitlement from HA and CA exceeds the UBII entitlement, while HA is only granted if the income of a household including the HA is at least 80 percent of the total UBII needs. This is intended to prevent households receiving HA from living substantially below the minimum income determined by UBII. Although it is possible to receive HA even if the resulting household income is slightly below the household’s need as defined by the UBII, microsimulation models typically assume that households are only eligible for housing benefit and means-tested child benefit if the combined amount of these two benefits exceeds the UBII entitlement. This is shown in Figure 1, which also shows the hypothetically calculated amount of HB and CB if UBII is more financially advantageous (shaded area).

Figure 1

Download asset Open asset

With standardized benefit levels for living costs (B) and special needs (BS), reported housing costs (HC) and clear rules for calculating the deductible household income (Y), the simulation of UBII with tax-benefit microsimulation models seems straightforward. However, due to the interplay with Housing Benefit (HB) and means-tested Child Benefit (CB), the calculation of UBII becomes more complex because potential entitlements to the two prioritized benefits must be considered when calculating UBII. In order to differentiate between benefits, existing tax-benefit models for Germany typically assume that the household is entitled to the maximum payment, with HA and CB considered together:

The brief summary of the institutional features illustrates that the simulation of UBII could be challenging. A lot of information is required due to the comprehensive means test, and discretion regarding the accepted HC as well as special individual circumstances not considered in B or BS may lead to wrong simulation outcomes. Furthermore, the distinction between the three different benefits can become complex and, in reality, may contain a certain degree of randomness due to, for example, locally different cooperation between the authorities responsible for the three different benefits.

Our analysis builds on a 2 percent randomly drawn sample of UBII recipients processed in the Sample of Integrated Welfare Benefit Biographies (SIG, version 0720) of the Institute for Employment Research (IAB) (Dummert et al., 2022). The information in the data is based on consolidated individual daily information from the administrative procedures of the local authorities responsible for UBII. This data contains personal characteristics of the recipients and daily information on UBII receipt. Our UBII sample is linked to other data from the administrative processes. For our analyses, this is in particular information about the other members of the household, which are also considered when calculating the benefit entitlement (members of the “benefit unit”), as well as all financial data from the benefit calculation process. The financial information is accurate to the month, with the information on the 15th of each month used to measure the characteristics of the recipient and the household. The financial data contain information on the households’ needs (the potential entitlement before income is deducted), the income considered in the means test, and the actually paid out UBII benefit. For our analysis, we use all the months of the years 2017 and 2018 (see Table 1). We focus on recipients between 25 and 64 years old and also drop a few observations because no adult is part of the benefit unit, for example because the household only receives UBII for special needs of children. In addition, we exclude observations from the analysis sample for which it is assumed that no valid simulation is possible. The main reason why entitlements cannot be accurately simulated is missing values in necessary variables (60 percent). Furthermore, no valid simulation can be carried out if benefit recipients live in a household with people who are excluded from UBII entitlements (36 percent), for example pensioners, because the data contains no reliable income information for people who are not entitled to UBII. With this last selection step another approximately 9 percent of the observations are excluded from the simulation sample. Table A1 in the appendix shows the distribution of key characteristics of the simulation sample and the observations that are excluded in this final step. Small differences between the two groups are evident in terms of age, living in East Germany, having no school qualification or non-German nationality. Female recipients are slightly underrepresented among the excluded observations compared to the simulation sample. Significant differences emerge between the household structure of included and excluded households. Children live in almost all excluded households (95 percent), either in couple households or with single parents. A closer inspection of the data shows that more than 50 percent of the excluded single parents have children above 18 years and younger than 25 years old, which also applies to almost all excluded couple households (97 percent). The final selection step therefore might slightly distort the representativeness of our simulation sample with regard to households with children in this specific age group.

We finally end up with a 731 thousand person-month-observations for the year 2017 and a 697 thousand person-month-observations for the year 2018 that enter the simulation model.

The simulation of the UBII entitlements is based on the German Taxes and Transfers Simulator (GETTSIM), an open-source static tax-benefit microsimulation model. The development of GETTSIM is led by researchers from the University of Bonn with contributions by colleagues from the ifo Institute, DIW, ZEW, IAB, and others. It has received funding by the German Research Foundation (DFG). Our simulation is based on the latest stable GETTSIM version v0.7.0. GETTSIM simulates the most important parts of the German tax and benefit system, such as income taxes, social security contributions, child benefit and means-tested child benefit, housing benefit and UBII. GETTSIM is implemented in Python. The model code, the code documentation, and other information on using GETTSIM are available online.¹ The goal of GETTSIM is to develop a microsimulation tool that can be used by many researchers with different data to investigate different research questions. GETTSIM is therefore data agnostic and can in principle be applied to data from different sources, as long as the input variables satisfy the documented definitions. The administrative data we use follows the logic of legal provisions and was collected for a specific purpose, namely the calculation of UBII benefits. Therefore, several steps or assumptions are necessary for preparing the required input variables for the simulation, which is explained in the following.

When assessing eligibility, only the needs and income of the members of the core family are considered in the means test (see section 2). Our data always refers to the family members relevant for the benefit calculation, thus inaccuracies when defining communities of needs (i.e., the core family in most cases) within a household are avoided. We distinguish between three approaches to data preparation to generate the necessary input variables for the simulation (see Table 2): First, the existing variables in our dataset can be used directly or in transformed form (E). Second, a new variable is derived or approximated from existing information (D). An example is assets considered in the means test. Although there is no information on assets in the data, all individuals in our sample are entitled to UBII and have therefore passed the asset test. The assets are therefore set to 0 so that no individual fails the assets test in the simulation. In the third approach, input variables for the simulation are generated by setting assumptions because no information is available in the data (A). An example is the tax unit ID. For the calculation of income tax, it is important to know whether the taxpayer has children. For unmarried couples, each partner receives their own tax ID. In the case of children living in the household of an unmarried couple we do not know which partner the children belong to. In this case they are assigned randomly between the two partners.

Table 2 gives an overview of the input variables used (original variable names used in GETTSIM) and the data preparation steps required to generate the input variables. Most variables can either be obtained directly from the data, such as most income data, or derived from available information. Variables for which assumptions are necessary have, on the one hand, no quantitative significance among UBII recipients, such as contributions to private health insurance or homeownership. On the other hand, most assumptions have no effect on the simulation of the UBII entitlement, since they are made for variables that are necessary for the simulation of other entitlements, such as pension payments (not relevant for the UBII sample) or parental leave benefits (these benefits are directly measured in the “other income” variable in the data). In summary, simulation errors resulting from missing or inaccurate information in the database should be of little relevance.

For our study, we make three adjustments to the code of GETTSIM in version v0.7.0 that are particularly important for the simulation of UBII. First, we present the simulation results for the GETTSIM version v0.7.0 (variant GETTSIM). In a first adjustment, we remove the limit on the maximum rent paid with the UBII benefit, since our data allow us to measure the rent accepted by the local UBII authorities (variant NoRentLimit). By adjusting the housing cost cap implemented in GETTSIM, the recipient’s rent recorded in the data is always fully taken into account in the simulation, since we know that this exact amount was paid to the recipient. In a second adjustment, we additionally implemented the rule that households can only receive HA if the sum of their income and the potential HA entitlement is at least 80 percent of the UBII needs (variant HAStricter1). This rule implies that HA recipients may also be worse off financially than if they were receiving UBII. This regulation leaves local authorities with discretion, and 90 or 100 percent can also be implemented. In the latest unreleased development version of GETTSIM, the rule is implemented with 100 percent, and for this reason, we include the implementation of the rule with 100 percent as a last third variant in the analysis (variant HAStricter2).

The simulation of social benefit entitlements with tax-benefit microsimulation models offers the opportunity to investigate many research questions. For example, the effects of policy changes on the income of those affected and on public budgets could be analysed. The analysis of benefit non-take-up also requires the simulation of benefit entitlements based on suitable data to determine the eligible population. However, securing the minimum income with a means-tested benefit is more complex than applying pure calculation logic, and the benefit simulations are usually inaccurate at various points, due to, for example, discretionary decisions. Against this background, we examine the results of an open-source tax-benefit microsimulation model (GETTSIM) for the simulation of a means-tested social benefit intended to ensure the minimum income in Germany (UBII). Our data allows us to apply the simulation to administrative data of UBII recipients collected during the application process and to compare the simulation outcomes with the recorded UBII entitlements. Using these accurate data minimizes simulation errors due to time period issues or typical measurement errors of survey data, such as over- or underreporting of income.

The results show that UBII entitlements are in most cases correctly simulated. In an adapted version of GETTSIM we used, only for 3 to 4 percent of all observations failed to simulate a UBII claim (beta error). The simulation also allowed a precise distinction between UBII and existing similar benefits (housing and means-tested child benefit) for most observations. To obtain these results, two adjustments to the GETTSIM v0.7.0 were made: First, the limit used in the simulation for the accepted rent in UBII was suspended because we can measure the accepted rent in our data. This data specific adjustment cannot be generalized as this information is usually not available in other (survey) data. Second, we integrated the regulations to distinguish between UBII and housing allowance more precisely. The high beta error of 9 percent in the results achieved with GETTSIM v0.7.0 without this adjustment supports the implementation of the regulation, as is already done in newer versions of GETTSIM. We further find a high match between the distributions of simulated and recorded UBII entitlements. The simulated mean values underestimate the mean of recorded entitlements only to a limited extent which should not introduce a significant bias to the estimates of fiscal aggregates. As expected, the variance of simulated entitlements is below the observed variance, as heterogeneity in the application process could not be reflected in the simulation. This can be particularly distorting for the analysis of subgroups, where we find varying degrees of fit to the recorded entitlements. A closer look at individual deviations between recorded and simulated entitlements reveals significant deviations for migrants, especially from crisis countries, which was particularly relevant in 2017 and 2018. In our data, this is probably related to the strong influx of asylum seekers to Germany and into the transfer system since mid-2015. Our results also suggest that also in households with many family members, with children or employed persons in the household, the simulation at the individual level is less precise. In summary, we conclude that GETTSIM is a powerful and easy-to-implement tool for the simulation of UBII entitlements and the analysis of policy change on aggregated outcomes. At the individual level, accurate results are also obtained for the vast majority of households. However, the composition of the households is important, and larger errors can be expected for more complex household structures.

The results also provide some insights for the analysis of eligibility based on tax-benefit simulations in general. In social policy systems with overlapping benefits, even with very good data quality, misspecification of benefit entitlements cannot be avoided to a relevant extent, especially when the benefits pursue similar objectives and discretionary decisions occur at the administrative level. For the analysis of policy measures on the number of recipients and the costs of various transfers, this implies that simulation results become less accurate depending on the degree of complexity and overlap. Since the mean values of various large sociodemographic groups are relatively accurately determined in the simulation, calibrating the simulated recipient numbers can compensate for these inaccuracies. In our case, we even found that the inaccuracy in the mean values tended to increase in the variants that achieved better results in the number of correctly simulated entitlements, implying that for households in the grey area between different available benefits (in our case UBII versus HA and CB) simulation outcomes are less precise. The analysis at the individual level has shown that simulation quality decreases particularly for subgroups with more complex life circumstances, such as households with children. This applies in particular to comprehensive last-resort minimum income systems that provide benefits in the household context and take all types of household income into account. Temporary special circumstances can also lead to simulation results that do not reflect actual payments. One example taken from our findings are the higher simulation inaccuracies for individuals with citizenship from a crisis country in connection with a strong influx of asylum seekers to the German benefit system. In crisis years, consideration should be given to excluding certain particularly affected subgroups from the analysis or to choose other simulation years, if possible.

If a high data quality can be assumed, the imprecision in differentiating between different benefits implies for the analysis of benefit non-take-up that among the households typically reported as beta errors, a large proportion is actually eligible. If the beta-error households are not included in the denominator of the non-take-up rate in this case, this leads to an overestimation of the non-take-up rate. Conversely, if the data quality is poor, beta error households would be to a large extent due to benefit overreporting and the inclusion in the calculation of the non-take-up rate would lead to an underestimation of true non-take-up. The favourable approach must be decided on depending on the quality of the data and the complexity and overlap of benefits. One possible approach for the analysis of highly overlapping benefits would be to consider a combined (non-)take-up rate for these benefits in the sense that a household is only classified as non-take-up household if it is simulated as eligible for one of the benefits but does not claim any of them.

1. 1
   No claim, no pain. Measuring the non-take-up of social assistance using register data

   1. O Bargain
   2. H Immervoll
   3. H Viitamäki

   (2012)

   The Journal of Economic Inequality 10:375–395.

   https://doi.org/10.1007/s10888-010-9158-8

   • Google Scholar
2. 2
   Short- and long-term participation tax rates and their impact on labor supply

   1. C Bartels
   2. N Pestel

   (2016)

   International Tax and Public Finance 23:1126–1159.

   https://doi.org/10.1007/s10797-016-9400-9

   • Google Scholar
3. 3
   Misreporting of program take-up in survey data and its consequences for measuring non-take-up: new evidence from linked administrative and survey data

   1. K Bruckmeier
   2. RT Riphahn
   3. J Wiemers

   (2021)

   Empirical Economics 61:1567–1616.

   https://doi.org/10.1007/s00181-020-01921-4

   • Google Scholar
4. 4
   The Trouble with Take-Up

   1. K Doorley
   2. T Kakoulidou

   (2024)

   The B.E. Journal of Economic Analysis & Policy 24:673–682.

   https://doi.org/10.1515/bejeap-2023-0170

   • Google Scholar
5. 5
   Sample of Integrated Welfare Benefit Biographies (SIG) 2007-2020

   1. S Dummert
   2. P Grunau
   3. H Hohmeyer

   (2022)

   Institute for Employment Research (IAB).

   • Google Scholar
6. 6
   Would a Flat‐Rate Tax Stimulate Entrepreneurship in Germany? A Behavioural Microsimulation Analysis Allowing for Risk*

   1. FM Fossen

   (2009)

   Fiscal Studies 30:179–218.

   https://doi.org/10.1111/j.1475-5890.2009.00093.x

   • Google Scholar
7. 7
   To claim or not to claim: estimating non-take-up of social assistance in Germany and the role of measurement error

   1. J Frick
   2. O Groh-Samberg

   (2007)

   Deutsches Institut Für Wirtschaftsforschung (DIW), DIW Discussion Papers 734, https://hdl.handle.net/10419/27258.

   • Google Scholar
8. 8
   TAKE: Reducing poverty through improving the TAKE up of social policies

   1. T Goedemé
   2. J Janssens
   3. J Derboven
   4. R Gestel
   5. E Lefevere
   6. G Verbist
   7. J Vergauwen
   8. M Vandervelden
   9. L Nisen
   10. S Linchet
   11. M Bolland
   12. J-F Reynaert
   13. B Thiry
   14. M Novella
   15. E Boucq
   16. K Bosch
   17. N Mechelen
   18. A Corselis

   (2022)

   Belgian Science Policy Office 2022.

   • Google Scholar
9. 9
   Non-Take-Up of Means-Tested Social Benefits in Germany

   1. M Harnisch

   (2019)

   Non-Take-Up of Means-Tested Social Benefits in Germany, Deutsches Institut Für Wirtschaftsforschung (DIW), DIW Discussion Paper 1793, 10.2139/ssrn.3352378.

   • Google Scholar
10. 10
    Leuven, FP7 InGRID project

    1. T Hufkens
    2. C Leventi
    3. O Rastrigina
    4. K Manios
    5. N Mechelen
    6. G Verbist
    7. H Sutherland
    8. T Goedemé

    (2016)

    Leuven, FP7 InGRID project.

    • Google Scholar
11. 11
    Top income adjustments and tax reforms in Ecuador

    1. HX Jara
    2. N Oliva

    (2018)

    Top income adjustments and tax reforms in Ecuador, World Institute for Development Economic Research (WIDER), Working Paper 2018/165.

    • Google Scholar
12. 12
    UKMOD – A new tax-benefit model for the four nations of the UK

    1. M Richiardi
    2. D Collado

    (2021)

    International Journal of Microsimulation 14:92–101.

    https://doi.org/10.34196/IJM.00231

    • Google Scholar
13. 13
    Improving the Validity of Microsimulation Results: Lessons from Slovakia

    1. Z Siebertova
    2. N Svarda
    3. J Valachyova

    (2016)

    International Journal of Microsimulation 9:77–105.

    https://doi.org/10.34196/ijm.00136

    • Google Scholar
14. 14
    AIM-AP: Accurate income measurement for the assessment of public policies. Final Report, STREP Project No. 028412, ISER

    1. H Sutherland
    2. A Decoster
    3. M Matsaganis
    4. P Tsakloglou

    (2009)

    AIM-AP: Accurate income measurement for the assessment of public policies. Final Report, STREP Project No. 028412, ISER, University of Essex.

    • Google Scholar
15. 15
    EUROMOD: the European Union tax-benefit microsimulation model

    1. H Sutherland
    2. F Figari

    (2009)

    International Journal of Microsimulation 6:4–26.

    https://doi.org/10.34196/ijm.00075

    • Google Scholar
16. 16
    Evaluating the performance of means-tested benefits in Bulgaria

    1. IV Tasseva

    (2016)

    Journal of Comparative Economics 44:919–935.

    https://doi.org/10.1016/j.jce.2016.02.003

    • Google Scholar

Author details

1. Kerstin Bruckmeier

   Institute for Employment Research (IAB), Regensburger Straße, Nürnberg, Germany

   For correspondence

   Kerstin.Bruckmeier@iab.de

   "This ORCID iD identifies the author of this article:" 0000-0002-4797-7543