Simulating Societal Change, by Peter Davis and Roy Lay-Yee

This is a flawed book on an ambitious project with disappointing results. I will first describe the project, then discuss the results, and finally explain why I think the book, despite its merits, is flawed as a report on the project.

Taking their inspiration from the reputation of New Zealand during the period 1890–1920 for being a “social laboratory”, Davis and Lay-Yee set out to make a virtual representation of modern New Zealand, including the social and demographic changes this society experienced between 1981 and 2006. If that would not be enough, the model, called SociaLab, is build “from the bottom up”, ie, is grounded on individual micro-data, but at the same time should incorporate higher-level entities and mechanisms.

Source of the micro-data is the NZ longitudinal Census for the years 1981, 1986, 1991, 1996, 2001 and 2006. The Census waves had been linked into five linked pairs, achieving linkage rates in the 70%–75% range. (Only 30% of census respondents could be linked across all census waves.) From these data both an analytical sample and the starting sample for the simulation were derived. The analytical sample was used to estimate transition probabilities, using a series of regression models for each time-variant dependent variable, eg, being partnered, education level, employment, housing tenure. The sequence of predictive equations followed a conceptual framework based on social-determinants and life-course perspectives. This system, which enabled simulation of transitions of persons which were present at the beginning and end of each five-year period, was complemented by a population dynamics module to account for exit from and entry into the population through death, emigration, birth and immigration.

Davis and Lay-Yee employed SociaLab to simulate a series of counterfactual scenarios, trying to answer large questions such as “What if the decade of deregulation had never occurred?” and “What if the baby-boomer generation had been much like any other?” The impact of alternative hypotheses is measured mainly in terms of employment, the proportion receiving welfare, personal and household income, and deprivation. As the authors readily admit, the results of these counterfactual simulations are disappointing: they show little impact of change in a single factor on down-stream factors. It is telling that the summary of these results in the 13-page concluding chapter merits a paragraph of just seven lines.

There are several reasons for this disappointment. The main reason is probably the rather small number of variables in the starting data. These are: age, gender, ethnicity, number of years in NZ, country of birth, living alone, partnership status, new-born in dwelling, living with dependent children, being in full-time education, education achieved, religion, income (personal and household), employment (employed, unemployed, out of labour force), welfare receipt, deprivation, and housing tenure. I quote the list of variables in full, because this makes immediately clear the severe limits the data impose on the simulations that are possible: no simulations concerning health, no simulations of the tax-benefit system, few possibilities for scenarios about the labour market, no simulations of future pensions (the latter is an important task for many dynamic microsimulation models). Another reason may be that the results are presented mostly at the level of the complete population, with some breakdowns by gender and ethnicity. Other breakdowns might have revealed a bigger impact of scenarios for some subgroups. Also, the statistics shown are limited to averages and proportions. Arguably, one of the important advantages of microsimulation models over meso or macro models is that they can produce distributional parameters, and not just averages and proportions, but SociaLab has refrained from this.

Third, the operationalization of at least some counterfactual scenarios seems inadequate. For example, deregulation in NZ involved, among other changes, a resetting of macroeconomic policy, removal of subsidies, comprehensive tax reform and redesign of welfare state functions. However, the alternative scenario which is supposed to indicate what would have happened if the decade of economic reform had never occurred is implemented – without any argument – by making the distribution of employment status (employed, unemployed, not in labour force) in 1986 and 1991 the same as in 1981. This seems both too little and too much. Surely, the reforms did not just affect the distribution of employment status; conversely, this distribution would have changed also in the absence of any reform: 1981 was a year of a deep world-wide economic crisis. A similar point can be made for the scenario “What if [a comprehensive family support and employment incentive package] had not been enacted”, which is operationalized by making, for women living with dependent children, the distribution of employment status in 2001 and 2006 the same as in 1996. As a final example, Davis and Lay-Yee address the issue of the baby-boomers, who are sometimes accused of squandering the advantages of good economic times, eg, in generous early pensions and tax cuts, at the expense of the next generation. So they try to answer the question; what if the baby-boomer generation had been much like the generation before? However, the scenario actually implemented is that for women aged 35–54, the distribution of employment status in 2001 is made the same as in 1981. Given the background of the question this choice of scenario is hard to understand. Apart perhaps from some reactionaries who would like to restore the male single-earner family, few baby-boomer-bashers blame the women in that generation for doing more paid work than their predecessors. It is admittedly hard to see how the authors could have improved their operationalization of these questions, given the available data. But if that was not possible, other questions could have been pursued, or the questions could have been reframed to bring them closer to the actual implementations. Surely more attention could have been given to the specification of the counterfactual scenarios, for instance taking into account past trends.

Apart from chapters 10 (reporting the counterfactual modelling) and 11 (the conclusion), the chapters in this book are of two very different kinds. Chapters 5, 6, 7 and 9 are clear and straightforward descriptions of, respectively, the data preparation, the statistical analysis underpinning the simulation model, the simulation model itself, and descriptive results on what happened in NZ during the period covered. These chapters each include a footnote stating that Roy Lay-Yee was the principal author of that chapter, while for other chapters no such principal authorship is mentioned. By contrast, chapters 1–4 and 8 are wide ranging, touching on various large themes, such as modelling social change at the level of an entire society while working with individual-level data (chapter 2), the position of SociaLab within the domain of social simulation (chapter 3), life course, the era of deregulation and the baby boomer generation as the major components of societal change (chapter 4), and the idea of the “seven ages of man” or life stages as a framework for understanding demographic and social change (chapter 8). These chapters read more like a programmatic statement about the direction the wide field of social simulation (or even sociological research more generally) could take, extending way beyond SociaLab. The consequence of this is that someone who reads the book from beginning to end only starts to understand how SociaLab actually works from p. 41 on. Also, the idea of the “seven ages of man” introduced in chapter eight seems to have been useful as a way to create age brackets that are not arbitrary, but otherwise does not inform much the discussion of results.

In fact, the best way to read the book is to go straight to the Conclusion (chapter 11), which provides a clear description of SociaLab and its results and is honest about the disappointments and limitations. After that, the reader can select the chapters he or she is interested in. The authors do not mention their target public, but judging from the book itself, the intended readership are social researchers who are not necessarily engaged in simulation models.

The authors note that all microsimulation models have been built for a particular purpose, implying that SociaLab was intended as a kind of all-purpose model. I think that this is the fundamental problem with the project from which this book emanates: building the model came first (and took a lot of time and energy), and thinking about the questions which it should answer came later. The authors refer several times to climate models as an example of what they aspire to achieve in the social domain. However, a review of such models (Flato et al., 2013), to which they refer, makes clear that, first, the choice of model depends on the scientific question being addressed, and, secondly, that there are several kinds of models with different purposes. There is no reason to suppose that this would not apply also to social science research. Nevertheless, a focus on the simulation model irrespective of its intended uses seems to be a not uncommon occupational hazard among builders of such models. Some of them resemble the men (mainly) who spend years and considerable funds to reconstruct a vintage car, only to drive it carefully around the block on a Sunday when it is finished. The machine provides the motivation, not its possible uses.

In important ways, therefore, the book does the project few favours. After the staggering ambition expressed in the first chapters, the actual model and its results can only disappoint. This is a pity, as SociaLab has several advantages. The linked census data covering 25 years would be the envy of microsimulation model builders in many countries. The limited number of variables and the large time interval of five years between census waves are of course important limitations, but also make the model less opaque than many other dynamic simulation models. More thought given to the scenarios that could usefully be simulated with the model might have resulted in a much more informative book.

A final thought: the authors say that they would have liked to create a user-friendly tool. Such user interfaces for the general public have been devised for a number of simulation models. However, anecdotal evidence suggests that these user interfaces are rarely used by their intended audience. It would be nice if someone would take the time to write a review of such user interfaces, so that either the considerable resources spent on them could be put to better use, or a way could be found to attract more people to click on their websites.