Explaining the Size and Nature of Response in a Survey on Health Status and Economic Standard

Nonresponse is probably the most severe problem in survey research. Today it is not unusual to find surveys with a response rate around or even below 50 percent. It is obvious that this high nonresponse will not only decrease sample size and correspondingly increase variances of estimates from these data, but the results might also become biased if response is selective. There is a large literature on methods to compensate for nonresponse ranging from calibration methods including standard post stratification, imputations, and to more sophisticated model-based methods. The key to a successful compensation is to understand the causes of nonresponse. This is also important because of its relevance to survey design, where resources have to be allocated between the possibly conflicting goals of increasing the precision of estimates and reducing nonresponse biases.

There is an increasing literature on the causes of nonresponse with more or less successful attempts to build models explaining response behavior. These attempts have been constrained by the usually very limited information available in the sampling frames. Researchers have then resorted to comparisons between the responding part of the sample and results from smaller but intensive studies of nonresponding sample members using the assumption that those who could be converted tell us something about those who belong to the hard core of nonresponders. They have also compared to larger surveys with more reliable measures and population statistics. Only rarely it has been possible to match individual survey records to reliable register data for the same individuals. The situation is somewhat different in panel surveys, because in a panel one can use information given by the respondents in a previous wave of data collection to explain response behavior in a more recent wave, see for instance, Brose and Klevmarken (1993), Lepkowski and Couper (2002) and Nicoletti and Peracchi (2005). The results from these studies are interesting and important, but they do not necessarily carry over to a cross-sectional survey or the first wave of a panel survey.

It is well-known that response is usually much lower in the first wave of a panel survey than in successive waves and that attrition thus takes place in an already selected sample. People who are notoriously difficult to trace and convince have already been eliminated from the sample in the first wave, see Laurie et al. (1999).¹ Lepkowski and Couper (2002) argue that the response process in the first wave is fundamentally different from that of subsequent waves. This is both because of self-selection of the sample units and because of the extra information and organizational experience gained by the survey agencies at each successive wave. Fitzgerald et al. (1998) reported the same experience from the Panel Study of Income Dynamics (PSID). The attrition between the first and the second wave was 12 percent, for the 20 next waves attrition was on average between 2.5 and 3.0 percent.

Another problem with using data from a previous survey wave to explain response behavior is that survey data always have measurement errors and other types of nonsampling errors. Depending on the variables used this might become a problem when estimating a response model.

In this study we have the advantage of having exceptionally good sample frame data that can be used to explain response behavior. The sample frame was the 2001 wave of the longitudinal register-based data set LINDA of Statistics Sweden. LINDA is a random sample including a few hundred thousand individuals from the Swedish population. Register data include population censuses, schooling, income, wealth and tax data, etc. The sources of these register data are various administrative and statistical registers of Statistics Sweden such as the register of educational achievements, the income register, the wage rate register and registers from the Swedish social security system and labor market authority.

From LINDA we selected by simple random sampling a smaller sample of 1,430 individuals 50 - 84 years old to which CATI interviews were administered by Statistics Sweden.² These telephone interviews included sequences of questions taken from the U.S. Health and Retirement Study (HRS) survey and the European Survey of Health, Ageing and Retirement in Europe (SHARE) and adapted to Swedish circumstances.^3,4 There were thus questions about health, labor force participation, wages, incomes and wealth. Most of these questions were about “facts” not about feelings, perceptions and attitudes. The average interviewing time was less than 30 minutes. The field work was done in the period April 3–May 11 2003 with nonresponse follow up June 2–June 22. In this period most Swedes completed their self assessment for income taxation, so the information needed to answer questions about incomes, assets and taxes should have been timely.

Prior to the field work the questionnaire was tested in the questionnaire laboratory of Statistics Sweden and in a small pretest. Interviewers were experienced telephone interviewer. They got a four hours long training session focusing specifically on our survey and they were afterwards asked to train on the questionnaire before they were allowed to work in the field. The nonresponse follow up was done by a few of the most experienced interviewers.

The contribution of this paper is thus an analysis of the response behavior in a cross-sectional survey with standard questions about health, incomes, taxes and assets using unusually rich sampling frame data from the registers of Statistics Sweden and a model which simultaneously explains contact and cooperation.

In the end of May 2003, the response rate was 56.5 percent and the share of refusals 19.6 percent. After the conversion attempts in June total response rate increased to 61.6 percent and the share of not found was reduced by 2.9 percentage units and the share of refusals by 3.0 percentage units. In the end 22.6 percent of the sample members refused and 15.8 percent could not be found. The latter figure, however, includes 12 individuals who were classified as over coverage and should have been eliminated. If this is done the response rate increases to 62.1 percent.⁸

Before proceeding to a multivariate analysis we start by motivating our choice of explanatory variables and analyzing a number of tables showing the association between response and the selected variables. With each table there is a chi-square statistic for a test of independence and the corresponding P-value. A significant test suggests that an association is stronger than one could expect by chance. These tables, however, only display bivariate relations and any association or lack of association could well change in a multivariate analysis. For instance, as shown in Table 1 there is virtually no difference in the response behavior of males and females, but we still prefer to include this variable in our multivariate analysis, because gender might be confounded with other variables and while it is often available in sampling frames it is of interest to find out if there is any partial effect. We expect to find that females have a higher probability of contact than males, because they are less mobile and more frequently at home. Their probability of cooperation might, however, be lower. Even if the time-cost of working females usually is somewhat lower than for males, because females have lower wages, females tend to be more sensitive to the issues of invasion of privacy and this effect might dominate.

Even if all studies have not found a clear relation between the age of the respondent and frequency of contact and cooperation, we expect to find one. Young people are more mobile than elderly and thus more difficult to contact. In our case the youngest cohorts are excluded from the study, but we expect to find that those who are in the peak of their career are more difficult to contact than those who are retired. However, many retirees in their sixties and early seventies might also be mobile, going on vacation trips, spending time at their vacation houses, visiting children, etc. We might also find that some of the oldest old are relatively difficult to contact because of the increased prevalence of health problems in these age groups, but as already mentioned our survey does not include people older than 84.

The relation between the probability to cooperate and age is more difficult to anticipate. Before retirement time cost is at its peak for many respondents, and for this reason one might expect to find a higher probability of cooperation among the elderly. However, elderly might be more sensitive to the issue of invasion of privacy than younger respondent, they might also find it tiring to spend half an hour in telephone and be more reluctant to bring out any documentation needed to give good answers.

The estimated age effects will also depend on other variables we choose to include. Some of them might pick up what otherwise would be interpreted as an age effect. It is difficult a priori to assume any particular functional form for the relation between response and age. For this reason, we have chosen to work with age group effects which will permit data to determine the shape of the relationship.

Table 2 shows that response rates are smallest among the youngest (50-55) and the oldest. But this result hides reversed age trends among refusals and not reached. Refusals increase with age while not found seems to be a bigger problem among people below the age of 70.

Schooling is expected to influence response behavior directly as well as indirectly as an indicator of other variables. A higher education might increase the understanding for the research issues involved in our project and make the respondent more sympathetic towards research. Schooling is also an indicator of labor market career and pay and thus of availability and time-cost. Also, after retirement respondents with long schooling are expected to be relatively more mobile, if for no other reason because they tend to have higher incomes. We thus expect schooling to have a negative influence on contact, while there are countervailing factors determining cooperation. There is no reason to believe that the effects of schooling are linear of take any particular nonlinear functional form, so we will work with three discrete dummy variables: Compulsory schooling, high school and university.

Register data on schooling are not ideal. They originate from census data, examination registers for all levels of education and from surveys to immigrants. The information on education obtained abroad is incomplete. The surveys to immigrants only provide part of the information. In all, data on schooling are missing for about 1.5 percent of the population covered. The major problem with these data is, however, that they only cover people in the age bracket 16 - 74. There are no register data for those who are 75 and above. In our sample we have missing data on the register schooling variable for 265 respondents. For these we have interview data for 150 respondents. Most of them, 73 percent, fall into the group with lowest education. In our multivariate analysis we have chosen to use the survey information for the 150 respondents and code the remaining 115 as having missing schooling data. We believe that most of these 115 respondents have at most basic schooling. The share of immigrants is twice that of the whole sample, 8 percent compared to 4.

Table 3 shows that there is no strong association between schooling and response behavior. (The high chi2-value is generated by the missing schooling category). The response rate is a little higher among respondents with university education, and respondents with only compulsory schooling are harder to convince than respondents with more schooling. None in the group with missing schooling data responded, most of them refused to participate. This result strengthens our belief that most of them are immigrants with limited knowledge of Swedish.

The variables household size and if married are expected to have a positive effect on the probability of contact, while any effect on cooperation is less obvious. Table 4 confirms this for marital status. Unmarried persons are less likely to respond than married. Most of this difference comes from a higher frequency of not reached for unmarried, while the difference in refusal rate is small.⁹

Table 5 gives similar information for household size. The bigger household the lower is the frequency of not reached, while there is no major difference in refusal frequency.

The most frequently used time-cost measure is the hourly wage rate. This variable is unfortunately not included in our register data, but we have a measure of a monthly wage rate. Because many sample members are retired, they do not have any wage rate. One approach to obtain a time-cost measure for those who do not work, is to estimate a wage rate for them had they worked. This can be achieved if a labor supply and an earnings function are estimated jointly with the contact (cooperation) function; for an application to panel data see Brose and Klevmarken (1993). In our case it is probably not very meaningful to estimate such wage rates for people who have retired, some many years ago. Very few Swedes work after the age of 65. Instead, we have chosen to use the wage rate measure only for those who have a wage. It is expected to have a negative effect on cooperation. If there is any contact effect it might also be negative. People with high wage rates tend to work long hours and might be difficult to reach. In addition, we introduce a dummy variable that takes the value one if the respondent has no wage income. We expect that it will have a negative effect on contact, because those who have no job tend on average to be more mobile, while we have no prediction as to its effect on cooperation.

There is no clear association between the monthly wage rate and response behavior, see Table 6. The refusal rate increases a little with increasing wage rate, while there is no trend in the share of not reached. We thus only find a very weak indication of a time-cost effect on cooperation.

Table 7, however, suggests that having a paid job influences response behaviour. Those who do not work are both more difficult to contact and to convince to cooperate. We note though that the employment indicator probably is confounded with age, schooling and other variables.

Previous results suggest that those who are relatively less well off are more difficult both to contact and to get to cooperate. For this reason, we have included in our analysis disposable income per capita and the indicators: if on welfare, if unemployed and if immigrant. Table 8 confirms the finding that respondents in low-income families have a much lower response rate. They are both harder to contact and convince to give an interview. There are no large differences between people with average incomes and those who have high incomes.

Although there are few respondents that have received any welfare, Table 9 suggests that those who are on welfare are difficult to find and also difficult to recruit for an interview.

If a respondent had been unemployed in 2002 had no significant effect on response, while immigrants are both difficult to contact and to recruit to an interview, see Table 10.

Previous studies have also found that it is more difficult to contact respondents in urban areas than in rural. We thus include indicators of the degree of urbanization. Living in one of the three major metropolitan areas is the standard of comparison in our multivariate analysis below, while we have dummies for other urban areas and rural areas. We also use a dummy indicator if the respondent has a secondary home and expect that the probability of contact will be relatively less for this group.

Table 11 confirms that it is more difficult to reach people in urban areas than in rural, while interestingly the refusal rate is higher in the rural areas. The leisure home indicator does not give the expected result (not shown). There is no difference in contact rate between those who have and do not have a leisure home, while people with a leisure home more frequently cooperate.

We use three indicators of the health status of the respondent: if the respondent got any sickness benefits in the survey year, if the respondent had stayed for at least one night in hospital during the year, and if the respondent had any psychiatric diagnosis in 1997 - 2002.¹⁰ None of these indicators necessarily show that the respondent is sick or in hospital at the time of the interview, but given that we know that the respondent has been sick, the probability of contact should be relatively high because the respondent is at home and – if not too ill – able to answer the phone. The probability of cooperation might however be low. Similarly, if the respondent was taken into a hospital, the probability of contact is likely to be low. Respondents with a psychiatric diagnosis might have both a reduced contact probability and a reduced cooperation probability.

Looking at raw data we found no significant difference in contact and cooperation frequencies between those who had collected sickness benefits and those who had not. Respondents who had stayed in hospital in the survey year had a lower contact frequency, while there was almost no difference in cooperation frequency. The P-value of the chi2-test was only 0.06.

Register data from the Centre for Epidemiology at the National Board of Health and Welfare include historical information about past psychiatric diagnosis for the period 1984 - 2002. Using all this historical information would increase the share with a psychiatric diagnosis from 1.3 percent in 2002 to approximately 8 percent. However, it is not obvious that all years contribute useful information. Some of those who got a diagnosis in, for instance, 1984 might have recovered by 2002. For this reason, we have only used data for a shorter period, 1997 - 2002.

A total of 3.4 percent of our sample frame had a psychiatric diagnosis at least once in this period and many of these individuals had a psychiatric diagnosis for more than one year, and some of them also had other problems diagnosed.

In Table 12 wee see the association between response and having at least one psychiatric diagnosis in 1997 - 2002. The response rate was 54 percent for this group. 18 percent could not be reached and 28 percent refused to participate. People with psychiatric problems are thus both more difficult to reach and to get to cooperate than an average respondent.

Just by looking at univariate distributions it is difficult to assess which variables are the most important to explain response, because many are confounded. We get, however, a very clear message from these tables, namely that response rates are much lower among low skilled and low-income people, many of whom are found among the oldest in the sample. We also confirm findings from previous studies that the contact frequencies are higher in large households and among households living in rural areas.

The sequence of events we wish to model is first the contact and if contact is established the event of giving an interview. Following Nicoletti and Peracchi (2005) we will use a bivariate probit model. Let Y₁ be a dummy variable that takes the value one if a contact is established and Y₂ another dummy variable that takes the value 1 if an interview is obtained. Assume the following model

where ${\displaystyle Y_1^{\ast }and{Y}_2^{\ast }}$ are bivariate normal latent variables, while ${\displaystyle {\epsilon }_1}$ and ${\displaystyle {\epsilon }_2}$ are bivariate standard normal. The X-vectors are vectors of exogenous explanatory variables uncorrelated with the ${\displaystyle ϵ:\mathrm{s}}$.

The parameters of the censored bivariate probit model have to satisfy certain constraints to make the model identifiable. If the covariates in the contact and the participation equations are the same, then the model is not identified. Identification becomes possible if X₁ and X₂ are not identical, i.e. exclusion restrictions are needed. In this respect we were guided by previous results and common sense. For instance, the variables “if having a leisure home” and “if having stayed in a hospital” were assumed to determine the probability of contact rather than the probability of cooperation. In the final specification a few insignificant variables were deleted from either equation. The model was estimated by maximum likelihood.

Table 13 gives summary descriptive statistics. Because the descriptive statistics suggested that the relation with age was not exactly the same for the contacts as for the response once contacted, two different age classifications were used, one in the contact equation and one in the response equation.

The maximum likelihood estimates are presented in Table 14. These results show that the probability of contact increases with age. Elderly people are more frequently at home to answer the telephone. There is no significant difference between males and females, while couples are easier to contact than singles. People with high school or university are somewhat more difficult to contact than people with only compulsory schooling, but these estimates are uncertain. The group with missing schooling data has a small probability of contact. When this group was deleted from the analysis, the effect of the immigrant dummy became stronger. This suggests that there is a positive correlation between having no schooling data and being immigrant. The missing schooling variable now picks up part of the immigrant effect.

The estimate of having sickness benefits is positive (sick people tend to be at home) but insignificant, while having stayed in hospital reduces the probability of contact.¹¹ We have also tried alternative specifications using the data on respondents having a psychiatric diagnosis. Replacing the hospital stay indicator with this variable in the contact equation also gave a negative and significant effect. If both variables were included both point estimates became negative, but the P-values increased. The P-value of the hospital stay variable became 0.06 while the value of the psychiatric diagnose variable increased to 0.16. For this reason, we only kept the hospital stay variable as our preferred specification. The other parameter estimates were robust to these changes in the specification.

People who do not work for pay and immigrants are much more difficult to contact than the average person, and the probability of contact is smaller in the big cities than in other urban and rural areas. The point estimate suggests that those who are on welfare are more difficult to contact than average, but this effect is not well determined. There is no significant effect of being unemployed in addition to not working. The estimate for the wage rate variable was small and insignificant and thus dropped from the equation. Household disposable income had no significant effect either.

The point estimate for those who have a secondary home has the expected negative sign, but the standard error is relatively high.

There are a few variables which we have tried but then dropped from the model. One of them is the number of children in the household, which in previous studies has been shown to explain response, in particular (many) children increase the probability of contact. In our case this variable became insignificant both in the contact and in the cooperation equations. This is perhaps not so strange, because in the age groups included in our study there are relatively few families with children and if they have children they are in the upper teens.

We have also experimented with a few interactions, namely gender x schooling, wage rate x schooling and disposable income x schooling, but they were all insignificant in both equations.

The probability of a successful interview increases with the age of the respondent. In particular those above 75 are willing to grant an interview. We do not find any gender effect in this case either. Household size and marital status are insignificant too.

Not having a job, being on welfare and being immigrant all reduce the willingness to cooperate. The immigrant effect is however rather uncertain. Disposable income does not contribute to the explanation of cooperation in addition to these variables.

There is a significant time-cost effect as the effect of the wage rate variable is negatively significant. If we drop the income variable from the equation the wage rate effect moves closer to zero and becomes insignificant. In this case the wage rate variable thus picks up some of the positive effect of the income variable. According to a conventional economic time allocation model both variables should be included.

The schooling variable does not contribute much to the explanation of cooperation with the exception that respondents with missing data on this variable have a much lower probability of cooperation than everyone else. As already suggested this result might mask effects that are unrelated to schooling.

In previous model runs the variables capturing the degree of urbanization became insignificant in the participation equation, and for this reason we dropped these variables.

Our model allows for a correlation between the contact and participation equation whereas Lepkowski and Couper (2002) assumed independence. They thus assumed that omitted variables have no joint impact on contact and participation. A more general assumption is to allow for unobservables influencing both the probability of contact and that of participation and thus creating a correlation between the contact and participation equations. Results are not conclusive. The correlation is moderately positive but insignificant.

From an economist’s perspective it might be reasonable to believe that time cost has a strong influence on the probability of contact and participation, and consequently that high wage earners and high-income people are difficult to convince to participate in surveys. Confirming previous results about nonresponse in cross-sectional surveys and attrition in panel studies this study shows that this notion is largely false. It is true that we have found a significant time cost effect on participation, but the major finding is that nonresponse primarily comes from the left tail of the income distribution. Respondents without work, on welfare, and immigrants are those who both are difficult to contact and to convince to participate. People at the peak of their career are also difficult to contact and to convince. Similar to many other studies we have also found that the probability of contact is relatively low in the big cities and among singles, but we could not find any significant decrease in cooperation.

This result would seem to have implications both for survey design and post survey compensation measures. The characteristics of the respondents that contribute to nonresponse suggest that this is a group which is rather uninterested in the research purpose of our survey and that measures should be taken to try to wake up a greater interest. The properties of those who do not respond also suggest that this is a group in an economic situation such that they should be sensitive to economic incentives even if they are rather small.

In addition to the major group of nonresponding, contact efforts should also focus on people who live in urban areas, who are single, have more than basic training, are in the peak of their work career and have a secondary home.

Recent nonresponse research has focused on the circumstances under which nonresponse damages inference to the target population and results in biased estimates of population entities (Singer, 2006). Groves (2006) demonstrated that high nonresponse does not necessarily result in biased estimates. In addition to the response ratio the magnitude of the bias depends on the correlation between the propensity to respond and the attributes the survey researcher is measuring. If there is no or only a very weak correlation, there is no or only a small bias even if the survey is burdened by nonresponse. This implies that researchers who in their analysis focus on variables that explain the contact and cooperation probabilities or on measures that are highly correlated with these variables will suffer from a biased inference unless proper compensation measures are taken. Our literature review and our own results are suggestive as to the nature of these variables. It also follows that calibration methods which try to compensate for nonresponse should use variables and population information that explain response. According to our findings gender is not such a variable while, for instance, age, marital status, labor force participation, if immigrant, health status, family income and population density of the area are such variables.¹²

1. 1
   Nonresponse in The American Time Use Survey. Who is missing from the data and how much does it matter?

   1. KG Abraham
   2. A Mailand
   3. SM Bianchi

   (2006)

   Public Opinion Quarterly 70:676–703.

   https://doi.org/10.1093/poq/nfl037

   • Google Scholar
2. 2
   Modeling Response in a Panel survey, Working Papers of the European Scientific Network on Household Panel Studies. Paper 81

   1. P Brose
   2. A Klevmarken

   (1993)

   Colchester: University of Essex. (Presented at the 8th session of the International Statistical Institute, Cairo.

   • Google Scholar
3. 3
   Can You Hear Me Knocking: An Investigation into the Impact of Interviewers on Survey Response Rates, Social and Community Planning Research

   1. P Campanelli
   2. S Purdon
   3. P Sturgis

   (1997)

   Can You Hear Me Knocking: An Investigation into the Impact of Interviewers on Survey Response Rates, Social and Community Planning Research, London.

   • Google Scholar
4. 4
   An analysis of sample attrition in panel data: the Michigan panel study of income dynamics

   1. J Fitzgerald
   2. P Gottschalk
   3. R Moffitt

   (1998)

   The Journal of Human Resources 33:251–299.

   https://doi.org/10.2307/146433

   • Google Scholar
5. 5
   Nonresponse Rates and Nonresponse Bias in Household Surveys

   1. RM Groves

   (2006)

   Public Opinion Quarterly 70:646–675.

   https://doi.org/10.1093/poq/nfl033

   • Google Scholar
6. 6
   Nonresponse in Household Interview Surveys

   1. RM Groves
   2. MP Couper

   (1998)

   Hoboken, NJ, USA: John Wiley & Sons, Inc.

   https://doi.org/10.1002/9781118490082

   • Google Scholar
7. 7
   Essays on Measurement Error and Nonresponse, Ph.d. thesis: Economic Studies 103

   1. F Johansson

   (2007)

   Department of Economics, Uppsala University.

   • Google Scholar
8. 8
   Culture and Survey Nonresponse

   1. TP Johnson
   2. D O’Rourke J
   3. L Owens

   (2002)

   Chapter 4 in Survey Nonresponse, Culture and Survey Nonresponse, New York, John Wiley & Sons.

   • Google Scholar
9. 9
   Strategies for reducing nonresponse in a longitudinal panel survey

   1. H Laurie
   2. R Smith
   3. L Scott

   (1999)

   Journal of Official Statistics 15:269–282.

   • Google Scholar
10. 10
    Nonresponse in the Second Wave of Longitudinal Household Surveys

    1. JM Lepkowski
    2. MP Couper

    (2002)

    New York: Survey Nonresponse, John Wiley & Sons, Inc.

    • Google Scholar
11. 11
    Non-Response Errors in Sample Surveys, Urval No 16

    1. HL Lindström

    (1983)

    Örebro: Statistics Sweden.

    • Google Scholar
12. 12
    Survey Nonresponse

    1. P Lynn
    2. J Clarke
    3. J Martin
    4. P Sturgis

    (2002)

    The Effects of Extended Interviewer Efforts on Nonresponse Bias, Survey Nonresponse, New York, John Wiley & Sons, Inc.

    • Google Scholar
13. 13
    Survey response and survey characteristics: Microlevel evidence from the European Community Household Panel

    1. C Nicoletti
    2. F Peracchi

    (2005)

    Journal of the Royal Statistical Society, Series A 168:763–781.

    https://doi.org/10.1111/j.1467-985X.2005.00369.x

    • Google Scholar
14. 14
    Estimation in Surveys with Nonresponse

    1. C-E Särndal
    2. S Lundström

    (2005)

    Chichester, UK: John Wiley & Sons, Ltd.

    https://doi.org/10.1002/0470011351

    • Google Scholar
15. 15
    Introduction. Nonresponse bias in household surveys

    1. E Singer

    (2006)

    Public Opinion Quarterly 70:637–645.

    https://doi.org/10.1093/poq/nfl034

    • Google Scholar
16. 16
    An analysis of attrition in the panel study of income dynamics and the survey of income and program participation with an application to a model of labor market behaviour

    1. JE Zabel

    (1998)

    The Journal of Human Resources 33:479.

    https://doi.org/10.2307/146438

    • Google Scholar

Author details

1. Fredrik Johansson-Tormod

   National Institute of Economic Research, Stockholm, Sweden

   For correspondence

   fredrik.johansson-tormod@konj.se

   Competing interests

   No competing interests reported

2. Anders Klevmarken

   Department of Economics, Uppsala, Sweden

   For correspondence

   anders@klevmarken.nu

   Competing interests

   No competing interests reported