2023 Post-enumeration Survey: Methodology

2023 Post-enumeration Survey: Methodology presents background information, operational and methodological details, and key results from Aotearoa New Zealand's 2023 Post-enumeration Survey (PES).

Read this paper online or download the PDF below.

While this result was driven primarily by a reduction in gross undercount (down from 171,800 people in 2018), an increase in gross overcount also contributed (up from 46,700). The level of gross overcount in 2023 was the highest recorded, at 1.2 percent. (Gross overcount rates are calculated using the census usually resident population count as the denominator.)

Figure 5

The 2023 PES linking design went through many improvements based on lessons learned from 2018 . As a result, the improvements were incorporated into the 2023 Post-enumeration Survey: Final linking methodology was very effective at finding duplicate census records that would previously have gone undetected due to limited or incorrect linking information (such as misspelled names or out of date address information). Consequently, the 2023 PES linking was able to detect a higher rate of overcount than would have been possible using the methodology from 2018. This is a major contribution to the higher gross overcount.

The 2023 Census implemented a combined model by design, with alternative data sources to supplement census responses. This includes the use of admin data to add records (referred to as admin enumerations) to the census file when an individual has not responded, and when we are confident in the quality of the admin record . The census use of admin enumerations to supplement census responses means that some individuals may have been counted as both a census response and as an admin enumeration, contributing to overcount (Linking 2023 Census responses to the Integrated Data Infrastructure).

A combination of factors may have also contributed to higher levels of duplicate responses (where an individual has been counted on more than one census form) in the 2023 Census. These included:

• an extended field collection period may have led to some respondents to forget they had already filled out a census form
• additional follow-up processes in some regions to mitigate the impacts of Cyclone Gabrielle - multiple visits may have encouraged respondents to fill out a second form
• more households were provided packs with paper census forms prior to census day - there is a higher chance that duplicate responses will not be recognised as the same person if one of the responses was paper-based due to potential scanning issues
• the introduction of a pre-census day reminder may have prompted individuals to respond more than once.

All these factors may have led to individuals having either filled out multiple census responses or having been counted through more than one census form due to proxy responses from a household member. These duplicate responses remained in the final 2023 Census dataset because their linking information (such as name, age, gender, and usual residence location) was not sufficient to correctly identify them as the same person.

4.2.2 Response rates

In 2023, census response rates increased compared with 2018 for almost all major groups for which this was measured . The national response rate was 87.6 percent in 2023, up from 85.8 percent in 2018. However, the 2023 response rate was below the response rates of 93.2 percent in 2013 and 95.1 percent in 2006.

Figure 6

These higher response rates, along with improved methods for including admin enumerations, likely contributed to the very high census coverage seen in 2023.

Table 6

4.2.3 Coverage by ethnic group and Māori descent

4.2.3.1 All groups

Most major level 1 ethnic groups saw their lowest net undercount since 1996 (figure 7). The Middle Eastern/Latin American/African (MELAA) and Other groups were not reported prior to 2018. MELAA was the only ethnicity not to see a drop since 2018, although its coverage estimates in 2023 were subject to particularly large credible intervals and as such should be interpreted with caution.

Figure 7

A large decrease in net undercount was observed in 2023 for people of Māori ethnicity and for Pacific peoples. For Māori, the net undercount was 1.6 percent in 2023, down from 4.4 percent in 2018. For Pacific peoples, the net undercount was also 1.6 percent in 2023, down from 4.9 percent in 2018. This finding is consistent with the continued focus of Stats NZ to deliver high quality data for these groups . However, for Pacific peoples, a large contributor to the drop in net undercount was the higher levels of overcount - 1.8 percent compared with 1.2 percent for the whole population.

Figure 8

4.2.3.2 Māori descent

Coverage rates for people of Māori descent were similar to those for people of Māori ethnicity, with the net undercount falling from 4.2 percent in 2018 to 1.5 percent in 2023. Māori descent is based on whakapapa while affiliation to the Māori ethnic group is a self-determined cultural affiliation. The net undercount in the 2023 Census for people of Māori descent and not of Māori descent is broken down by age in figure 9 below.

Figure 9

The PES uses the electoral concept of Māori descent, which is used to determine electoral populations. This differs from the census concept, which allows respondents to state that they don't know if they are of Māori descent. The 'don't know' responses have been imputed to 'Māori descent' and 'No Māori descent' for PES. See Māori descent electoral - 2023 Census: Information by concept - Stats NZ DataInfo+ for more information.

4.2.3.3 Ethnicity by age

People of European ethnicity tend to have higher coverage rates than other major ethnic groups. Net undercount rates for people of European ethnicity remain lower across all age groups than for people of other ethnicities (figure 10).

Figure 10

4.2.4 Coverage by age group

Young adults (15-29 years old) have traditionally had the highest net undercount of all age groups. However, in 2023 it was 1.4 percent, down from 4.7 percent in 2018 and 4.8 percent in 2013. The 0-14, 15-29, and 30-44 age groups had similar rates of net undercount in 2023, within the credible intervals of each other.

Figure 11

Due to the large uncertainties in the estimates for people of another gender, we are excluding this group from subsequent graphs as comparisons may be misleading. Any direct comparison of the sex/gender variable estimates between 2023 and previous years must be made with extra caution. See Gender - 2023 Census: Information by concept - Stats NZ DataInfo+ for more information.

The broad coverage trend of males having higher net undercount than females continued to be observed in 2023. The net undercount for males was 1.5 percent, compared with 0.8 percent for females. However, this trend does not hold for all subgroups - for example, males do not have uniformly higher net undercount than females across all age groups.

Figure 13

The 45-64 age group stands out, with females showing a significant difference in coverage rates compared with males - 0.1 percent net undercount for females compared with 1.4 percent for males. When this is broken down into gross undercount and overcount , we see that levels of miscount in this population are not actually small. The driver behind the low net undercount of females aged 45-64 is not so much a reduction in gross undercount (9,600), but instead a high amount of gross overcount (9,100).
 
Figure 14

4.2.6 Coverage by geographic area

In 2023, PES results identified two regions that had particularly high or low net undercount - Auckland and Hawke's Bay, respectively.

4.2.6.1 Auckland

The Auckland region had the highest rates of net undercount in 2023 at 1.4 percent. However, all Auckland local board areas saw a reduction in net undercount compared with 2018 .

Results from 2018 PES showed that coverage patterns varied widely between these areas, with high net undercount concentrated in the southern local boards. Similar patterns were observed in 2023, although with a slightly more even coverage distribution.

Figure 15

Figure 16

Text alternative for Figure 16, Net undercount (percent) by territorial authority and Auckland local board areas, 2023 Census
The map shows the undercount percent in the 2023 Census, for all territorial authorities and Auckland local board areas. Each territorial authority or local board has a colour to represent the undercount. The colours are grouped into net undercount of: less than 0.8 percent; greater than or equal to 0.8 percent but less than 1.0 percent; greater than or equal to 1.0 percent but less than 1.2 percent; greater than or equal to 1.2 percent but less than 1.4 percent; greater than or equal to 1.4 percent but less than 1.6 percent; and 1.6 percent and over. The net undercount percent ranged from 0.4 percent in Napier City to 1.8 in Ōtara-Papatoetoe Local Board Area.

4.2.6.2 Hawke's Bay

Point estimates of coverage in Hawke's Bay show a very low net undercount of 0.5 percent . However, credible intervals around this estimate are large, at -0.7 - 1.8 percent.

The main driver of this uncertainty is the large number of census forms received after the start of PES collection operations, likely a result of Cyclone Gabrielle. These late forms are excluded from coverage measurements as they may have been influenced by PES activity.

See 2023 Post-enumeration Survey: Standard design for coverage estimation for more information.

Figure 17

For cyclone affected areas such as Hawke's Bay and Gisborne, the PES undertook some mitigation steps to reduce the impact of low PES response on the quality of PES estimates. The sample size in these areas was increased, collection methods were targeted (for example, through the use of incentives), and potential adjustments to the estimation process were investigated. These mitigations, alongside a robust linking process, helped to reduce the impact of Cyclone Gabrielle on PES estimates.

The low net undercount in Hawke's Bay was partly driven by a high gross overcount of 1.5 percent . Gisborne, also cyclone impacted, saw a similar high gross overcount. This trend is likely a result of relatively low census response rates in both regions, meaning that higher amounts of admin enumeration were required to increase coverage. This increases the likelihood of duplication between census and admin responses, and therefore gross overcount.

Figure 18

4.2.7 Coverage by birthplace

4.2.7.1 Known birthplace

Both New Zealand-born people and people born overseas experienced a large decrease in net undercount from 2018 to 2023. People born in New Zealand continued to have a lower net undercount (0.8 percent) compared with those born overseas (2.0 percent).

Figure 19

4.2.7.2 Unknown birthplace

In the 2023 PES, a negative gross undercount value was recorded for individuals with an unknown birthplace. This is a result of the way gross undercount is calculated.

An adjustment is made to remove records that fail linkage acceptance criteria as they do not have enough information for us to be able to be confident that they have been linked correctly .

There are very few records with an unknown country of birth and a high proportion of these have failed linkage acceptance criteria due to limited linking information. This leads to the situation seen in the 2023 PES, where the undercount of the adjustment-eligible population is lower than the count of records which failed linkage acceptance criteria. Therefore, there is a negative gross undercount in this group.

5 Glossary

achieved sample rate (ASR) - the proportion of the selected sample that successfully provides responses. It is calculated as the number of completed responses divided by the total selected sample.

admin data - data collected by government or other organisations to operate, such as the collection of tax; to deliver health and education services; or to record events such as births, deaths, and marriages.

admin enumeration - use of administrative data to add people to the usually resident census population when a census response has not been received.

causal independence - the likelihood of being recorded on one list has no relationship with the likelihood of being recorded on the other. For example, there is no structural or operational reason PES would be more or less likely to capture those records already captured by the census. One of the five assumptions used in dual system estimation.

census coverage rate - compares the number of people who were counted in the census with the number who should have been counted (as estimated by the PES).

census response - a census record that comes from a response to an individual census form, or from a paper dwelling form or online household set-up form where an individual form for the person was not received.

census response rate - the proportion of the population counted from responses on census forms, compared with the number who should have been counted (as estimated by the PES).

closed population - the population captured in list 1 is the same as that for list 2, with no members entering or leaving the population between the two captures. One of the five assumptions used in dual system estimation.

combined census model - a model that combines data collected from census forms with alternative data sources, such as admin data and historical census data, to produce the census dataset.

census usually resident population (CURP) count - count of all people who usually live in and were present in New Zealand on census night. It excludes visitors from overseas and residents who were temporarily overseas. Usual residents who were temporarily elsewhere in New Zealand on census night are included in the count for where they usually live. For example, a person who usually lives in Christchurch city and was visiting Wellington city on census night would be included in the census usually resident population count of Christchurch city.

census night population count - a count of all people who were present in an area on census night. It includes visitors from overseas and from other areas of New Zealand, but excludes residents who were temporarily overseas or in other areas of New Zealand on census night.

coverage adjustment ratio - a factor used to adjust census counts to account for coverage errors, such as under- and over-coverage. Coverage adjustment ratio is produced from the combined under- and over-coverage model outputs.

desktop canvassing - the process of cleaning and updating an address list through desk-based investigation instead of field activity. Desktop staff use a range of desk-based resources (such as satellite imagery, rates databases, and internet searching) to classify addresses and record relevant information.

deterministic imputation - a method of filling in missing response information by directly using data from linked information for the same individual.

dual-system estimation (DSE) - a statistical procedure used to estimate the size of a population by comparing two independent data sources (for example, census and PES). DSE is based on an approach, often referred to as the 'capture-recapture' technique. By identifying individuals captured in both data sources as well as those missed in either, DSE can be used to estimate the population missing from both sources.

erroneous inclusion - records that are not part of the target population for capture. One of the five assumptions used in dual system estimation.

estimated resident population - an estimate of all people who usually live in an area. The estimated resident population of an area includes all residents present in New Zealand and counted by the census, residents who are temporarily elsewhere in New Zealand and counted by the census, residents who are temporarily overseas (who are not included in the census), and an adjustment for residents missed or counted more than once by the census (net census undercount). Visitors from elsewhere in New Zealand and from overseas are excluded.

false negative link - two records that should have been linked because they correspond to the same unit (that is, they are a true match) but were not linked.

false positive link - two records that were linked in error and do not correspond to the same unit (that is, they are a not a true match).

gross overcount - people who should not have been counted by the census but were counted in error, and people who should have been counted once but were counted more than once.

gross undercount - people who should have been counted by the census but were missed.

homogeneity of capture - the assumption that all records have the same likelihood of being captured in a given list. This is not achievable at a population level, so we use characteristics (such as age or gender) to group records until they have equal likelihoods of capture. One of the five assumptions used in dual system estimation.

household eligibility status - indicates whether a household meets the criteria to be included in the estimation part of a survey. In the PES, it is used to determine which households are included in the calculation of survey response rates.

IDI spine - an ever-resident population list of New Zealand, derived solely from administrative data sources.

linkage acceptance criteria - criteria used in the PES linking process to remove records that do not have enough linking information for us to be confident that they will link correctly.

linkage error - where two records are either incorrectly linked (false positive error) or incorrectly not linked (false negative error).

linking information - information that enables us to work out whether two records refer to the same person. In the PES linking process this is name information, date of birth, address information, gender, and country of birth.

Māori descent (electoral) - concept identifying if a person is descended from Māori for electoral purposes. This differs from the census concept, which allows respondents to state that they don't know if they were descended from Māori. The 'don't know' responses have been imputed to 'Māori descent' and 'No Māori descent' for PES. See Māori descent electoral - 2023 Census: Information by concept - Stats NZ DataInfo+ for more information.

meshblock - the smallest geographic unit for which statistical data is collected by Stats NZ. Meshblocks vary in size from part of a city block to large areas of rural land. Each meshblock abuts another to cover all of New Zealand, extending out to the 200-mile economic zone (approximately 320 kilometres). Meshblocks aggregate to build larger geographic areas, such as area units, territorial authorities, and regional councils.

meshblock enumerations - admin enumerations that do not meet the criteria to be included in a dwelling, but we are confident that they are part of the usually resident population of the area.

net undercount - the difference between gross undercount and gross overcount, that is expressed as a number and percentage of the PES estimated population.

overlap controls - the process to reduce the number of PSUs (Primary Sampling Units) selected for multiple surveys. By minimising this overlap, it helps reduce the burden on respondents, so fewer people are asked to take part in multiple surveys.

over-sampling - the goal or action of sampling a particular group of interest at a higher rate than they contribute to the wider population. At Stats NZ, we use techniques such as stratification, allocation, and sampling selection methods to implicitly over-sample groups of interest. Less commonly, we may also use additional techniques such as booster panels to explicitly over-sample groups of interest.

perfect linking - error-free linking between the two lists including no missed links, and no records incorrectly linked. More information on how this is achieved in the 2023 Census Coverage project can be found in 2023 Post-enumeration Survey: Linking design. One of the five assumptions used in dual system estimation.

PES estimated population - a population estimate including the coverage-adjusted census usually resident population count, late census responses, and administrative enumerations in prisons.

primary sampling unit (PSU) - geographic areas consisting of between 50 and 100 dwellings and are formed using meshblocks (Stats NZ's standard geographic classification).

post-enumeration survey (PES) - a sample survey of households run after the census to measure the completeness (undercount and overcount) of the census count.

statistical imputation - a method for filling in missing response information using response information from another unit, known as the donor. Typically, the donor is chosen in such a way that it resembles the imputed unit as much as possible on one or more background characteristics.

ISBN 978-1-991307-61-3

Enquiries

Joel Watkins
03 964 8807
[email protected]