In 2014, then Conservative cabinet minister Francis Maude declared that it was time for the census to be replaced. The once-in-a-decade population survey, which first began in 1801 as a means of assessing how many men of fighting age were available in the country to serve in the Napoleonic wars, had become seen as something of a whimsical oddity in a world increasingly turning to big data, to acquire information about society.

For the last seven years, politicians and civil servants have repeatedly stressed the need to find a more modern alternative for the census, arguing it is too expensive, too slow, too inefficient for the needs of 21st century policymaking. A 2018 government report called for new ways of monitoring how the population is changing in real-time, while the cost of the 2021 census – estimated at around £1bn – has been labeled overly exorbitant.

In the wake of this, the coronavirus pandemic might have been expected to sound the death knell for the census. The think tank Centre for Cities called for it to be postponed until 2022, citing that Covid-19 would distort the data on working patterns and immigration levels, with many migrants having returned home for the duration of the pandemic.

But in reality, the 2021 census may prove more important than ever. When the data is made available in a year’s time, it will provide us with the first comprehensive picture of exactly how and why the pandemic has wreaked havoc in different parts of the country, across geographic zones, ethnic groups and age ranges. While transport planners may find it difficult to make decisions given the number of people working from home, the census will enable local governments to construct policies on how best to regenerate town centres devastated by lockdowns, and instruct healthcare bodies organising future vaccination programs, should Covid-19 booster jabs be required to deal with new variants of the virus.

“Some things will be skewed, but it’s absolutely crucially important that we have a decent picture of what’s going on in the country right now,” says Oliver Duke-Williams, census Service Director at the UK Data Service. “A lot of the things we’re interested in regarding infection and death rates in different parts of the country rely heavily on having accurate population counts.”

While the NHS can monitor how many people have been infected with or died from Covid-19, our understanding of exactly how the Sars-CoV-2 virus has worked its way through different households and communities across the UK, have been based on models of the country as it looked like in 2011.

Analysts are keen to use the new census data to get a more precise idea of the particular hotspots where the pandemic has been most devastating, and why this was the case. “Having that knowledge is an important part of our response to this pandemic,” says Duke-Williams. “For example, the census data will give us a good idea of how many households have multiple generations living under the same roof, and where they are distributed. That will allow us to look at parts of the country where the pandemic was driven by the mixing of age groups.”

The data will also play a critical role in the reshaping of care and healthcare systems over the coming years to try and address the many inequalities which have been laid bare by Covid-19. “We will know a lot more about the vulnerability of different population subgroups, and particular neighbourhoods because of their social composition,” says David Martin, professor of geography at the University of Southampton, who has spent more than two decades working with census data. “This is going to be used to adapt the way services are provided, in case this happens again.”

Martin explains that the level of detail provided by the census makes it particularly valuable in the aftermath of a pandemic. While it comes with a high price tag, and can be subject to manipulation by pranksters – during the 2001 census, 400,000 people chose to identify as Jedis, much to the chagrin of census analysts – it remains the single most important means of gathering information on kinds of subjects ranging from ethnicity to the number of people who provide informal care for family members, and the rise of single parent households. As such, it will form the basis of almost all funding decisions made by central and local governments over the next decade, from social care to school allocationss, and hospital beds.

“The use of the census data is actually pretty widespread,” says Guy Goodwin, chief executive of the National Centre for Social Research. “In terms of resource allocations and planning, from the fire service to the police, just about anyone you can imagine uses that population data in some shape or form. Continuing with the census for now is very important as you need that evidence base to underpin everything.”

The main downside of the census, highlighted by the fact that many decisions made during the pandemic were based on the 2011 population, is perhaps its infrequency. Duke-Williams points out that population patterns are shifting faster than ever before, and having to wait a decade until that information is updated is far from ideal.

As a result, the Office of National Statistics (ONS) – the government department responsible for planning and carrying out the census – has spent considerable time investigating whether it would be possible to replace the census with continuous streams of big data, from sources ranging from tech providers to the NHS.

The idea is that these different sources could be stitched together to create a giant patchwork picture of the population, in a similar manner to the census, except one which updates in real time. However in reality, there are a myriad of practical hurdles. For example, while the NHS register, and other datasets compiled by the Department for Work and Pensions, and HMRC, cover a sizeable sector of the population, there are problems with trying to join them together.

“The NHS register is about four per cent too large and that’s because people either leave the country, move between GPs, and get left in the system when they’re not really there,” says Martin.

In addition, while the NHS database contains information on the age and gender of the population, and pensions, tax and benefits data records basic information on employment status, this still misses out critical details captured by the census such as ethnicity, the structure of families living at the same address, and the qualifications of the workforce.

Over the past four years, experiments have been conducted using mobiles phones locations data to try and replace the segment of the census which examines how the UK population travels to work, but with relatively limited success. “It gives you an aggregate pattern,” says Martin. “But it’s not connected to the detailed data about individuals. You get a broad brush picture of traffic flows and people’s movement which is still useful but it doesn’t give you any intelligence about whether people of different age groups are particularly disadvantaged in a certain part of the country due to mobility, or whether people working in a particular industry are travelling very differently.”

In 2023, the ONS will make a decision on whether to proceed with the census in 2031. One cost saving option which has been considered is to switch to a system similar to the Netherlands, which combines the use of big data with a survey of a proportion of the population to obtain estimates similar to the census. However, the task is made easier by the fact that all Dutch people have a national identification number, a concept which has never been popular in the UK.

If analysts tried to do the same in the UK, linking mobiles phones records, NHS details and other datasets for large segments of the population, it could potentially raise concerns around data privacy and security. “The census is very straightforward as it is governed by specific legislation,” says Martin. “There’s a really strong confidentiality commitment about your individual response. No analysts ever see what is written on your form. The difficulty with the alternative data sources is that you need to work out some common identifiers to link them together, which makes it a lot harder to regulate.”

As a result, while many predict that the census will ultimately be replaced, it is likely to be a while yet before big data is ready to take over. “I think ultimately they will get rid of the census,” says Goodwin. “I’m not convinced yet whether they’ve got good enough quality data to do it for 2031. They’ve got to be pretty confident that they can generate the same level of information by linking this to other data sources, and I’m not certain they’re there yet.”

This article was originally published by WIRED UK