The Office for National Statistics (ONS), the UK’s main producer and quality-controller of official data, turned 30 this year and with that milestone came some soul-searching. A year earlier, the agency had been at probably the lowest point in its history, marked by the abrupt departure of its director, National Statistician Sir Ian Diamond.
An official review carried out in 2025 identified ‘deep-seated issues’ with performance, culture and morale at the ONS, and led to the National Statistician post being split into one role to lead on statistics and another to run the business side of things. A new permanent secretary was appointed for this second purpose.
The temporary restructure seems to have borne fruit, with sensible decisions being made to focus resources on the biggest problem areas. There’s also now a noticeably more humble attitude in how ONS interacts with external experts and the public, in contrast with the defensiveness of the past. And yet problems remain with the statistics themselves.
Now André Loranger, the current Chief Statistician of Canada, has been appointed to fill the leadership role long-term and should be taking up his position in November. It seems like an excellent idea to have someone external come in, and particularly someone whose experience couldn’t be a closer match to the role.
Loranger won’t be the first foreign national to head up UK statistics – that accomplishment belongs to Claus Moser, who was appointed as Director of what was then the Central Statistical Office in 1968. Moser was an interesting character, having come from Germany to the UK as a teenager to escape Nazi persecution and learning statistics from a maths professor with whom he was detained in a Liverpool internment camp for ‘enemy aliens’ in 1940. But we’re getting way off track (go read Sum of Us for more historical stats trivia).
One question which Loranger will have to deal with right away is whether day-to-day running of the ONS should be restored to him as National Statistician or if it should continue to be carried out by a permanent secretary. The technicalities of who is responsible for what are probably not very important – what will matter is whether Loranger has a vision for the ONS and whether he’s able to rally the organisation to work towards that.
The big problem which urgently needs addressing – and for which we need that vision – is the decline in response rates to our Labour Force Survey (the LFS) and its extended version the Annual Population Survey (APS).
Non-response to the LFS is still way too high. In Canada’s equivalent, the response rate was around 73% in 2024 and this was considered low in comparison with the 87% response rate they achieved in 2019. Such rates are currently unthinkable here. The UK’s average LFS response rate in 2024 was 19%, down from 39% in 2019. I don’t know when we would last have hit 87% – possibly in the late 1980s/ early 90s but indeed it could be much longer ago.
Household surveys became a bit of an unsexy topic under the previous statistics administration, headed by Sir Ian Diamond. In this era of mass data held by government departments, modern thinking was that bureaucratic data was more accurate, efficient, and realistic to gather. And with those hopeless-looking response rates, you can understand why some people would conclude that surveys were a thing of the past – perhaps even beyond revival.
The fact is though that we still deeply dependent on the LFS and on other household surveys in a great many areas of important decision-making. Yes, we can point to improved administrative data within HMRC, or the DWP, or the NHS, which we’re now also using to generate core statistics. But look a little deeper into how most estimates are produced and survey data will still be in there somewhere, often playing a more prominent role than producers would like it to.
The general problems with the LFS are well-documented. Low response means the survey has been inaccurate for many years in telling us about the number of people in and out of work which, for a labour force survey, is kind of its core purpose. In practice, we also used to use it to learn about much more than economic activity. The LFS/APS also used to tell us with relative accuracy what the population was of local areas, and where people were moving from and to. This was pretty essential for planning, and there’s a void left while the ONS is still trying to wrangle administrative data to tell us the same thing.
But new problems are also still revealing themselves. This September, the ONS had to admit that using the LFS in its weakened state had likely led to them over-estimating the number of hours people worked. This had been skewing one of our important economic indicators for years.
Productivity figures going back to the Financial Crisis have been revised, with the ONS now estimating that productivity per-person-per-hour improved by 1.3 per cent from 2009 to 2019, more than double the 0.7% previously estimated. This completely rewrites the narrative on that particular indicator, and must have absolutely enraged economists, journalists, and policy people who spent years ruminating over the UK’s ‘puzzle’ of low per-head productivity compared with other countries.
Many of these other countries were already using an improved method to estimate hours worked. We were not only estimating it poorly but letting the problem worsen as the number of people answering our crucial labour force survey dwindled to almost none. I say this a lot, but this really was a classic case of bad data.
The LFS is not the only survey where we’ve seen a decline in enthusiasm from the people – or indeed businesses – selected to participate. The Living Costs and Food Survey, which supplies data used in the calculation of GDP, has suffered from a dip in response rates in recent years. So too has the Annual Survey of Hours and Earnings (ASHE), with response falling from roughly 59% before the pandemic to a low of 47% in 2022 and climbing back to 54% since.
This should worry us too, given that it is ASHE data which is used in the ONS’s new model for calculating hours worked, and which was used to conclude that the LFS was completely wrong on this front. We’re replacing bad survey data with slightly better survey data – but let’s not kid ourselves that any of it is likely to be fully accurate at this point.
If we want to carry on relying on survey data so prominently – and we do still need it, given that the administrative data alternatives are patchy too – its quality must be improved.
Under Diamond’s leadership, there was a strong ideological swerve away from surveys and towards using data gathered for administrative purposes. Some really beneficial work came out of this, but it’s clear now that the UK admin data landscape is still too fragmented for this data to be an improvement on surveys in most policy areas.
What Loranger should pick up on – if he’s not already well aware – is that the British public also don’t feel very warmly about what might be perceived as underhand ways of collecting and using their data. The backlash last year to the now-abandoned plan of having digital ID cards suggests that many of us don’t like being in administrative lists, even when there’s convenience and benefit in it for us. The viciousness with which many people oppose efforts to repurpose NHS data for research is another sign that we don’t view the linking and plundering of public service datasets as the innocent and purely bureaucratic exercise that some in the statistics world would like it to be seen as.
It’s going to be a tricky one for Loranger to puzzle out. We don’t like answering surveys, but we’re also not big fans of having our administrative records used for analysis behind our backs.
In my view, restoring survey response rates is the priority. Three quarters of non-response to the LFS is currently down to people being successfully contacted but refusing to take part, which I think is quite telling. It’s not an issue of people being hard to reach in the era of mobile phones. We can be contacted, but our willingness to cooperate is just not there.
It’s up to Loranger now to figure out why that’s the case and how to get it back. Of course though, it’s not like the ONS hasn’t been trying to do this already for many years. There are a range of levers which could be tried, chief among them adapting the survey, increasing the monetary incentive to participate, or making it compulsory.
But if someone who currently presides over comparable surveys in Canada which are hitting response rates of 73% and above can’t set us back on the right path, I don’t know who can.
Originally published on Substack: https://georginasturge.substack.com/p/dear-national-statistician