Last week I featured on David Aaronovitch’s Radio 4 programme The Briefing Room, in an episode which asked the question ‘Is the UK in a data crisis?’. My role was to talk about the history of data collection up until the post-war period when mass social surveys took off as a staple of official data-gathering.
The 1960s to 1980s were something of a golden era for social surveys. The post-war vision of a data-driven, planned economy meant a lot of information had to be collected which could only be obtained by asking people. During the Second World War, the National Register gave us a fairly accurate live database of the nation’s manpower as well a simple indication of where people were. You can imagine the feeling of ‘flying blind’ when the Register stopped being updated in 1952.
Another census came in 1961 but it was felt to have become quickly out of date, such was the perceived pace of population change. In a fit of panic, we carried out a ‘ten per cent’ census in 1966 (those of you who’ve read Sum of Us will fondly remember the ‘ten percenters’) but it was still felt that censuses were too cumbersome and too blunt of an instrument to satisfy our data needs.
Surveys came to the rescue, offering a more detailed picture of what was going on in people’s work and private lives albeit at a smaller scale that could give us estimates of how the national version looked.
In quick succession, we got the Annual Survey of Hours and Earnings (1970), the General Household Survey (1971) and the Labour Force Survey (1973). These gave us vital information on what exact type of work people were doing, how many hours they worked, what combinations of benefits they were on, and what their income and expenditure looked like, along with other characteristics of households and the people in them. The data gathered through these kinds of survey is far richer than that we get from a census and certainly more insightful than administrative data. The only caveat is that the right people have to respond to a survey, and in the right numbers.
From golden era to a lead balloon
The response rate to the Labour Force Survey (LFS) when it first started was typically north of 75% and in the early 1990s it was consistently around 85%. After that began a steady decline which takes us to the present day and to The Briefing Room’s question about UK statistics being in crisis.
Also on the programme were Denise Lievesley (of the Lievesley review into UK statistics oversight), economist Chris Giles, and Peter Lynn, Professor of Survey Methodology at the University of Essex. The prognosis was fairly bleak: response rates have collapsed for all sorts of reasons, at the same time that the people in charge of UK statistics seem to have been asleep at the wheel or at least very distracted from the road.
I was pleased to be reminded by Chris Giles of this extraordinary quote from the Chair of parliament’s Public Administration and Constitutional Affairs Committee (PACAC) about the ONS’s management under National Statistician Sir Ian Diamond, who departed in May of this year:
“What I find slightly peculiar to understand is that in a modern, social media, sunlight-is-the-best-disinfectant type approach to life by which Ian Diamond appeared to have run this organisation, he was a sort of hybrid of a Medici prince and Blofeld, and seemed unwilling or uninterested in anything anybody had to say. He managed to pull wool over the eyes of Ministers and the board, saying that everything was hunky-dory and innovation was going to be the great salvation to all our prayers.”
PACAC is the very best of select committees. And Blofeld is that Bond villain with the cat. I had to look it up (he doesn’t look like a Blofeld, but I’ve also been corrupted forever by Austin Powers.) Suffice to say, Sir Ian’s tenure is being roundly blamed for allowing response rates to plummet, and those advising him could have also sounded the alarm on this much earlier on.
Still, a look at the most recent data on response rates and the quality of the LFS sample suggests that things are looking up slightly. The ONS has been trying to address the LFS problems since 2023 and made some more changes following the critical Devereux review in June.
Firstly there’s the response rate.
Wave 1 response rates are still stagnant, but response in follow-up waves is rising, which really helps the quality of the pooled data. Still, we are nowhere near the overall response rate from ten years ago (around 47%) let alone the 85% of the early 1990s.
Strangely, the number of person interviewed seemed to really suffer in 2023, after the pandemic, but to also be recovering now:
Further to this, the ONS said, “we have seen Wave 1 response levels [of people interviews] almost recover to pre-coronavirus (COVID-19) pandemic levels” although I wasn’t able to find the full dataset to back that up.
It’s not a rosy picture, that’s for sure – but we can see some things changing and that gives us an idea of where the main problem areas are now. What’s particularly enlightening is the latest data on reason for non-response.
Non-response or outright refusal?
In The Briefing Room, my fellow panellists talked about the difficulty of making contact with the people who’ve been sampled for the LFS.
Because the sampling frame is not actually of people but addresses, the letters which go out asking for someone’s participation are addressed to ‘The Occupier.’ Understandably, many people will bin these or at least be suspicious. The ONS – and other statistical authorities around the world who are in the same position – see it as a priority to figure out new ways of reaching people. There is talk of using mediums other than letters and phone calls to reach younger people, since they’re considered to be particularly wary of these formal approaches.
But a look at the data suggests that the main problem is no longer ‘non-contact’ but people being contacted but refusing to participate. I’m not completely clear what would count as ‘outright refusal’ but the explanation (in the chart notes) seems to suggest that it’s when interviewers go in person to a sampled address, after a letter has been sent, and the person opening the door essentially tells them that it’s never going to happen. This is clearly a major problem – possibly a trust issue, possibly something to do with enumerator training – that needs attention.
What’s really striking as well is that over half of refusals in recent quarters have been ‘refusals to HQ.’ This is where people open the letter, read it, go to the webpage or dial the number on the letter and tell the ONS that they do not want to respond. Again, clearly not a problem of physically reaching people but perhaps more one of trust and incentives.
Earlier this year, the ONS trialled offering some of the sampled households a £20 voucher to respond, up from the £10 that is sometimes offered, and found that this did boost response rates. Whether this could go higher, for a survey that can take hours to complete for a full household, is a question that probably needs to be asked.
There is also always the nuclear option of making the LFS compulsory, like the census, although this feels like a particularly bad time given the public reaction to the idea of being ‘forced’ to submit data in the context of ID cards.
I’ll leave it there for today, although there is more to say on LFS data quality which I’ll try to get to in a future post.
Originally published on Substack: https://georginasturge.substack.com/p/are-there-cautious-signs-of-recovery