This week on the podcast, Ed Humpherson of the Office for Statistics Regulation (OSR) talked to me about a crisis in UK statistics which has not only hit data quality but damaged public trust. Find the full episode here on Spotify (also available on Apple podcasts) as well as the full transcript below.
If you have any suggestions for people I should invite on the show, let me know in the comments!
TRANSCRIPT
Georgina:
Ed Humpherson is head of the Office for Statistics Regulation, an independent regulator and the official watchdog tasked for quality and transparency in UK statistics. Ed’s background is in accounting and, before heading up the OSR, he worked at the National Audit Office, where he managed his work on economic and business affairs. Ed sits on various different boards, including that of the Regulatory Policy Institute and is a regular expert witness at the House of Commons.
We first met several years ago at the Houses of Parliament where we were both speaking on a panel about misinformation. Ed Humpherson, welcome to the show.
Ed:
Hello, it’s really good to be here.
Georgina:
Now Ed, I mean this in the nicest possible way, but in a world where you could be anything, why is statistics regulation? Why do you do what you do, Ed?
Ed:
I do it because I love it, I suppose, is the glib answer, but let me unpack that and give you a bit of an understanding of what makes it such a great thing to do. If you look at anything that we do as OSR, you’ll see we bang on, we drive home this point about the public good provided by statistics. That’s because whenever we encounter a new topic where statistics come up, we can see that actually statistics are this public asset that are available for citizens, for charities, for communities, for companies, for all of the actors in society to kind of understand what’s going on in the world. every, so everything we do has this sort of touchstone to this real sense of serving the public good, serving the public. And that’s really, really wonderful.
Sort of implicit within what I’ve just said as well, that it’s an incredibly varied job. In the morning, it might be water pollution in the afternoon, might be migration. The next morning, it might be tax policy and in the afternoon, it might be mental health. It’s an incredibly varied and wide range of topics and that’s really stimulating. I think the other thing which, I don’t know, gets me out of bed in the morning is a sense that what we aim to do is to propagate a philosophy. A philosophy that good use of statistics, trustworthy, high quality, high value statistics is
really an important thing. And I think that philosophy is quite, if I say inspiring, I don’t know, maybe I’m beating myself up too much. It’s quite motivating. Let me put it that way.
Georgina:
People listening might not really be aware of what the OSR is or that there is a regulator for statistics and some have described it as kind of like Ofsted or Ofwhat but for data and statistics. Can you tell me a little bit more about why we need it and what it’s been able to achieve in the decade I think since it started?
Ed:
So let’s start by saying what’s the scope? Government produces a lot of statistics that it makes available. And that’s not just the Office for National Statistics doing the big things like inflation and unemployment and the size of the economy. It’s all government departments ranging from the very largest like the Department of Work and Pensions doing statistics on poverty all the way through to quite small bodies. The Forestry Commission talking about the size of the UK’s forest base.
It’s a task of government to make information available. What we do is set the basic standards that any government department anywhere in the UK can do that to an appropriate standard. We do that through a code of practice, which really is the standards, the basic requirements that they must follow. And then my team, members of my team will undertake various kind of reviews to check that those standards are being followed in particular. cases. So if you think about it in very simple terms, we set the standards and then we follow up those standards.
You made a comparison with an organization like Ofwat. Really interesting distinction between us and them is that we don’t have very strong powers of compulsion. We can’t fine a government department that does bad statistics. We can’t compel them to produce statistics. We can’t strike anybody off the register of statisticians like the General Medical Council. But what we can do is we can use our voice and our influence. And what’s really striking to me, you said, have we achieved, how often when we make an intervention, we look at a particular topic, the government department wants to follow what we say because they want to avoid criticism by us. They want to stay on the right side of us. And that’s a huge privilege to be that influential.
Georgina:
What’s a recent example of where you’ve intervened?
A really good recent example I mentioned water pollution. So fairly recently, the Secretary of State for the Environment for England made a statement that water pollution in Scotland was worse than water pollution in England. His team collected a few numbers which appeared to support this. But if you scrutinize the issue as we did, you’d see that actually the whole range of statistics which compare water quality and water pollution. And the picture is really not clear at all that Scotland has worse water pollution than England. If anything, Scotland has probably better water quality, certainly equivalent water quality. So we stepped in and we high-lou- we didn’t say, you you’ve got this completely wrong, you’re an idiot. We weren’t dismissive about it. We laid out the evidence in a very straightforward and clear way. And that enables the public debate to be undertaken appropriately. That’s a small example. But we do that all the time, that’s clarifying the appropriate interpretation.
Georgina:
And is that more commonly what you have to do rather than to say this is just outright wrong or you you’ve got it wrong? Yes. It’s more that you often find yourself saying your statistic may be one interpretation, however there’s a lot more context or the picture’s just not clear enough for you to say what you’re saying with certainty.
Ed:
Yes. In fact, by far and away, the most common intervention we make is that a statement has been made and an evidence base on which the statement is made is not clear. We talk about intelligent transparency. What we mean is that a statement is made and it’s not possible for a citizen to make intelligent sense of what’s been said because there’s no sources, there’s no context. It’s just a kind of naked number. If you look at our track record, we’re very often making interventions to say the government department should make the underlying data available so that a citizen can make intelligent sense of this thing which has been claimed. So yeah, that’s by far the most common. It’s actually, there are examples of us saying that’s wrong, but it’s quite unusual for us to say that something is completely wrong. Usually because there isn’t a kind of completely clear-cut case of an erroneous statement. It’s usually much more either the interpretation or the transparency.
Georgina:
Now I wanted to have you on the podcast because you sent me a very intriguing speech that you did recently in The Hague and that was at a statistics conference, not at the War Crimes Tribunal. you. The speech was about leaving shoes outside the house and we’ll come onto that in a minute. I wanted to talk to you more broadly about what’s being referred to by some people as a crisis in UK statistics. So what’s happened slowly in recent years is that the quality of some of our important datasets has declined. And a review this year, earlier this year, found that there were cultural and institutional problems at the Office for National Statistics, the ONS, and there have been a series of mistakes that they’ve made in recent years and things that they’ve had to revise. The chief statistician, Sir Ian Diamond, who resigned earlier this year has been attracting quite a lot of quite harsh criticism in a recent select committee hearing which you attended as a witness. So Ian was likened to a cross between a Medici prince and a Bond villain, or at least that’s how some people have been making him sound. Ed, what’s going on?
Ed:
I was, as you say, was in the room when that quote came out. It was by the chair of the committee. He used this quote. He actually said a cross between a Medici prince and Blofeld. Blofeld is one of the main villains in the bond, Ian Fleming bond works. And, but when I first heard it, I, I’m a cricket fan, there’s a very famous cricket commentator, now a side called Henry Blofeld. It was a reference to that. And I almost made a joke, you know, on the floor of the committee.
I’m rather glad I didn’t because I don’t think it would have been a place for any jokes. Do you know who Henry Blofeld is? He’s his very, very colourful, posh, very posh cricket commentator. If you think of English radio cricket commentators and they’re always going about, lovely cake, he’s just the icon of that. So when it first came up, and then I thought, that’d be a funny joke to make.
Georgina:
It doesn’t sound much like Ian Diamond, let’s put it that way.
Ed:
To be honest, he’s definitely not like that Blofeld. That’s for sure. There you go. There you go. So yeah, there’s definitely, you said it is a crisis. That’s not too strong a term. I don’t recall from that. There is a crisis. In fact, the speech that I gave that you referred to, I talked very openly about the crisis in UK statistics. I think there’s two things going on at the same time. One is there’s a kind of a broad societal change which has led it to be harder for the Office for National Statistics to get people to respond to surveys. That’s been going on for some time. It’s happening in many different countries. It’s probably happened more acutely in the UK, to be honest, but it’s societal rather than something specific about the ONS.
The question then is, well, how does ONS respond to this? They depend on getting information back from citizens a very wide range of their statistics, how do they respond to that decline? Essentially, their response was maybe not urgent enough and was not effective. That’s the first thing. The second thing, I think, is a broader cultural feature. I’m not speaking out of turn. This is what I said at the committee when I was under oath as a witness, that the ONS has had quite a closed institutional culture in general, which has tended to be quite defensive. What that has meant is that it quite often will be reluctant to really recognize the scale of the challenges it faces. Sometimes I think there’ll be a feeling of emperor’s new clothes, that they will do something, that will introduce a new method. They’ll say it’s the best method ever without really outlining the potential weaknesses or downsides.
And then when those weaknesses or downsides manifest, stakeholder confidence declines. And I think what really happened in this whole period was a toxic mix between that challenge, the declining quality of the surveys with this culture that I think has tended to be a little bit on the defensive side. And I think the resolution at ONS has got a number of plans now, recovery plans and so on. I think it depends on resolving both of those problems, not just one.
Georgina:
Do you have an opinion on which area of data is worse than any of the others or I suppose the worst at the moment? Or we don’t want to put it in a kind of judgmental way, which one still has the most challenges?
Ed:
I think for ONS statistics, I think there’s a whole swathe of household surveys, surveys of households that continue to be really challenging. And the statistics which are based on those, think the, let me just clarify one thing. When I talk about response rates, that’s an indicator. The thing which is really the problem is representativeness. Does the survey represent the population if you’re getting responses from one kind of, one segment of the society and not from other, you’re only representing that segment, it’s really representativeness is a problem. And I think there are a series of surveys that ONS is doing where they do have this representativeness challenge. And they’re now very clear about it. And it’s really in the open and they have good recovery plans. But I’d say all of those are problematic. By contrast, actually, some other areas are less affected. So business surveys going out to get data from companies that has held up better, at least in part because there’s more obligation on companies to respond to the surveys.
ONS is doing some innovative things on migration that I know you’re very involved with Georgina. And I think they’ve really moved away from surveys completely now. And that’s real progress. And they’re doing some interesting things on prices to just gather much more automated data, supermarket scanner data in particular. So there’s lots of sort good things. I think the area of weakness, come back to your question, is, what’s the most troublesome area, is that anything which relies on the household surveys.
Georgina:
And you said that we’re doing a little bit worse than some other countries. Do you have any opinion as to why that is? What are your ideas about why that might be?
Ed:
Well, I am now going to be speculating because I haven’t done a kind of in-depth piece of analysis of other countries. I couldn’t see the headline figures. So just to be clear, the headline figures, if you look at survey response rates, they are all in decline everywhere. But the trend line is the same. It’s just that the UK trend is to a lower level. My speculation, Georgina, would be that it’s about the… It’s partly to do with social acceptability, cultural factors about what people regard as being their individual responsibilities to the state and their relationship with the state. There’s probably also some kind of execution factors in how households are approached, how they get kind of like the fieldcraft of collecting information from people in surveys. And it’s also worth saying in some countries, particularly in Europe, there’s a really, really extensive system of administrative data that provides much more information. Holland, Scandinavian countries have much more administrative data that is easier for statisticians to access.
Georgina:
I want to come back to that administrative data in a bit. First, I’d like to talk about this, big idea about the shoes. So in the speech that you gave in The Hague a few weeks ago, you talked about an idea that had come to you when you saw a pair of children’s shoes sitting neatly on the street outside a house. And this idea was not, my God, a child has been abducted. You thought of something else when you saw those shoes. What was that?
Ed:
When I first saw the shoes, I didn’t think a child had been abducted, but I did think a child had lost their shoes and somebody had left them there in case the parent came back for them. But then I realized that it was part of a wider phenomenon that I see in the part of London where I live, which is people leaving outside their house a box of things they don’t want anymore. So it might be a pair of children’s shoes or it might be scarves or crockery.
I saw a lampshade, know, somebody just left the lampshade outside their house. People leaving things just right outside their house that they don’t want. the thought that I had was years ago, if people had unwanted stuff like that, they would probably take it to a local church or a local community group for that community group to distribute. would be, they would go to an institution and then they would sort of get rid of their unwanted stuff that way. But now they just leave them outside their house, the nature of people’s relationship with each other is maybe changing. That maybe people are now more interested in sort of horizontal relationships of equal as opposed to going up to an institution which acts as a clearinghouse.
I developed that into an idea of saying statistical producers like the ONS and their equivalents around the world need to be very thoughtful about that change in dynamic. That if they present themselves as experts on a pedestal full of competence and technical skill and integrity, they’re going to seem remote and distant and people aren’t going to have confidence in them. They need to find a way of being on the same eye-to-eye level as the people they’re serving. That was basically the idea.
Georgina:
Interesting. this kind of like peer to peer interaction is happening more now than people trusting institutions and going via them. So what you’re saying is that the ONS needs to kind of come across as a friendly neighbour rather than a distant part of the state infrastructure. Is that kind of shift in perspective realistic over what timeframe and how do you think that could come about?
Ed:
I think it is really realistic. it’s got a push and a pull element to it. The push element, statistics producers, not just the ONS but anyone, the push factor is what they do to present themselves. The institutional architecture is we do these very formalized press releases that are in fairly dry language that are cold.
I think those things probably don’t really engage very effectively and they don’t really do that eye to eye interaction. And then the pull factor is when people come to them, how do they respond? And I talked earlier about defensiveness. think that the defensiveness or the saying that we’ve these really excellent statistics, which are new and improved, just is the wrong, wrong approach. I think it’s much better to say, we’ve got these statistics, we know their flaws, they know their limitations, here are their limitations and here are the ways in which you as a user can help improve them. So it’s very much about a series of stances, I think, that can shift and those stances give a signal and they give the signal of openness and vulnerability.
Georgina:
Vulnerability, I’m glad you used that word because that was going to be my next question. So yeah, you’re describing the producers of statistics becoming a bit more vulnerable and admitting to flaws. But when we’re talking about the producers of statistics, we’re not just talking about the ONS, we’re talking about government departments, which have quite big responsibilities in people’s lives. We’re talking the NHS, we’re talking even the police. So when you’re saying that institutions like that need to make themselves a bit more vulnerable and open and admit flaws. What exactly do you mean by that? How do you see that working in practice?
Ed:
I think firstly, it’s the openness to recognize when something hasn’t gone right, the way in which errors are recognized and communicated. It’s being really honest about uncertainty. And above all, maybe I should have emphasized this more in my first answer, it’s about demonstrating user engagement, demonstrating a willingness to provide avenues for people to come in with their feedback and their commentary, creating forums.
And I always think the thing about those forums is, you know, public dialogues and user engagement. Of course, it has an impact on the people who participate, but it also has an impact on the people who don’t participate but see it happening. So they see the organization giving a signal of openness. I think those things are entirely credible for all organizations. There is a real movement actually, certainly in the UK, of emphasizing the value of public engagement and public involvement growing, it’s burgeoning, it grows out of the communication of science. That’s the sort of the discipline where it emerged from. think it’s really powerful and it is available to all organizations. And I actually sort of disagree with the premise of your question, Georgina, because you said, you know, this is the ONS could do it, but also for government departments, it might be harder because they’re, you know, they’re politically led. I actually think if the ONS does it and gets it right, they will lead the way and it will become much more of an accepted norm of how organisations which face the public interact with the public.
Speaker 2 (21:33.112)
What I was thinking in bringing up institutions like the police is that they need to have a certain level of authority, right, in order to do kind of their core function, in that case, law and order. But you’re talking here about openness and putting hands up and admitting mistakes. Isn’t there a risk, and I’m just pushing you on this, I love the idea of there being a lot more user engagement in this eye-to-eye thing, but isn’t there a risk that in looking too human, the public and maybe politicians as well might lose even more faith in these institutions.
Ed:
Yeah, I don’t belittle that risk. think that is a real factor. actually when I gave the speech, which wasn’t at the International War Crimes Tribunal, but the World Statistics Congress, when I gave that speech, that was what some of the audience questions sort of pointed to, which is actually by sharing vulnerability, don’t you make everything worse? And I don’t dismiss that. I think it’s worth being really thoughtful about that being a risk.
And perhaps undertaking this move towards vulnerability and openness, not in one massive big bang, but in a series of steps. You do it in an incremental set of steps, precisely to give yourself the reassurance about that risk. Because the reason I say incrementalism will help here is I think the point about openness is that if this was a one-shot game, if you just were doing it once, making yourself open probably does make you more open to criticism and might weaken things.
But it’s not a one-shot game. It’s a repeated set of interactions between the producer and the user. And over time, that openness to repeated interactions generates trust. But you have to do that incrementally. I would certainly not encourage any organization to just sort of read my speech and then suddenly lurch into this new thing completely in one go. Because I think that would generate risk. I think it’s a much more incremental process.
Georgina:
I want to ask about something that you raised near the beginning of our discussion about administrative data. So this is something else that the ONS and others in government are trying to do and that’s use more administrative data and have less reliance on surveys. So how does that fit into your vision for trust-based statistics? Does having more and better admin data simply bypass the need for the public to cooperate with surveys and the like? And is that a good or a bad thing?
Ed:
I definitely don’t believe it obviates the need for this engaged approach. There are two reasons for that. One is certainly in the UK, the public needs to be reassured that administrative data that are kind of records of their own lives are being used appropriately and safely and respectfully. And you can’t do that as a producer without having this openness and this awareness of public attitudes and sort of what is sometimes called in the literature, the social licence. You’ve got to take all that on board to have the licence to use people’s data.
And the second reason is, of course, administrative data, you don’t need to go to somebody’s house and knock on the door and look them in the eye in that sense. But you still produce an output. You still produce an output which says the population of England and Wales is this many people, the migrations of the UK is this many people. And those outputs need to be outputs that people have confidence in and there’s no way of generating that confidence under my argument other than by really focusing on the issues that people care about, presenting the information in a way that is accessible to them and so on. So I think that just because it’s administrative data, I don’t think it takes away the need for anything I’m talking about at all.
Georgina:
This move to admin data was also really more the brainchild, I think, of Ian Diamond, who, as I mentioned, left as head of the ONS earlier this year. And before he left, it was also unknown whether we were going to have a census again in 2031. Now we’ve been told that we will. But up until that point, there really had been this kind of idea that we don’t need to rely on surveys and we don’t really want to rely on them anymore. You don’t need to tell us what you’re up to, member of the public. We’ll already know because we’ve got data from HMRC, we’ve got data from DWP and so on. And this is kind of still in motion. Just because Serene has now left doesn’t mean that’s all ground to a halt. And as I know very well from migration statistics, this kind of move to admin based data is in a very healthy state now. And actually I personally think it’s very good thing, but in trying to still get people to respond to surveys while also building a whole system to use people’s administrative data, whether they know that it’s being used or not. Don’t you think that ONS is kind of trying to ride two horses at once? Should they not just pick a single approach?
Ed:
I’m a big fan of riding two horses in this context, actually. Funny though, it may seem to say. And that’s because the administrative data and data collected directly from subjects sort of cross validate each other. Obviously over time, you might want to have more of an emphasis on one and more of an emphasis on the other and there are cost implications and so on. But I wouldn’t say that you should be making a wise choice and just to plump one and just to use that as your primary source. There are some circumstances where administrative data is absolutely ideal. There are other circumstances where surveys are ideal. And there’s a big hinterland where they both mutually reinforce each other. And a really good example of that actually will be the 2031 census, where ONS will have more use of administrative data than it ever has had in the past. And it will use that to generate a kind of overall sense of the size of the population and the location of the population.
But there will be questions that the administrative data can’t tell us, like people’s religion. That will not be very reliably recorded in the administrative data, if at all. And the census will give us that. And the richest picture, I think, comes from merging the two of them. Over time, having said all of that, I think there will be greater use of administrative data in UK statistics. That will grow. But I would never want to argue that we should go 100 % to riding that horse. We should also keep the other horse of surveys there as a source of extra insight and validation.
Georgina
The consultation has now started for the 2031 census. Members of the public can write in with their ideas about what information should be collected. They can even suggest new questions that should be in there, which I’m very curious to see what they come up with. Is this going to make people feel connected enough, a write-in consultation, or is there more they’re doing that maybe I don’t know? You’re emphasizing the need for this kind of eye-to-eye consultation and dialogue. Could they be doing more to establish a proper dialogue with the public on the census?
Ed:
I really hope they do a lot. Actually in the 2021 census at the ONS, and it’s not just the ONS, it’s also in Scotland, National Records Scotland, and in Northern Ireland, the Northern Ireland Statistics and Research Agency. They did a great deal of engagement where they went out to communities, geographic communities, and then communities of interest to ascertain what they were interested in, what they wanted to learn about, how they wanted to be represented.
They did that well. I think there are lessons to learn from the 2021 experience. For example, there was some controversy about the question about trans identity, whether they gave enough thoughts to how people who didn’t have English as a first language might understand the question. But essentially, they’re quite good at engaging. And I think that’s exactly the kind of spirit that should be continued into 2021.
Even though there’ll be more use of administrative data in 2031, I think that community engagement is absolutely essential. And I think it sort of captures something about the census being this great civic endeavour where people represent themselves in their words to the state, not vice versa. And I think your book, Sum of Us, a way, kind of you unpacked-
Georgina:
Thanks for the plug.
Ed:
No, it’s a great book available in all good bookshops, I believe. Yeah, you do get into that, don’t you? You talk about that sort of census of civic connection. The only thing I think that could be built on is the feedback loop. If you really take the eye-to-eye, shoes in the street, vulnerability, openness thesis that I’m putting forward, you wouldn’t just do that community engagement at the point of wanting to find out how people wanted to see themselves represented. You would then go back to them afterwards and say, is how you have been represented. You’d close the loop of the conversation. I think there’s more scope to do that,
Georgina:
Do we know whether they’re doing that on the trans question, on the gender identity one? Because that was the one where they had the biggest problems last time and in fact the official statistics status of it was kind of taken away in the end because of this issue with people who didn’t speak English very well not necessarily understanding the question.
Ed:
Just to be really clear for your listeners, your thousands of listeners, I think it’s fair to say your thousands of listeners. When you say the official statistics status was taken away, that was me, that was my organization. The Office for Statistics Regulation looked at that issue, said there’s clearly a concern and our conclusion was that those particular statistics on the trans population did not fully comply with the code of practice. That was our decision.
In terms of the follow-up, the ONS, to some extent at our behest, has been doing a big engagement exercise to think about how best to capture data, both on gender identity, but also on sex as well. So they have been doing quite a lot of that feedback.
Georgina:
Okay, great, so that’s ongoing. And final question, if you could add a question to the census, what would it be?
Ed:
Well, it’s a sort of question where I wish I had like a witty response, but I’m a serious kind of fellow really when it comes down to it. So my serious response would be, I think it’d be really fascinating to use the census to ask people what information sources they most rely on and most trust. Misinformation and information sources is such a big issue in our society and I think many of them, and it’s such a source of kind of policy anxiety, social anxiety, people getting information from less trustworthy sources perhaps. Here’s your one opportunity to ask everyone and get a really good picture and then to get the demographics which underpin that picture. I think that’d be a great question to add in, a question about information sources. mean, currently we ask people how many, you know, whether they have central heating, right? I think that’s kind of interesting and important and people need to have warm homes. I think they also need to have a safe information environment.
And I think this would be a great opportunity to do that. Yeah, I was, was trying to think of some witty version of shoes in the streets and vulnerability, but I couldn’t come up with that. So I can’t with the serious, serious proposition, actually use it to find out how people think about their information environment.
Georgina:
Ed, thank you so much for coming on the show.
Originally published on Substack: https://georginasturge.substack.com/p/podcast-episode-trust-in-statistics