I’m thrilled to announce that Talk Data to Me has now moved into the world of podcasting, with the first episode available now!
Each week, I’ll be interviewing a guest on an important issue to do with data and current affairs. For this first episode, I was lucky enough to get Jonathan Portes, Professor of Economics and Public Policy at King’s College London, to speak to me about the difficulties the UK has faced with economic data in recent years and the statistical challenges the Chancellor faces with this week’s budget.
Listen now on Spotify and Apple Podcasts, or read the full transcript below (with a trigger warning for any Americanisms which may have slipped in via the AI.)
TRANSCRIPT
Georgina (00:02)
Data. It’s the most important currency in our modern world. I’m talking statistics, numbers, all that information that governments and politicians use to know what’s going on. Without it, virtually nothing would function. But, as a statistician and someone who writes about data, what if I told you that it’s very difficult to count and measure things well? Life isn’t as neatly organized as statistics would have you believe, and avoiding the trap of bad data is a fine art. I’m Georgina Sturge, and on this podcast, I’ll be uncovering the surprising reality behind some of the numbers that power our world. Join me each week. This is Talk Data To Me.
Jonathan Portes is a professor of economics and public policy at King’s College London, where he’s been based for nearly nine years. Prior to becoming a hardcore academic, Jonathan had already had a very substantial career in the civil service, which makes him a unique person to talk to about both the technical side of data and how it’s used in the real world by politicians. In his time at the civil service, he worked at the treasury, was an economist at the Department for Work and Pensions, and was chief economist at the Cabinet Office from 2008 to 2011, so during the financial crisis. He’s also a senior fellow at the UK in a Changing Europe, which examines the UK’s post-Brexit relationship with the EU. He’s the author of books, including the 2024 50 Capitalism Ideas You Really Need to Know and the 2019 What Do We Know and What Should We Do About Immigration? He’s appeared as an expert witness at parliamentary committees
and can frequently be found providing commentary to the media on current affairs. And he’s still able to reply pretty much instantly to any email which I send him. Jonathan, welcome to the show. I would like to talk to you about data on the economy. And the big question that I want to get into is, is the data that we have in the UK, is it good enough for the task that it’s been set? So we have a budget.
Jonathan (02:20)
Thanks for having me.
Georgina (02:35)
coming up, which is one of the reasons why I wanted to have this discussion. But more generally, this is an area where there have been some problems in recent years. And I want to understand the impact that that’s been having on politics and policy. So first of all, I want to start with a really broad question. In an ideal world, what information would we want our government to have in order to run the economy?
Jonathan (03:00)
I think there are probably at least three different types of data you can imagine you’d want the government to have. One is on what you might call sort of macro level or what economists anyway would call macro level trends in data. you know, what is happening to the size of the economy? ⁓ What proportion of people are unemployed? What is the rate of inflation? That is the rate at which prices overall are rising. That’s the sort of
macroeconomic level data that the Treasury and the Bank of England need to set things like interest rates to make economic forecasts and so on and so forth. The second is sort of the data which we need to make good detail policies. So if you want introduce a new program for young unemployed people that trains them to be bricklayers, the level of youth unemployment overall is not that relevant. The question is,
Well, we had this other program five years ago that trained people to be plumbers and we spent 10,000 pounds per participant. And this is what it did to their employment outcomes five years on. So we conclude this sort of program typically worked quite well, or actually typically they’re a waste of money and that helps us decide what to do next. So that sort of micro level, you know, remember if you think about what government economists and indeed policymakers spend their time doing, there’s probably about.
20 or 30 people who matter on the decision to raise interest rates or lower it. There are thousands of people across government working on whether to have a work visa for this particular occupation or whether we should introduce this program to train young people to be bricklayers and so on. And then the third sort of thing. So for that sort of data, as I say, you need individual level data really to do that properly or firm level data sometimes or local data or something, but it’s quite detailed.
But you don’t necessarily need to know the name of the person. There’s clearly huge potential in using the amount of data that we now have and that other organizations have about ourselves to make government more efficient, more cost-effective, and give us better outcomes and a better service. And this, of course, is where data becomes quite political, quite contentious, and so on as the debate with ID cards demonstrates.
Georgina (05:25)
What you described there is kind of in an ideal world, what type of information do governments need? How far away do you think we are from that vision? ⁓
Jonathan (05:36)
I mean, I actually think that the glass is at least half full in some sense. We’ll come on to talk about, no doubt, some of the recent problems with some of this, especially macro level data, which generated headlines. But on the other hand, I do think that in many respects, things are a lot better than they used to be because of course of the advances of technology.
So take an example of something which I think where things have changed, data has sort of changed for the better. The Windrush scandal happened in part because, well, it happened a combination as these scandals and errors usually do, a post office scandal being another one, of people having the wrong data or the data not being in the right place at the right time. And then politicians or policymakers deliberately ignoring the warnings that they couldn’t use that for what they were trying to use it for. So in the Windrush scandal, we did not have records of lots of people who had entered the country perfectly legally and were here perfectly legally.
And politicians decided nonetheless to plough on and go down a course which resulted in those people being treated appallingly as we know. But a few years later, indeed under the same government, and the person who was responsible for that, Theresa May, now as prime minister, ⁓ when it came to European Union citizens who were here at the time of Brexit, ⁓ we actually used data, in this case, the tax and benefit records of those individuals, to give for the vast majority of the six million odd people who applied for a new settled status, it was a user-friendly, digital first application where their entitlement was demonstrated on the basis of their tax benefit national insurance records held on DWP and HMRC systems. And for most people, not for all, of course, for significant people there’s always going to be difficult cases and errors. And I’m not saying that some people didn’t have a difficult experience, but for the vast majority of those people, it was pretty seamless. They could use those records that were held and that would not have been, and know, and the wind brush thing demonstrated that would absolutely not have been possible, ⁓ 15 years ago. So that in some sense demonstrates a lot of progress.
Similarly, ⁓ the immigration statistics, a, a topic you and I know extremely well, but a lot of the sort of headlines that you see as indeed we saw yesterday when we ⁓ announced revised estimates for the numbers of Britons ⁓ emigrating and immigrating into and from the UK are because things are better, right? We used to measure these statistics on the basis of a survey which had become progressively unreliable to the point of being unusable. Now we are measuring them again using, ⁓ in the case of British citizens, a combination of tax and benefit records to establish whether people are in the country or not. And that’s clearly progress and the data is better.
The headline is, ONS discovers that its figures were wrong. But underlying that headline is ONS is improving the way that it measures this. some of these are sort of transitional problems. That said, I think there are still some really big problems and a lot of them have to do with, as I said, it’s never the fault of the data or the computers or usually not even the fault of the people who work with the data, but the people who use the data and take decisions. So I gave you the example of HMRC using again, tax and using home office data to make individual level decisions on whether to withdraw benefit. And that’s a clear example of where it’s absolutely clear to anybody who understands or knows the data.
This data was not the right data to use for that purpose. And it was going to lead to ⁓ significant ⁓ errors that were never going to be politically acceptable. So it was doomed to failure from the start. And anyone I think could have told them that. you still see errors like that across government. ⁓ Another big issue is around ⁓ data sharing and data linking across government, where again, a sort of combination of political obstacles, understandable concerns about privacy and sort of, think a fair amount of sort of bureaucratic inertia and turf guarding ⁓ means that ⁓ we don’t do this as nearly as efficiently or effectively ⁓ as we should.
Georgina (10:47)
Now there’s another whole area which I’d like to get into a little bit, which is this micro level data and specifically data that comes from surveys. So a couple of years ago in 2023, the ONS had to delay the publication of its headline labour market statistics because of concerns over data quality and collapsing response rates to its economic surveys. And a few weeks after that, the regulator revoked the official status of the labour force survey, which underpinned that information. So this was kind of a big moment where problems that some people had known about for a while came to the surface. The ONS also suspended the official statistics status of its wealth and assets survey earlier this year, which is the only comprehensive source really of evidence about savings, debt, income, retirement planning all in one place.
And the Financial Times has reported that 79 different official data series have either been decertified or cancelled outright in the UK since 2010, 25 of them in the past year alone. Can you tell me a bit about what’s happened to our labour market statistics and other economic data that comes from surveys in recent years and what the impact has been?
Jonathan (12:05)
On the Labour Force survey, survey response rates had been dropping before the pandemic, ⁓ but they dropped. And of course, during the pandemic, this couldn’t work at all. ⁓ But when we restored the survey after the pandemic, ⁓ response rates didn’t recover nearly as much as we thought. This downturn has been worse in the UK than almost anywhere else. So I think there’s several things that are going on here.
One is that ONS and others perhaps were a bit slow to work out even before the pandemic that this trend was ultimately going to ⁓ make things very difficult. Second, the pandemic gave this sort of shock to people’s willingness to respond, which we still don’t really understand. And then third, and I think this is part of the UK specific explanation, is that the survey had simply become too long and too burdensome because every department and interest group wanted to add its own set of questions. So the labour force survey no longer just asked you, well, are you looking for work? Are you in work, out of work? Are you looking for work? How much do you earn? What’s your name, date of birth, education, et cetera. But also a bunch of questions about health and disability, a bunch of questions about transport, a bunch of questions about community participation or whatever, all of these things. So we overloaded the survey to the point where it just frankly collapsed under its own weight. So that’s how we got here.
Georgina (13:38)
When you say that’s where we got here, what state is the LFS in now?
Jonathan (13:43)
Well, we are trying to build the new plane as we’re flying the old one, which is not an easy task, believe me. That is to say, we are trying to patch up the LFS and get it back to a state where it’s good enough for the moment. Introducing a number of things that are designed to increase response rates, like a small financial incentive, ⁓ like nudging people to respond and so on.
⁓ and simply increasing the number of people we contact to boost the number. So we’ve boosted the numbers back. We’re pretty happy that it’s a lot better than it was and that it’s not too bad for judging where we are at the moment. And we have a bunch of evidence that so it’s okay. But we’re also clear that in the long run, we do actually need to do better. And so we’re introducing the new transformed LFS where we will have a core element that focuses much more just on those sort of need to know labour market questions that can be completed much more quickly and which we will hope mean that we have a more representative sample. And we’re trying to do both of those at once. And of course we need to run them in parallel because we need to be pretty damn sure that we’re getting results that are comparable. And so there’s not some sort of huge transformational break between an LFS series for which we have data going back 50 years, and the new series, need to ensure that they are broadly telling us the same thing.
Georgina (15:08)
There was a moment after the pandemic when the labour force survey wasn’t really giving us reliable information, but it made it look like a lot of people were not in work. So economically inactive, I think, was the main category. Later, it kind of transpired that there were problems with response rates to the survey and other sources of data showed that perhaps this wasn’t actually the case. Now economists kind of agree that it was it was a bit of a mirage in the data. The government, however, had already put quite a lot of policy effort into trying to kind of get people back into work and stop them malingering after the pandemic. So where are we with that now? Have we managed to get a handle on what’s going on there?
Jonathan (15:52)
So I think this is where some of the concerns are overblown. mean, sure, the data probably overstated the immediate rise in ⁓ health-related inactivity after the pandemic ⁓ somewhat, and we’re now closer to having a better picture. But I think the idea that this has led to some sort of huge policy error or mismatch, ⁓ I don’t think is really true. We have administrative data, as you know, from the department of work and pensions on the number of people claiming both ⁓ incapacity benefits and ⁓ personal independence payment. Both of those have skyrocketed since the pandemic. So the idea that there’s nothing going on here and that government would have responded ⁓ with, you know, that there wouldn’t be huge policy concern and efforts to respond by government, I think is, is, is frankly nonsense. So of course you do want that data in the labour force survey to reflect what is actually happening because that, as I said, tells you other things about individuals that you don’t get from the administrative data. But the idea that it sent us down some sort of desperately wrong track in terms of what the media were focused on or what the worry was or what policymakers, I think that’s frankly just wrong. ⁓ And I think that’s also true some of the reporting about the problem with the LFS.
The idea that we’re flying, you know, I’ve seen the expression flying blind used. That is not true. We are not flying blind when it comes to the labour market. Not only is the LFS broadly back to the point where it gives us overall a reasonably good picture of the labour market, but all throughout this, we had, as I said, data on the number of people on payrolls from HMRC. And that’s actually data we didn’t have 10 years ago. So that is an improvement. Not to say that this isn’t one of matter of serious concern.
Particularly when you look at more detailed data within the labour force survey, but the idea we’ve sort of been had a fundamentally wrong picture of the labour market. And that meant that we were making big economic decisions with no data at all. ⁓ I don’t think that really holds up.
Georgina (18:06)
So with the Labour Force survey, we’d love that to be perfect. you’re saying perhaps people are looking at that and saying, what a crisis because we can’t use this survey. But actually, there are lots of other data sets that we can use.
Jonathan (18:10)
It never was.
There are lots of other data sets we can use and it was always the right thing to do to look at a number of different data series to get an idea of what the overall ⁓ state of the labour market was. Nor was it the case that things were wonderful 10 years ago and now they’re terrible, as opposed to saying, well, some things have got better, some things have got worse.
Georgina (18:41)
I wanted to ask about something slightly different, and I want to move on to the macro level picture in just a second. But first of all, I wondered what you thought about this idea that there’s been a cultural or institutional problem at the Office for National Statistics, the ONS. So there was a review earlier this year called the Devereux Review, which found what it called deep-seated problems with the governance and culture at the ONS. To what extent do you think that there are problems or were problems and that that contributed to problems with the surveys.
Jonathan (19:17)
As I said, I think the ONS was slow to realize that the ⁓ problems were, I think there was a desire to think, well, of course the pandemic messed everything up. And our task is to get things back to normal, as opposed to saying, this is a significant structural problem, which was present before the pandemic, which the pandemic has made clearly made a lot worse. We need to really think how to get, you get things back on track and that is not a matter of simply going back to what we were doing before, only better. More broadly, as a number of people have said, it’s clear that the move to Newport, is now what, although that’s quite a long time ago now, ⁓ had a pretty significant and long lasting impact on the making the ONS in some ways disconnected both sort of in policy terms, but also personally, professionally from government in a way, which, has not been helpful. And there’s just the sort of fundamental challenge, which of course, many different bits of government where you need economic quantitative or data skills all face, but O &S is obviously in the front line here is look, the fact that the public sector doesn’t, doesn’t pay that well for people with those skills. There is no getting around that. That’s a sort of fundamental structural problem that.
If you’re really good at quantitative data science, you can get a lot more money and for the foreseeable future, we will continue to be able to get a lot more money anywhere outside government.
Georgina (20:55)
I want to turn now to the macro level picture and I want to also ask you a bit about your experience of having worked in the civil service for central government because I find that fascinating. First of all, we have a budget coming up and it feels like we’ve had a running commentary in the press of the Chancellor’s dilemmas as she reacts to internal and public assessments of the UK’s financial position. How would you summarise the challenges that the Chancellor faces.
Jonathan (21:25)
Well, the choices she faces are fundamentally economic and political, which is the British people want both improved public services and they don’t want to pay as much taxes they’re paying now, let alone more. And over the next 10 to 15 years, we’re going to get older. ⁓ And since so much of what we spend on public services is driven by the demands of the elderly, that is only going to increase demands on those public services. That sets up a whole bunch of interacting economic and political problems and against that, you know, what the OBR forecasts on the basis of what data for what the deficit will be in five years is something that preoccupies the people in the treasury a lot as they try and make their spreadsheets add up. But if you are the chancellor and the prime minister, you should be stepping back from thinking, well, actually that is not my main problem. My main problem is not what the data say about what the budget deficit is going to be in five years or what the OBR forecast on the basis of imperfect data. And an imperfect forecast is about that. It’s about this sort of fundamental, well, which of these different political and economic roads do I want to get down and how am I going to persuade the British people that’s the right thing to do? I think the attention that is focused on the OBR’s forecasts and their accuracy or not, this idea of headroom or black holes, which you’d think economists like me would love. Frankly, I don’t. think that they are fundamentally not what the debate ought to be about.
Georgina (22:59)
The headroom and black holes, it’s an intense preoccupation for politicians and the media, but maybe quite a small part really of what the chancellor should actually be worrying about and focusing on. Do you think the chancellor has enough data, has good enough information to actually be making the right kind of decisions that she needs to make about the economy?
Jonathan (23:26)
I think at the macro level, yes, for the reason I’ve said, which is that, you know, if the decision is we need to persuade the British people, need to pay somewhat more tax and, you know, so that we can improve public services and deal with the demographic challenges. Then, you know, that’s all she doesn’t need from a macro level point of view. That’s what you need to know. And then you have to decide whether to do that. You need to put up taxes by 20 billion or 25 billion, but that sort of second order consideration and in fact getting it wrong doesn’t really matter that much. Now, in order to decide exactly how you put up the taxes, you’re then into the sort of the second level down type of data. Well, if I increase the VAT threshold by X or decrease it by 10,000 pounds, what impact does that have on tax revenue and economic activity? That decision map, that’s very much that sort of
policy decision where you don’t need to know what the deficit is. You need to know, well, what happened the last time I made the change to the VAT threshold? How many people paid it? How many people dropped out in business entirely? How much more revenue did we raise? Therefore, is this a good thing to do or a bad thing to do? ⁓ 12 or 15 of those decisions matter quite a lot. ⁓ And to make those decisions, you do actually need to have the data to evaluate them, that is not a sort of macro level question. Again, that is very much where you need the micro level data and the people who are capable of analysing it. I think that’s much, as I say, think having that would improve the quality of the decision making much more than having a precise fix on whether the forecast ought to be a headroom of 20 billion or 25 billion.
Georgina (25:06)
You had a quite long career in the civil service before you moved into academia, working at the Treasury, the DWP, the Cabinet Office. You were an economist there, chief economist, in fact, during the financial crisis. What was your experience of how politicians used economic data then?
Jonathan (25:25)
Well, they use it for a variety of purposes. Most obviously, course, politicians like numbers to put in speeches. There was a sort of constant sort of cottage industry of producing numbers that at least in principle have some relationship with the facts that you can put in speeches. that sort of, think, somewhat even politicians know that sort of surface level really. There was a lot of use of data in setting targets or ambitions. So you have all these different levels at which the data and analysis is used or misused, hopefully mostly in a ⁓ positive or constructive way. Broadly, think good politicians do try to use the data responsibly. That is to say, they don’t say things which have no relationship to the facts. They don’t use numbers which have no relationship to the policy outcomes.
They do understand in internal discussions when people say, well, this number is wrong for you shouldn’t use this number for this work. But this is how we should think about the impact of our policies.
Georgina (26:32)
Do you miss being in that type of role, being in the central government, heart of decision making?
Jonathan (26:40)
⁓ I do, yes. ⁓ Particularly of course, when government do things that I disagree with on policies that I think I know something about immigration being ⁓ one obvious one. ⁓ On the other hand, if I was in government, I wouldn’t be able to come on and say what I think on podcasts like this. there’s a trade-off. I don’t, yeah, I miss being in government. On the other hand, I don’t regret leaving it when I did ⁓ at all. am involved in my role at the ONS at the moment in trying to help repair the labour market statistics, but I don’t have any particular desire to become a full-time civil servant again, put it that way.
Georgina (27:20)
Would you or I suppose are you more or less confident now with the data landscape being the way it is in being able to say things and advise politicians or even to advise the ONS?
Jonathan (27:35)
I mean, in some ways I am less confident because I know more in the sense that, mean, back when I was chief economist at DWP, which was quite a senior position, but only on the hand, I think back and looking back at myself now that I know a lot more about how data is produced, the actual sausage machine than I did, at least when I started there, ⁓ I think I would have been less confident in some things that I would have said.
So I think I put an awful lot of reliance on the labour force survey then, this is back in the mid 2000s, in advising ministers. I think looking back now, I think I would have been actually quite a lot less confident in some of the advice that I’d given because I think probably I didn’t think about the processes involved in producing it. for example, looking at some of the stuff that the way that we talked about GDP and indeed some other series, one thing that really amazes me is that we still talk about movements of 0.1 % of GDP as meaning something. And once you actually look at how GDP is produced, you realize that the margins of error here are so much greater than that, that you’re really talking about changes which are effectively at best a very, very, very noisy signal what’s going on. And I would be much less confident in basing my analysis on them that I would have been perhaps would have been 20 years ago, simply because I know a lot more about it.
Georgina (29:03)
I want to round off the conversation by turning towards the future. And you’ve already struck quite a positive tone, I think, in comparison with some of the more doom-mongering things that one reads in the press. Looking towards the future, I think there are kind of two separate discussions to be had about data. One of them is about quality and improving that. And that’s something that we’ve already talked about quite a bit. The other area, I think, is about trust.
And this is something that’s been coming up bit more in recent years, that there is an impression that people’s trust in institutions has gone down a bit, how much we know from polling data and things like that. What’s your impression of people’s trust in data and in official statistics?
Jonathan (29:49)
Sadly, I think that there is considerably less trust than there used to be. And that reflects the sort of the way that a generalized form of trust in government combined in particular with the way that some aspects of the media, and people talk about social media, but actually a lot of the problem is frankly the mainstream media as well. The way that data is used increasingly in a sort of way of serving different political agendas.
And that both makes it harder for anyone trying to present a sort of objective account of the data to cut through as well as further eroding trust. So it’s always been the case, of course, that lots of people will never believe anything the government tells them about that involve numbers. So that’s not the case that there was ever a sort of golden age here, but I think it has got worse. It feels to me that it is much more common for politicians to say, things which simply clearly aren’t true about data and have little or no penalty for it than it was 20 or 30 years ago when I was in government. That seems to be much less the case now.
Georgina (30:58)
My impression is that, however bad things are in the UK, in the USA they might be a little bit worse and that there’s quite a significant crisis of trust there. Earlier this year we saw the head of the Bureau of Labour Statistics being fired from her position because of some revisions, I think, to payroll employment. And that was widely interpreted as being politically motivated rather than perhaps out of a genuine concern for accuracy. In the US, you’ve got swathes of the public and powerful politicians who just say that whole institutions are biased or individual civil servants. This is quite worrying, isn’t it? Do you think that we’re on our way towards that in the UK? Could we end up there? Or is this something quite America specific?
Jonathan (31:51)
I think it’s highly unlikely. It’s quite interesting, of course, because of course, the head of the ONS was effectively fired or forced out as a result of the review you just talked about by the government. yet here, I don’t think anybody across the political spectrum or in the media has suggested that this was for political reasons. There’s a lot of criticism of the whole affair and what went wrong and suggestions that this was a bad thing. But nobody suggested that this happened because the Labour government wanted to install someone politically sympathetic in the ONS or that this was somehow a desire to manipulate the statistics in favour of the government because nobody thinks that. So at the moment, at least, I think that those two things happening in parallel illustrate how different we are from the US. So at the moment, that would not be high on my list of concerns. think that the politicization of statistics is a long way down the list of worries when it comes to data.
But of course, you know, who knows where we’ll be in five years. But for the moment, at least I wouldn’t be too worried about it. We’re in a lot better place in the US on that.
Georgina (32:57)
Well, we have to wrap things up there for today. Jonathan, it’s been a pleasure to have you on the show. Thank you so much for talking to me and for sharing your insights. And thank you listeners for supporting the show. If you like what you just heard, follow or subscribe to Talk Data to Me, where I’ll be continuing to talk to a guest every episode about the big issues in data. I’m your host Georgina Sturge, and I’ll be back soon with a brand new episode.
Jonathan (33:06)
Thank you.
Originally published on Substack: https://georginasturge.substack.com/p/podcast-episode-are-we-flying-blind