Why opinion polls are in danger of giving a false impression about the state of the Senedd race

Jac Larner, Politics Lecturer at Cardiff University and Research Fellow at Edinburgh University

Every opinion poll contains uncertainty and error. Pollsters only interview a sample of the population and therefore the figures they report may not exactly match the ‘true’ balance of opinion in the whole population.

Given the weight and importance that media and pundits give to these opinion polls, we need to do a better job of being upfront about this uncertainty. This is especially important considering how small changes in polling figures can produce substantially different media narratives, which in turn can influence how voters behave at the ballot box.

In 2017, for example, a narrative that Wales was ‘turning Blue’ (with voters swinging to the Conservatives) emerged from early opinion polls – and likely went on to affect voting behaviour in Wales on polling day. British Election Study data shows unusually high numbers of Plaid Cymru supporters ‘switching’ to Labour not because they particularly liked Labour, but because they really (really!) didn’t like the Conservatives.

Keep in mind the margin of error 

Looking to some key principles that underpin polling numbers can help us to better navigate this uncertainty. The first is the margin of error, which is a reflection of the uncertainty inherent in polls due to sampling choices.

As a rule of thumb, a random sample of approximately 1,000 respondents – the size of most polls conducted in Wales – has a margin of error of 3 points for estimates of support for a specific political party. This means that if a poll reports that 30% of respondents would vote for the Labour in the Senedd constituency ballot, the ‘true’ figure in the real world is somewhere between 27-33%. We can’t make predictions that are more accurate than this range with any real degree of certainty.

Seat projections

On their own, of course, vote intention figures don’t mean much. What matters is how these figures translate into seats won and lost. As such, opinion polls in Wales are almost always reported alongside a projection of what the make-up of the Senedd would look like under these results (for more on how this is done, see Roger Awan-Scully here and here).

However, the margin of error inherent in opinion polls means that many seat projections are making assumptions that the data does not warrant.

Marginal constituencies, along with nearly every one of the final regional list seats, are decided by vote margins that are smaller than the margin of error in most polls. This means that we just cannot say who will win these seats with any confidence.

It would be far more useful to report these seat projections in a way that takes into account this degree of uncertainty. The graph above showing today’s YouGov poll, and below showing the most recent polls, for example, attempt to show the range of outcomes we might take away from recent opinion polls ahead of next month’s election.

The numbers next to each party denote the range of seats that the poll suggests a party would win at the Senedd election. So, if we look at ICM’s St David’s Day poll, we can say that if the final vote shares reflected the figures reported in the poll, Labour would win between 27 and 32 seats. We cannot confidently be more precise than this.

This method also highlights how small changes in vote share result in big changes in the number of seats a party can win. Take, for example, the Abolish the Assembly Party. Based on the results of just one poll (Yougov’s March survey) the range of possible seat outcomes is substantial, somewhere between 0 and 7 seats. This is because they are reliant on winning those last list seats which are likely to be determined by very small margins.

As Wales prepares to go to the polls in just two weeks’ time, the stakes are high. It is important that polling helps to guide and inform the public, not give voters a false impression. That starts with levelling with the public about what we don’t know, as well as what we do.

Jac Larner is a Lecturer in Politics at Cardiff University and Research Fellow in Political Science at the University of Edinburgh. He is currently a co-Investigator on the Welsh Election study, and a Research Associate on the Scottish Election Study.