Why the latest Welsh independence poll needs to be treated with caution

Roger Awan-Scully, Professor of Political Science

Results from the most recent opinion poll about Welsh independence arrived just when I was packing for my holidays. Hence my uncharacteristic silence thus far about the findings. But it has been useful to have time to examine the results before commenting.

The poll was commissioned by Plaid Cymru, but conducted by YouGov according to their standard methods. (YouGov, as most readers of this blog will be aware, also conduct the Welsh Political Barometer polls). The findings engendered plenty of comment, particularly from independence supporters, with some suggesting that poll showed rising support for their cause. Perhaps inevitably, there was also some push-back from opponents; the First Minister was quoted as saying that there was ‘no appetite’ for independence in Wales.

A few thoughts on this – which I suspect will please almost no-one with strong views on the matter.

First, and at the risk of rehearsing some overly basic points, we should remember that there are limitations to all polling and surveys. Poll findings are, innately, no more than a set of answers to questions. How meaningful those collective answers depend on two major factors. One is the quality of the sample: are you talking to the right people? Even the best polling agencies produce data that are subject to sampling error; the results are, at most, estimates of where opinion lies in the relevant population as a whole. The other factor is the question. Lots can go wrong here. Questions may be biased (deliberately or otherwise), prompting people to respond in a particular direction. They may be confusing or ambiguous, leaving respondents unclear as to how to respond appropriately. Or questions may simply ask people about things of which they have little awareness or no clear view.

A second preliminary observation, partly following on from the first, is that there is no obviously right way to ask about public views on independence. Two broad approaches have generally been taken in most previous work:

• The first is to include independence as one option among several in a question that asks respondents how they would prefer to see Wales being governed. Past discussion on the blog (for instance, here) has covered the two main forms of this ‘constitutional preference’ question
• The second is a form of question that directly asks people whether or not they support independence.

Past evidence indicates that the latter form of question attracts higher levels of reported support for independence. And thus, those deriding the idea often cite evidence from multi-option questions – with one particular BBC Wales poll, conducted in the immediate aftermath of the 2014 Scottish independence referendum, attracting particular attention even though its reported 3 percent support for independence was very much an ‘outlier’ finding. (The May 2019 Welsh Political Barometer poll, for comparison, found 11 percent support for independence on a multi-option question). Conversely, supporters of independence tend to highlight evidence from binary question forms – and often to present that evidence in ways that maximises the apparent level of independence support. Such is politics.

The recent Plaid poll used a standard question preamble: “If there was a referendum held tomorrow on Wales becoming an independent country and this was the question, how would you vote?”. There is nothing obviously biased or otherwise strange about this question form (though there is always a little unreality about such questions when people mostly know full well that there isn’t going to be such a referendum the following day).

The main question, to which people were being asked to respond, was: “Should Wales be an independent country?” This wording might be seen as somewhat slanted: given the very well-established acquiesence bias in survey responses, one would normally look to balance a question like this along the lines of ‘should it, or should it not…’. However, that would also have made this question more unwieldy. Furthermore, the Plaid question wording is virtually identical to that used in the Scottish independence referendum (except, obviously, replacing the word ‘Scotland’ with ‘Wales’) and consequently in much of the opinion polling about independence in Scotland over recent years. Without some sort of split-sample design (where we ask one, randomly-selected half of a sample one version of a question, and the other half a different version, and compare the results) it is impossible to be certain, but the Plaid question wording is plausibly is a little more ‘indy-friendly’ than, for instance, the wording used in the Sky Data December 2018 Welsh poll (“If there were a referendum tomorrow on the issue of Wales becoming an independent country, how would you vote?”). But both are, I think, plausible and defensible question wordings.

The overall result for this question was, as many will already have seen, the following:

Yes: 24%

No: 52%

Would Not Vote: 6%

Don’t Know: 14%

Refused to answer: 3%

It is notable that a poll which causes optimism among independence supporters still sees them outnumbered by more than two to one. Unsurprisingly, independence was rejected overwhelmingly by 2017 Conservative voters, and 2016 Leave voters. Greater minorities of Labour and 2016 Remain voters, however, were favourable. More or less half of Plaid sub-sample chose yes, but this was small sub-sample, so particular caution is warranted here. Among age groups, support levels for independence were fairly consistent, except for it being notably lower among those aged 65 and older. Support for independence is also greater amongst men, but so is opposition; women, as is typically the case across many types of polling question, were more likely to reserve judgement.

Do these figures demonstrate an increase in support for independence in Wales? That is very difficult to judge. I am not aware of any previous poll that has asked this particular question form before in Wales. Back in December last year Sky Data found 17% indicating that they would in favour of independence with 67% against. But that was a different question, as well as a different polling agency. One cannot be confident that the differences between Sky’s earlier figures and this more recent data from YouGov represents a genuine change in opinion, rather than simply being an artefact of differences in who conducted the surveys and what they asked. (Two polls for Yes Cymru, using yet another different question form, and conducted by YouGov respectively in 2017 and 2019, have indicated some increase in support for independence).

The Plaid poll also asked a second question about independence:

“And please imagine a scenario where the rest of the UK left the European Union but Wales could remain a member of the European Union if it became an independent country. If a referendum was then held in Wales about becoming an independent country and this was the question, how would you vote?”

Respondents were then again invited to respond to the question, “Should Wales be an independent country?”

There are a few things to say about this second question. The first is that it is very hypothetical: it asks respondents to make a substantial leap of the imagination, into a future scenario which – even in these extraordinary political times – is very difficult to envisage actually transpiring. Such questions should always be interpreted with considerable caution. As Anthony Wells of YouGov has observed previously, “people are not necessarily very good judges of how they would respond in hypothetical situations”. The one posed in the Plaid poll requires considerable mental gymnastics from respondents, most of whom will – quite reasonably – normally spend far less time thinking about such matters than many readers of this blog might do.

A second point is that while the results from this question do show a higher level again of support for independence, even now it continues to be very much a minority position. The overall results were:

Yes: 33%

No: 48%

Don’t Know: 17%

Refused to answer: 3%

Wholly unsurprisingly, a question which raises the possibility of remaining in the EU attracts a narrow majority of 2016 Remain voters to support Welsh independence. There are also bare majorities in favour among the (small) sub-samples of 2017 Plaid Cymru and Liberal Democrat voters, and a plurality of 2017 Labour voters. But 2017 Conservative voters and 2016 Leave supporters (groups between whom there is a fair overlap) continue to be staunchly against independence even in this scenario.

A final point to make on this question is that it would, of course, be quite possible to construct a hypothetical scenario in which support for independence came out rather lower than in a standard ‘how would you vote in a referendum?’ All one would need to do is phrase a question that emphasised the potential risks or downsides of independence. Two can play at that game.

Something is happening on the independence issue in Wales. It has moved up the agenda in recent months. There is more discussion about it in various fora; recent marches in support of the idea have been conspicuously well-attended; and even Carwyn Jones has declared himself ‘indy-curious’. Independence has moved from being a mainstream issue within Plaid Cymru, and the active concern of a small number of other people, towards possibly becoming a regular and central part of political debate across Wales in general.

But the debate in Wales remains a long way from that in Scotland. Not only is support for independence in Wales still much lower. The entire character of the debate is in a different – and, to be blunt, much less mature – place. To talk about independence seriously doesn’t mean simply discussing whether or not you think Wales should be independent in principle. It also means widespread and thorough discussions about what an independent Wales might look like. What sort of political institutions and structures would it have? How would the constitution be constructed? And what might we seek to do with independence if it occurred? A serious debate about independence also means thinking through realistic scenarios regarding the political process. How might independence be achieved? In all these respects, Wales still has a long way to go.

This article was originally posted on the Elections in Wales Blog.