Why the latest Welsh independence poll needs to be treated with caution
Roger Awan-Scully, Professor of Political Science
Results from the most recent opinion poll about Welsh independence arrived just when I was packing for my holidays. Hence my uncharacteristic silence thus far about the findings. But it has been useful to have time to examine the results before commenting.
The poll was commissioned by Plaid Cymru, but conducted by YouGov according to their standard methods. (YouGov, as most readers of this blog will be aware, also conduct the Welsh Political Barometer polls). The findings engendered plenty of comment, particularly from independence supporters, with some suggesting that poll showed rising support for their cause. Perhaps inevitably, there was also some push-back from opponents; the First Minister was quoted as saying that there was ‘no appetite’ for independence in Wales.
A few thoughts on this – which I suspect will please almost no-one with strong views on the matter.
First, and at the risk of rehearsing some overly basic points, we should remember that there are limitations to all polling and surveys. Poll findings are, innately, no more than a set of answers to questions. How meaningful those collective answers depend on two major factors. One is the quality of the sample: are you talking to the right people? Even the best polling agencies produce data that are subject to sampling error; the results are, at most, estimates of where opinion lies in the relevant population as a whole. The other factor is the question. Lots can go wrong here. Questions may be biased (deliberately or otherwise), prompting people to respond in a particular direction. They may be confusing or ambiguous, leaving respondents unclear as to how to respond appropriately. Or questions may simply ask people about things of which they have little awareness or no clear view.
A second preliminary observation, partly following on from the first, is that there is no obviously right way to ask about public views on independence. Two broad approaches have generally been taken in most previous work:
- The first is to include independence as one option among several in a question that asks respondents how they would prefer to see Wales being governed. Past discussion on the blog (for instance, here) has covered the two main forms of this ‘constitutional preference’ question
- The second is a form of question that directly asks people whether or not they support independence.
Past evidence indicates that the latter form of question attracts higher levels of reported support for independence. And thus, those deriding the idea often cite evidence from multi-option questions – with one particular BBC Wales poll, conducted in the immediate aftermath of the 2014 Scottish independence referendum, attracting particular attention even though its reported 3 percent support for independence was very much an ‘outlier’ finding. (The May 2019 Welsh Political Barometer poll, for comparison, found 11 percent support for independence on a multi-option question). Conversely, supporters of independence tend to highlight evidence from binary question forms – and often to present that evidence in ways that maximises the apparent level of independence support. Such is politics.
The recent Plaid poll used a standard question preamble: “If there was a referendum held tomorrow on Wales becoming an independent country and this was the question, how would you vote?”. There is nothing obviously biased or otherwise strange about this question form (though there is always a little unreality about such questions when people mostly know full well that there isn’t going to be such a referendum the following day).
The main question, to which people were being asked to respond, was: “Should Wales be an independent country?” This wording might be seen as somewhat slanted: given the very well-established acquiesence bias in survey responses, one would normally look to balance a question like this along the lines of ‘should it, or should it not…’. However, that would also have made this question more unwieldy. Furthermore, the Plaid question wording is virtually identical to that used in the Scottish independence referendum (except, obviously, replacing the word ‘Scotland’ with ‘Wales’) and consequently in much of the opinion polling about independence in Scotland over recent years. Without some sort of split-sample design (where we ask one, randomly-selected half of a sample one version of a question, and the other half a different version, and compare the results) it is impossible to be certain, but the Plaid question wording is plausibly is a little more ‘indy-friendly’ than, for instance, the wording used in the Sky Data December 2018 Welsh poll (“If there were a referendum tomorrow on the issue of Wales becoming an independent country, how would you vote?”). But both are, I think, plausible and defensible question wordings.
The overall result for this question was, as many will already have seen, the following:
Would Not Vote: 6%
Don’t Know: 14%
Refused to answer: 3%
It is notable that a poll which causes optimism among independence supporters still sees them outnumbered by more than two to one. Unsurprisingly, independence was rejected overwhelmingly by 2017 Conservative voters, and 2016 Leave voters. Greater minorities of Labour and 2016 Remain voters, however, were favourable. More or less half of Plaid sub-sample chose yes, but this was small sub-sample, so particular caution is warranted here. Among age groups, support levels for independence were fairly consistent, except for it being notably lower among those aged 65 and older. Support for independence is also greater amongst men, but so is opposition; women, as is typically the case across many types of polling question, were more likely to reserve judgement.
Do these figures demonstrate an increase in support for independence in Wales? That is very difficult to judge. I am not aware of any previous poll that has asked this particular question form before in Wales. Back in December last year Sky Data found 17% indicating that they would in favour of independence with 67% against. But that was a different question, as well as a different polling agency. One cannot be confident that the differences between Sky’s earlier figures and this more recent data from YouGov represents a genuine change in opinion, rather than simply being an artefact of differences in who conducted the surveys and what they asked. (Two polls for Yes Cymru, using yet another different question form, and conducted by YouGov respectively in 2017 and 2019, have indicated some increase in support for independence).
The Plaid poll also asked a second question about independence:
“And please imagine a scenario where the rest of the UK left the European Union but Wales could remain a member of the European Union if it became an independent country. If a referendum was then held in Wales about becoming an independent country and this was the question, how would you vote?”
Respondents were then again invited to respond to the question, “Should Wales be an independent country?”
There are a few things to say about this second question. The first is that it is very hypothetical: it asks respondents to make a substantial leap of the imagination, into a future scenario which – even in these extraordinary political times – is very difficult to envisage actually transpiring. Such questions should always be interpreted with considerable caution. As Anthony Wells of YouGov has observed previously, “people are not necessarily very good judges of how they would respond in hypothetical situations”. The one posed in the Plaid poll requires considerable mental gymnastics from respondents, most of whom will – quite reasonably – normally spend far less time thinking about such matters than many readers of this blog might do.
A second point is that while the results from this question do show a higher level again of support for independence, even now it continues to be very much a minority position. The overall results were:
Don’t Know: 17%
Refused to answer: 3%
Wholly unsurprisingly, a question which raises the possibility of remaining in the EU attracts a narrow majority of 2016 Remain voters to support Welsh independence. There are also bare majorities in favour among the (small) sub-samples of 2017 Plaid Cymru and Liberal Democrat voters, and a plurality of 2017 Labour voters. But 2017 Conservative voters and 2016 Leave supporters (groups between whom there is a fair overlap) continue to be staunchly against independence even in this scenario.
A final point to make on this question is that it would, of course, be quite possible to construct a hypothetical scenario in which support for independence came out rather lower than in a standard ‘how would you vote in a referendum?’ All one would need to do is phrase a question that emphasised the potential risks or downsides of independence. Two can play at that game.
Something is happening on the independence issue in Wales. It has moved up the agenda in recent months. There is more discussion about it in various fora; recent marches in support of the idea have been conspicuously well-attended; and even Carwyn Jones has declared himself ‘indy-curious’. Independence has moved from being a mainstream issue within Plaid Cymru, and the active concern of a small number of other people, towards possibly becoming a regular and central part of political debate across Wales in general.
But the debate in Wales remains a long way from that in Scotland. Not only is support for independence in Wales still much lower. The entire character of the debate is in a different – and, to be blunt, much less mature – place. To talk about independence seriously doesn’t mean simply discussing whether or not you think Wales should be independent in principle. It also means widespread and thorough discussions about what an independent Wales might look like. What sort of political institutions and structures would it have? How would the constitution be constructed? And what might we seek to do with independence if it occurred? A serious debate about independence also means thinking through realistic scenarios regarding the political process. How might independence be achieved? In all these respects, Wales still has a long way to go.
This article was originally posted on the Elections in Wales Blog.
Support our Nation today
For the price of a cup of coffee a month you can help us create an independent, not-for-profit, national news service for the people of Wales, by the people of Wales.
Constitutional Convention asap.
But if respondents had been asked a question on the lines of “If independence for Wales meant a hard border between Wales and England with customs checks on those travelling between the two countries and tariffs having to be paid on goods crossing the border, would you be in favour of it then?”, I guess that far fewer respondents would have said “Yes”. Yet that is how it would be if an independent Wales was an EU member and England was outside both the single market and the customs union . . .
You have completely confused Welsh independence with that of the Irish backstop and the [cough] clown circus negotiation’s of the ‘uk’; nothing to what you have said would need to happen. The uk is a mess, a calamity a broken system which is undemocratic to that of voters in Wales and Scotland. And the sooner Wales is independent of wm ‘The empire’ the better.
No country to have left the empire has made any attempt to rejoin.
Mr A3, I don’t know what “the empire” has to do with the point that I made earlier, I certainly made no mention of any “empire”. But I will just ask you a straightforward question based on your response to me, are you denying that an independent Wales that was in the EU and an England that was completely outside of it (i.e. not in the single market or the customs union) would necessitate customs and tariffs between the 2 countries then? If you do, would you please care to share your reasoning with us?
Hi Richard, firstly I’m not here to be the person who has ‘all the answers’ (I haven’t said that you said that either) I’m here to read and try to understand why people want independance and why some people are against it; personally though I am pro independance, also I didn’t say that you mentioned the empire; when I mentioned the empire I was referring to the state rule of the current situation which does come under the remice of what the empire is; which is to control all of it’s ‘states’ in terms of overseas possessions, trades and economic… Read more »
The UK Government has also suggested a kind of “digital declaration” system for the Irish border and as an alternative to the Irish backstop but the EU have rejected it and correctly said that such a system is just pie in the sky at the moment with nothing of the sort having ever been tried anywhere else in the world. What reason is there to believe that it would be practical option for the Wales/England border?
A price that an increasing number of people would be willing to pay. Anyway, I don’t believe that an issue like cross-border commuting and trade can’t be settled amicably between England and an independent Wales.
Rhosddu, I honestly can’t see how frictionless trade and movement could be secured if there were different tariff regimes and product standards etc between the 2 countries. We would essentially be left with the same problem as the one concerning the Irish border in the current Brexit impasse, it’s inevitable that goods crossing the border would be subject to some kinds of checks. And if such a complex problem could be settled “amicably”, the Irish problem wouldn’t be an issue with Brexit and there would be no need for the controversial “backstop”. Regarding your other point that an increasing number… Read more »
Sorry, Richard, why can’t there be a free trade agreement between two sovereign states? Many businesses in Wales are subsiduaries of English-based companies, anyway; likewise in Scotland. What’s to prevent ‘business as usual’? A hard border would be there to control settlement in Wales from England, not to impede the flow of goods, surely? Mind you, it would get complicated if Wales was in the EU and England wasn’t (the Irish situation, as you point out). I see an ‘English backstop’ looming. You’ve raised a very interesting and potentially problematic point about a post-Brexit independence settlement. I do believe, though,… Read more »
Rhosddu, my points were based entirely on the assumption of an independent Wales being an EU member and England being outside both the single market and the customs union. If an independent Wales decided not to join the EU and England was also outside of it then of course a free trade agreement and a customs union would be possible between the two countries (although if it was just a bilateral arrangement it’s inevitable, given the relative size of the 2 countries, that England would be the dominant partner and have the upper hand in negotiating the terms, thus calling… Read more »
Then you can kiss goodbye to Cymru as a separate social, cultural and linguistic entity a few generations down the line, Richard. However, you’ve strayed into the realm of speculation, and I’ve now followed you into it. I don’t want to enter into a war of attrition to see who blinks first, so I give you leave to have the last word in this exchange. Pob lwc i chi yn y diddorol.
The ONLY reason that the Independence marchers boasted that thousands marched in Merthyr, is probably something to do with the fact that people from North Wales were bussed down by coach to swell the numbers! Probably the favour was returned in the March in the North of Wales! Hopefully we wont see an independent Wales for many years if ever!
Three busses came down from the north and two buses from the west; hardly a significant swell to represent a the 5,200 who attended. There are plenty of people in the south who are for independence and long may it continue
I live in S. Wales, I just drove up the road
England depends on Wales for it energy and water. Without energy produced in Wales the lights will go off in England at peak times, probably more often. Without Welsh water England wouldn’t last a day. It’s not in England’s interests to play the silly buggers they have been playing with Brexit. Wales may have to compromise to get independence. When S. Ireland got independence an agreement forced on Ireland was the British army would control Irish ports. An agreed that run out jut before WWII and which earned Ireland the fury of the British media when they refused to allow… Read more »
You’re right. The compromise would be called Dominion Status. Massive amounts of home rule, just not 100%. And we could kick on from there to Indy, when Wales agrees. But to start this journey, as jr humphreys says, we need that Wales Constitutional Convention.
Jonathan Edwards, a federal UK would also mean “massive amounts of home rule” but without the disruptive effects that I mentioned in my earlier posts.
A federal or confederal UK might be better solutions but they would be just as bard to achieve as independence. Considering it has proved difficult to even organize a revival of the home international championship in football ; it will be even more of a struggle to get all.our many politicians to agree. However crises breed many unexpected consequences so nothing can be ruled out. When emotions are involved people can be quite prepared to cut off their own noses rather than compromise. That has been the lesson of the last three years. Most revolutionary moments have witnessed someone inspirational… Read more »