Now there’s a question!
After UK elections for parliament in 2015, a referendum in 2016, the Scottish referendum in 2014 and their national election last year, not forgetting voting for the Welsh and Northern Ireland assemblies, are we ready for another full parliamentary battle? In particular – can the opinion pollsters cope? All researchers will be well aware of the doubts about the reliability of the recent opinion polls expressed by many observers both in the UK and overseas.
These doubts have led to considerable controversy concerning research and sampling methods, with much of the controversy surrounding the interminable dispute between telephone and online data collection. Theoretically there should be no doubt that a genuine telephone study using an RDD (Random Digit Dial) sample should have the edge over the self–selecting process endemic to on-line research.
However, in practice, the results from the two methods aren’t as readily distinguished as the different sample methods would suggest. With declining response rates and the cost / time pressures frequently placed upon telephone research, the quality of the ‘sample’ as selected can be quickly eroded into what is effectively also a self-selecting sample, albeit one with more control information available. This similarity is further emphasised by the manner in which both online and telephone polls will be subject to similar demographic quota controls.
Despite those similarities, there will, of course, be some major differences in the way the two methods achieve the final sample
For example, the landline telephone sample will rapidly ‘fill’ the older (65 and over) age group because older people are more likely to be at home whatever time of the day the phone rings. Indeed in the UK it was a finding from the 2015 polling that, probably, it would be important in future to split the age quota, because too many over 75’s were filling the 65+ group and (possibly) that fact had introduced a labour bias into some of the polls.
Nowadays, of course, no-one would do a telephone study without using both a landline and a mobile sample to ensure coverage of the population
But do we understand all the probabilities? How many of the mobile phone sample could also have been selected within the landline sample (and vice versa)? Reading the literature I can find no reference to weighting by the varying probabilities associated with a telephone sample and, possibly, it is the case that possession of landline only / mobile only / both mobile and landline is entirely neutral in regard to voting intentions. However, I have some doubts about this neutrality, since there is increasing evidence that social class is nowadays less significant as an indicator of voting intention than usage of social media.
Whichever way you look at it, the pollsters are once again in for a difficult time
There simp
ly is no consensus as to how to fix whatever it was that went wrong in the past. A lot of the old models built to re-calibrate the results by distributing ‘don’t knows’ based on past behaviour have had to be thrown away and new ideas wait to be tried and tested.
My own idea is that time of day is important both for online and for telephone work. I would like to see much more attention paid to the time of day the response is obtained and examine the differences in responses by time of day. That has been found to influence some online survey results and could similarly affect telephone research.
Are any of the pollsters looking at that?
One thing for sure is that the rest of us will be looking at the polls with interest. Will it strengthen the ‘Remainers’ and allow a soft Brexit? Will Jeremy Corbyn break through to the wider electorate? How will the Scots and the Northern Irish vote?
It has been said that the Chinese curse was ‘may you live in interesting times’! With Donald Trump, North Korea, Putin, Syria, Brexit, Scottish Independence and, above all, non-probability samples – I can safely say enough is enough!
Tony Dent, Chairman, Sample Answers
