Surveying is the process of gathering information from a sample of people and then using that to indicate the likely views across a wider population.
We use this at the AA to understand the views of our Members on transport issues and be a voice for them to Government and transport decision makers.
Mathematical formulas have been developed over time determining how large a sample has to be to accurately reflect the wider population to within a certain level of confidence (usually that there is only a one in twenty chance of error).
Those samples can be surprisingly small, which is why political polls quoted in the news are usually based on less than one thousand responses. The AA's own experience over many years of surveying has confirmed this, with proportions changing little once response numbers go beyond 1000, which is in line with the sample sizes we generally seek.
Of course the math only works if the sample of people being surveyed is reasonably random and reflective of the overall population. In general, the bigger the sample the more accurate it will be but it is also vital that the method of drawing the sample doesn’t distort proportions by selecting for any particular quality.
This page has information on:
The AA Membership population
With more than 1 million personal Members, the AA’s membership is large and broad.
Naturally, AA members are drivers but they also include users of all other forms of transport.
This means we have Members from right across society but they are not an exact representation of the New Zealand general public.
Comparing our Members to the wider population, they are generally older (about 8 years on average), less likely to live in areas of high deprivation and we have slightly more female than male Members.
AA Members will commonly be middle class, established, somewhat conservative about risk, environmentally conscientious, and consider themselves safe, sensible and reasonable people
Drawing a random sample
AA surveys are conducted using email invitations to an online questionnaire. Our system randomly selects the email addresses to receive a survey and because the AA’s Membership is large we are able to cast a wide net for these random samples.
Only one response is possible per email invitation sent out, to prevent an individual from answering a survey multiple times.
We sometimes hear from Members asking why they have never been surveyed? With such a large Membership, the reality is that each survey will only go to a small proportion of the entire population. The sample sizes are still large enough though for us to have confidence that they will reflect the wider population of our Members.
At 1000 random responses, the margin of error for a survey is about +/- 3%.
Email surveys are very efficient but, like any survey method, there are limitations. While we can track the emails we send we cannot tell whether they go to old or unused email addresses, or if they appear in a recipient's main inbox or get redirected to a sub-folder that makes them less likely to be seen.
So we know not every survey invite we send out will be seen and obviously not all Members that see it will choose to take part. This means that there is a further random element in whether the Members sent surveys decide to complete it.
On some issues we will not use a random sample approach and instead give all Members in a specific area an equal opportunity to express their view. This is normally done for local issues affecting just one place and/or where the overall number of Members in the survey area is not very large.
In these cases we send invitations to every Member in a certain district offering them the same opportunity to have their views recorded. As previously mentioned, every Member sent the survey will not necessarily see the email or choose to take part but the numbers we get in response are the best measure possible to base a district view on.
The question writing process is a lot more difficult than many people realise. Ensuring the questions we are asking are neutral and unbiased is always a focus of our survey development process and we go through multiple stages of internal testing for this.
Questions are reviewed so we can be confident they are not leading and where topics are especially sensitive we will sometimes employ an independent consultant to review the questions.
Because the AA has such a large and broad Membership we are used to responses from a wide range of perspectives and anticipate these in our questions.
Sometimes survey results can be surprising because it is natural for people to assume that most others think or do the same things they do. The reality can sometimes be quite different. For example, many of the people commuting to work every day would likely be surprised to find out that they are a minority within the population overall.
Analysing the responses
When the survey reaches a suitable level of response we gather in the data and analyse the results in a range of ways.
We sense check against other surveys or data sources and also see if there are demographic qualities that make a difference to the responses. Respondents' gender and age frequently makes the most difference - reflecting the fact that different groups can see the world from different perspectives.
If the proportions of these demographic qualities do not match the AA’s Membership we use a mathematical weighting process so that any over-represented group's views are scaled back, and under-represented views scaled up in proportion with the AA Membership's demographics. This means the survey results on the substantive questions end up reflective of what we would have got if the demographic proportions were in line with the AA's overall Membership population.
Levels of support
Because our Membership is so large the AA expects to see conflicting results within our surveys. Just like within any group of people, there will always be a range of views.
Typically unless there is a two thirds majority for any view the AA will not claim the majority of its Membership supports any particular stance. It will instead speak of the Membership being split on that issue.