Leading Practice
Sampling Bias - Who You Survey
If you are looking to draw conclusions about a certain group, you'll need to ensure that everyone in that group has an equal chance of receiving an invite to respond to your survey.
For example, if you are looking to draw conclusions about all of your customers, you will need to share your survey in such a way that all people have a chance of responding to the survey. If you only share a link to your survey via social media, it's very likely you are biasing your results. Chances are, not all of your customers have social media accounts. Further, only a fraction of those that do, will have liked or followed your company page.
In survey methodology terms, this is called Sampling Bias, which is introduced when some members of your intended population are less likely to be surveyed than others.
How can I reduce Sampling Bias?
The easiest way to remedy this trouble is to share your survey via various methods. Send out an email campaign, share it on your website, post QR codes in your physical locations. You might also consider hiring telephone or in-person interviewers to recruit respondents who don't have access to internet in their home or do not otherwise spend much time on the web. You can also mail out paper surveys to segments of your population who are not particularly tech savvy.
Nonresponse Bias - Who Responds
Even if you get the sampling just right; your next battle is to fight the bias introduced by who responds. Regardless of how careful you are about inviting everyone to respond, there will be some people who are either unwilling or unable to respond. These people are often systematically different than those who do respond. This is called Nonresponse Bias, when respondents differ in meaningful ways from nonrespondents.
How can I reduce Nonresponse Bias?
Increasing your response rate will increase the chances that everyone in your population (the group that you are looking to draw conclusions about) is represented. What is a good response rate, you ask? The answer is, it depends. That's a frustrating answer. We know. It's the truth though. Fortunately, while some of the factors that affect response rate are out of your control, e.g. your population, survey topic, there are a lot of factors that are under your control.
Here are some things you can do to increase your response rates:
- Send an pre-notification email with information about the survey to come
- Send a personalized invite
- Send a reminder
Response Bias - Are Your Respondents' Answers Accurate?
Humans are complex and quirky creatures. Getting people to respond to your survey is hard enough, but you also need to think about how to get them to respond accurately. Both subconscious and conscious cognitive factors can result in less-than-truthful responses.
In general, people want to be agreeable. Often, survey respondents will tell you what you want to hear. If your survey includes agree/disagree type questions they're more likely to agree. This tendency to concur is referred to as Acquiescence Bias or yea-saying.
Simply being part of a survey might change your respondents' answers. When they are the subject of an survey or experience people often try to figure out the purpose and behave accordingly. This type of bias is referred to as Demand Characteristics.
When presented with a scale, say a 5-point scale from 1 to 5, people are often biased to only select the most extreme options. This type of bias is referred to as Extreme Responding. Interestingly, the opposite of this bias occurs when participants only select neutral responses. This tendency towards neutral or extreme responses is generally culturally specific.
Finally, survey respondents are motivated to answer in such a way that the ascribe to behaviors and characteristics that are desirable and deny undesirable characteristics and traits. This is referred to as Desirability Bias.
How can I reduce Response Bias?
Conducting your survey online automatically reduces the potential for response bias because the questions are self-administered thus making it easier to be honest. This is particularly helpful when asking about loaded topics. Here are some good tips for reducing response bias:
- Ask neutrally worded questions
- Make sure your answer options are not leading
- Make your survey anonymous
- Remove your brand as this can tip off your respondents on how you wish for them to answer
Order Bias - The Order of Questions and Answer Options
The order of both questions and answer options in your survey matters! Questions that come early in your survey might influence how respondents respond to questions later in your survey.
Take, for example, the below satisfaction questions:
What's wrong with the order of questions? The placement of the overall satisfaction question at the end of the list is biasing the response to it. We are priming the response to this question by focusing the respondent on components of Alchemer that, depending on the respondents experience with them, may cast a positive or negative light on them. This is referred to as an assimilation effect, where the response to a latter question is more similar to former questions than it would be if it preceded it or was asked on its own.
The opposite can also happen. Where response to a latter question is more extreme in contrast to a former question than it would be if it preceded it or was asked on its own. This is referred to as an contrast effect.
In addition to the bias that is introduced by the order of questions, the order of answer options can also bias your results. In self-administered surveys respondents usually prefer the first few answer options in a list. For surveys phone surveys and in-person interviews where an interviewer is reading the answer options aloud respondents tend to choose the later options.
How can I reduce Order Bias?
- Reduce the number of scale questions to the bare minimum
- Group your survey by topic
- Leave demographic questions until later in the survey
- Ask question that engage respondents
- Randomize your question and answer options
