Leading Practice
In this tutorial, we will cover some leading practice considerations for constructing survey questions. Following these tips will help you to collect meaningful, actionable data.
Know Your Survey Goal
Define a goal and treat it as your north star through the entire survey building process. When creating questions make sure that each question serves your project goal.
Questions to ask when defining your goal:
- What do you want to learn?
- What actions do you plan to take?
- Who will take your survey/Who is your audience?
Consider Reporting & Exporting
Probably the single most important consideration when choosing survey questions is to consider how a question reports and exports.
If you plan to use Alchemer reports, we recommend checking out how each of your questions will report. Generate some test data and run some test reports. Ask yourself, "Will this data allow me to draw conclusions I set out to address?" If not, consider a different question type.
Radio Button Question: What is your favorite ice cream flavor?
Checkbox Question: What ice cream flavors do you enjoy?
If you plan to export your raw data to analyze outside of Alchemer, it is equally important to generate some test data and run some test exports to be sure you have the data you need to perform your analysis.
Beware the Checkbox (and Other Multi-Select Questions)
Because multi-select questions allow respondents to select more than one answer, using these types of questions will considerably change your reporting and statistics.
In single-select questions the total will represent the total number of survey respondents (unless it is an unrequired question) and percentages for each answer option will be a percent of this total. However, in multi-select questions, the total will almost always be greater than the total number of survey responses (unless each respondent only selected a single answer option). This means that your percentages will add up to more than 100 (see above).
In addition, exports will have a field per answer option as opposed to a single field for the entire question. Considering these reporting differences in advance will save you a lot of heartache in the reporting phase of your project!
Each of our question type tutorials have examples of how the question will report and export. This is a great place to start when choosing question types!
Use Open-Text Fields Sparingly
Open-text questions are very fatiguing for survey respondents. If you use open-text questions, use them sparingly and whatever you do, do not require them. Finally, make sure that you act on this very expensive data.
Use Validation and Other Data Cleaning Tools
One of the trickiest things about open-text data is making it actionable in your results. Unvalidated open-text fields generally mean a mess of data as in the below example.
We recommend using some of the below tools when setting up your question to clean your open-text data and ensure that it is usable in your results.
- One of the quickest methods for cleaning up data entered into textbox questions is to select the option on the Validation tab to Capitalize each word. This will clean up your reporting a bit by eliminating case-sensitivity as a variation in your data.
- We have Basic and Advanced Validation tools available on the Validation tab of the question editor to ensure respondents' entries are in a format you expect.
- Our Auto Suggest Answers feature will suggest answers while the survey respondent types.
- Finally, if you know the list of possible answers and want to ensure that your data is tidy use a Dropdown Menu question type instead!
Asking Yes/No Questions
A common mistake new survey designers make is to choose a multi-select question type, like a checkbox, when asking a yes/no question. Because respondents will be able to select both yes and no, you'll risk collecting bad data.
If you really prefer the look of the checkbox there are a couple of options to achieve this without risking bad data.
1. On the Validation tab of your checkbox question you can set up a limit to make a checkbox work as a single-select question. To do so go to the Validation tab and scroll to the Answer Requirements and enter 1 in the Limit answers to field.
2. You can take advantage of our icon options to change the selected radio button to have a check. See our Icons Tutorial to learn more!
Grid Questions Are Not Space Savers
Grid questions should only be used if you wish for your survey respondents to make comparisons. For example, in the below question survey respondents are implicitly making comparison across the various ice cream brands listed in the rows. If this is not the purpose of your grid question it is best not to use them. Packing a bunch of questions in a grid to save space and make your survey "seem" shorter will not serve you in the long run; grid questions are very fatiguing and can lead respondents to abandon your survey or, worse, give you bad data.
Watch Out for Double-Barreled Questions
A double-barreled question basically means that you have asked more than one question but have only allowed for a single answer. See the below example. What if I am very satisfied with the produce selection but not the meat selection? Some respondents might select one or the other. Others might average their satisfaction. Either way, you're getting bad data!
Consider Interactive Questions versus Traditional Questions
Some questions types are tried and true, like Radio Buttons, Checkboxes and Rating (Likert Scale) Questions. Others are more novel and interactive, for example, the Drag & Drop Ranking and Slider questions. It is important to take this into consideration when choosing your question types. Questions that are more interactive are generally more engaging for survey respondents. While this will almost always be a good thing, there are some populations, those that are less tech savvy for example, who may prefer the more traditional questions.
If you are planning to compare data over time, changing from a traditional question, say a Rating question, to a more interactive question, like a slider may bias your results. New respondents might be encouraged to answer based on the fancy new interaction, thus changing the characteristics of the sample you are using to draw conclusions. Similarly, respondents who previously answered in a different format may answer questions differently in a survey with a new interaction, further biasing any changes you wish to measure.
Keep it Short and Sweet
Keep language in both question titles and answer options short and simple. Question titles should be meaningful on their own.
Ensure Your Answer Options Do Not Overlap
Regardless of the question type, it is important to be sure you do not overlap answer options. This most often happens when you use ranges as answer options. For example, date ranges like the below.
Add Opt-Out Answer Options to Required Questions
Required questions must have an answer option for everyone; otherwise, you risk collecting bad data. If you are not 100% sure that your list of answer options will apply to all survey respondents, add an opt-out answer option such as other, none of the above, not applicable. Check out our Answer Option Special Settings Tutorial to learn how to set this up!
Ensure Opt-Out Answer Options Are Exclusive
Whenever you provide opt-out answer options, such as None of the Above and NA, make sure that they are exclusive, meaning, they cannot be selected in combination with other options. Using our Answer Option Special Settings this is is easy to do! See the below example where selecting None of the above makes the other answer options unselectable.
Keep Scales Consistent
If you are using scale questions, layout matters. We've found that for left-to-right languages, scales that are laid out from worst/least to best/most are a standard layout with which most respondents will be familiar.
A perhaps more important concern is to keep your scales consistent. Flipping the layout, so that best is on the left and worst is on the right, is generally not a good idea as respondents might miss the change.
Our position with respect to the debate around odd numbered scales with a neutral point in the middle versus even scale is to use whatever works for you. We do, however, firmly believe that your scales should remain consistent, within a survey and across surveys. This will make it easy to make comparisons across questions. Using consistent scales will also make data comparable over time.
Follow Conventions for the Orientation of Answer Options
Scale or quantitative questions should be horizontal.
Categorical answer options should be vertical.
