Build Logic Based on Respondent Sentiment

If you are using a Textbox and/or Essay / Long Answer question to collect open-ended responses in your survey, you can now leverage Sentiment Analysis in your Logic conditions.

Alchemer uses Natural Language Toolkit (NLTK); specifically the Valance Aware Dictionary and sEntiment Reasoner (VADER) Sentiment Analysis Package¹ to analyze your respondents' text responses for sentiment, in realtime! Once the text is analyzed, a sentiment score is stored. You can then use this score to trigger questions, actions, or messaging in your survey via logic.

This allows you to tailor the survey experience based on an open-ended text response. For example, you might want to ask a respondent if you can use their response in marketing material, but it's probably inappropriate to ask that if the response was not positive. Using Sentiment Analysis you could trigger a follow-up question for positive responses only.

What is Sentiment Analysis?

Sentiment analysis² is the process of identifying (statistically) whether a piece of text is positive, negative, or neutral. In the case of Alchemer, the text refers to a respondent's answer to one of the compatible question types.

Compatibility

Sentiment Analysis cannot be performed retroactively. It is only performed on responses collected after the release of this feature.

• At this time, sentiment analysis is performed on English language surveys only. A sentiment score of 0 (neutral sentiment) will be returned for non-English open-ended responses.
• Sentiment Analysis is not performed on responses collected via Offline Mode.

The VADER Sentiment Analysis Package is automatically run on open-ended responses collected via the following question types:

• Textbox
• Essay / Long Answer

If either of the above questions is included as a sub-question of a Custom Group question, sentiment analysis will also be performed.

Sentiment Scoring

After a respondent's open-ended response is analyzed, a sentiment score is assigned to the response. The score ranges from -1 to 1. Depending on the specific score returned, the sentiment will be categorized as one of the following:

You can target these sentiment categories to trigger survey questions, actions, or other items via logic. For example, if a respondent's open text answer is deemed to have negative sentiment, perhaps you want to trigger an email action to your Customer Service team.

How is Sentiment Calculated?

Alchemer relies on the VADER Sentiment Analysis package to analyze* the open text data, as well as the sentiment score assigned to the respondent text.

*VADER Sentiment Analysis has been installed on Alchemer servers. Open text response data does not leave Alchemer servers.

The following factors contribute to determining the sentiment score³:

• VADER sentiment analysis relies on a dictionary (lexicon file) which contains sentiment scores for words, acronyms, slang terms, emoticons, and other things used for text-based communication.
  
  
  • Sentiment score within the lexicon file is measured on a scale from -4 (most negative) to +4 (most positive). 0 represents neutral sentiment. View the lexicon file content.
    
    
• The sentiment score of a sentence is calculated by adding up the sentiment scores of each dictionary/lexicon word in the sentence. A mathematical formula is used to normalize the score to a value between -1 and +1.
• Emphasis is added/subtracted based on punctuation. Depending on whether the overall sentence was found to be positive or negative:
  
  
  • an empirically derived mean increases/decreases the sentiment intensity rating (0.292 x exclamation point count, up to four). If more than four exclamation points have been added, the count is set to four.
  • an empirically derived mean increases/decreases the sentiment intensity rating (0.18 x question mark count If there are between 2 to 3 question marks). If there are more than three question marks, the mean increases/decreases by 0.96.
    
    
• An empirically derived mean increases (+0.733)/decreases (-0.733) the sentiment rating when a respondent uses all capital letters to emphasize a word (depending on whether the capitalized word is deemed positive or negative).
• The VADER sentiment package uses a dictionary of booster words. Using an empirically derived mean, these words either increase (+0.293) or decrease (-0.293) the sentiment intensity rating.
• The sentiment package also checks for the presence of the word but in the response text. All sentiment bearing words before the but have the rating reduced by 50%, whereas the words after the but have their rating increased by 150%.
• VADER uses a list of negation words. Negation words decrease the sentiment intensity rating by multiplying the sentiment score of the sentiment-laden word, emoticon, acronym, etc. by an empirically-determined value (-0.74).
• If the open text provided by a respondent contains multiple sentences, the sentences' scores are averaged to determine the overall sentiment score.

Sentiment Considerations

Before you proceed with using respondent sentiment for logic, consider the following:

• Creating logic based on sentiment is not the same as asking someone explicitly for a rating on a scale and then following up based on a specific answer. When you use sentiment for a logic condition, you are relying on the VADER package interpretation of the respondent text. Something that you might consider as a positive response may be analyzed as neutral sentiment, and so on.
• The mechanism behind the VADER sentiment analysis as well as the thresholds for the positive, neutral, and negative categories cannot be customized.

Set Up Logic Based on Sentiment

There are a couple of things that you will need to have in place before you can use the sentiment score in logic conditions:

• You will need to use one of the two compatible question types to collect the open-ended information: Textbox or Essay / Long Answer. Sentiment Analysis is automatically performed on these questions. The Textbox question must not be validated to collect email, date, number, or percent.
• The question or action that will be triggered based on the sentiment score cannot be on the same page as the above-mentioned open-ended question(s). The question or action must be on a later page.

Sentiment Logic Setup

Let's say you want to show a follow-up question or send an email to your Customer Service team based on customer feedback that was deemed to have negative sentiment. To do so, follow these steps:

1. Edit the question/action that you want to trigger and navigate to the Logic tab.
2. Check the Only show this question/action based on... checkbox.
3. Use the Select a condition dropdown to select a question that is being analyzed for sentiment. Remember, this must be a Textbox or Essay / Long Answer question.
4. Next, click on the Logic Operators dropdown menu and locate the Sentiment section. There are several options for sentiment-based logic.
   
   • Positive (> 0.3) - trigger the item if the sentiment score is greater than 0.3
   • Neutral (-0.3, 0.3) - trigger the item if the sentiment score is greater than or equal to -0.3 and less than or equal to 0.3.
   • Negative (< -0.3) - trigger the item if the sentiment score is less than -0.3.
   • > (greater than) - trigger the item if the sentiment score is greater than the score that you specify.
   • < (less than) - trigger the item if the sentiment score is less than the score that you specify.
5. Choose the appropriate operator for your scenario. In our example, we want to trigger our item based on negative sentiment, so we selected the Negative (< -0.3) operator.
   
6. Make sure to Save when you are finished setting up your logic condition.

Sentiment Score in Individual Responses

If you need to review the sentiment score for a given open-ended response, you can do so by navigating to Results > Individual Responses.

Click on an Individual Response that you want to view and locate the Textbox and/or Essay / Long Answer questions in the response. Each of these question types will have an associated sentiment score. The score will be displayed below the answer, and color-coded as follows:

• Positive Sentiment - green
• Neutral Sentiment - gray
• Negative Sentiment - red

Download Sentiment Score via Individual Responses

When you download and individual response to a PDF, you will have the option to include sentiment scores within the PDF.

On the Customize your response PDF screen, check the box associated with the Include Sentiment Scores option.

If a question is compatible with Sentiment Analysis, the sentiment score will be included directly after the question, within your PDF file.

Add Sentiment Chart to Standard Report

If your survey utilizes one of the question types that is compatible with Sentiment Analysis (Textbox, Essay / Long Answer), you will have access to a new Sentiment Question Chart reporting element within the Standard Report.

• The Sentiment element will display the sentiment for the associated question on a scale of -1 to 1 (in increments of .1).
• The sentiment will be color-coded along a range from red (negative) to gray (neutral) to green (positive).
• Each individual score will display an associated response count (along the y-axis). The taller the bar, the more responses with that specific sentiment score.

Add Sentiment Chart

1. To add a Sentiment Chart to your Standard Report, use the Insert tool between any two report elements.
2. After clicking Insert, choose the Question Chart option.
3. Locate the Textbox or Essay / Long Answer question for which you want to add a sentiment chart. Directly beneath the question, you will see an indented option that reads Sentiment - Question Title. Click on this option to select it.
   
4. Click Add Question to add the chart.

Export Sentiment Score

The sentiment score associated with each respondent's open-ended answer can also be included in a CSV / Excel Export of your survey data.

1. When you create an Export via Results > Exports, you will see a number of initial display options, including Include Sentiment Score Data.
2. Check the box associated with this option to include the sentiment score in your export file.
   

If a question is compatible with Sentiment Analysis, a Sentiment column will be included directly after the question, within your Excel file.

FAQ

Can I customize the scoring ranges for the Positive, Neutral, and Negative categories?

The default scoring ranges cannot be customized. However, you can create logic based on sentiment that is greater or less than a specific value.

Is Sentiment Analysis performed on translated surveys?

At this time, Sentiment Analysis is performed on English language surveys only.

 What happens if a word is misspelled?

Misspelled words will not be analyzed for sentiment as a match will not be found in the lexicon file. A neutral sentiment score will be returned.

Further Reading

• VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text
• VADER Sentiment
• VADER lexicon file
• VADER Sentiment Analysis Explained
• Using VADER to handle sentiment analysis with social media text

Sources

1. Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.

2. “Using VADER to handle sentiment analysis with social media text,” Standard error, last modified April 08, 2017, http://t-redactyl.io/blog/2017/04/using-vader-to-handle-sentiment-analysis-with-social-media-text.html.

3. "VADER Sentiment Analysis Explained," DATA meets MEDIA, last modified April 10, 2017, http://datameetsmedia.com/vader-sentiment-analysis-explained/.