If you’re anything like me, you’re often asked to participate in surveys. Some of them are short and simple. Others are very long, very complicated, or both.
You may also design and implement surveys from time to time like I do. If you want some insight into the effectiveness of your survey designs and their outcomes, pay attention to the responses you get.
Notice the Drop-off Points
Complicated surveys that take 15 or 20 minutes to complete tend to reflect drop off points at which the respondents decided that the time investment required wasn’t worth whatever incentive was offered. After all, not everyone actually cares about survey results or a 1-in-1,000 chance of winning the latest iPad, for example. If there’s no incentive whatsoever, long and complicated surveys may be even less successful, even if you’re pinging your own database.
A magazine publisher recently ran such a survey, and boy, was it hairy. It started out like similar surveys, asking questions about the respondent’s title, affiliation, company revenue and size. It also asked about purchasing habits – who approves, who specifies, who recommends, etc. for different kinds of technologies. Then, what the respondent’s content preferences are for learning about tech (several drop-down menus), using tech (several drop-down menus), purchasing tech (several drop-down menus), and I can’t remember what else. At that point, one was about 6% done with the survey. So much for “10 – 15 minutes.” It took about 10 or 15 minutes just to wade through the first single-digit percent of it. One would really want a slim chance of winning the incentive to complete that survey.
In short, the quest to learn everything about everything in one very long and complex survey may end in more knowledge about who took the survey than how how people feel about important issues.
On the flip side are very simple surveys that take a minute or two to answer. Those types of surveys tend to focus on whether a customer is satisfied or dissatisfied with customer service, rather than delving into the details of opinions about several complicated matters.
Survey design is really important. Complex fishing expeditions can and often do reflect a lack of focus on the survey designer’s part.
Complex Surveys May Skew Results
Overly complicated surveys may also yield spurious results. For example, let’s say 500 people agree to take a survey we just launched that happens to be very long and very complex. Not all of the respondents will get past the who-are-you questions because those too are complicated. Then, as the survey goes on, more people drop, then more.
The result is that X% of of the survey responses at the end of the survey are not the same as X% earlier in the survey. What I mean by that is 500 people started, maybe 400 get past the qualification portion, and the numbers continue to fall as yet more complicated questions arise but the “progress bar” shows little forward movement. By the end of the survey, far less than 500 have participated, maybe 200 or 100.
Of course, no one outside the survey team knows this, including the people in the company who are presented with the survey results. They only know that 500 people participated in the survey and X% said this or that.
However, had all 500 people answered all the questions, the results of some of the questions would likely look slightly or considerably different, which may be very important.
Let’s say 150 people completed our survey and the last question asked whether they planned to purchase an iPhone 7 within the next three months. 40% of them or 60 respondents said yes. If all 500 survey respondents answered that same question, I can almost guarantee you the answer would not be 40% . It might be close to 40% or it might not be even close to 40%.
So, if you genuinely care about divining some sort of “truth” from surveys, you need to be mindful about how to define and structure the survey and that the data you see may not be telling you the entire story, or even an accurate story.
The point about accuracy is very important and one that people without some kind of statistical background likely haven’t even considered because they’re viewing all aggregate numbers as having equal weight and equal accuracy.
I, for one, think that survey “best practices” are going to evolve in the coming years with the help of data science. While the average business person knows little about data science now, in the future it will likely seem cavalier not to consider the quality of the data you’re getting and what you can do to improve the quality of that data. Your credibility and perhaps your job may depend on it.
In the meantime, try not to shift the burden of thinking entirely to your survey audience because it won’t do either of you much good. Think about what you want to achieve, structure your questions in a way that gives you insight into your audience and their motivations (avoid leading questions!), and be mindful that not all aggregate answers are equally accurate or representative, even within the same survey.