At some point in your professional career – be it as a business owner, executive, manager, or consultant – there’s a high likelihood you’ll be part of creating a survey.
If that happens then you should know about three survey pitfalls. There are two about biases and one about expectations; and ignoring any of them can horribly skew your results.
I say this assuming you don’t want to horribly skew your results and that you would like an as accurate picture as possible from your survey.
Because, you can use what I’m about to tell you for evil. You can make survey results say whatever you want them to say by engineering these pitfalls to steer the outcome.
So don’t do that.
1. Sampling Bias
Not just any group will give you accurate responses for what you’re trying to measure. If you don’t take this into account you could get some results like this comic.
2. Question Bias
The way you word your questions absolutely matters.
Dr. Neil Postman, who taught at the New York University, sums it up best.
“A question, even of the simplest kind, is not, and never can be unbiased.
The structure of any question is as devoid of neutrality as its content.
The form of a question may ease our way or pose obstacles. Or, when
even slightly altered, it may generate antithetical answers, as in the case of the two priests who, being unsure if it was permissible to smoke and pray at the same time, wrote to the Pope for a definitive answer. One priest asked, “Is it permissible to smoke while praying?” and was told it is not, since prayer should be the focus of one’s whole attention. The other priest asked if it is permissible to pray while smoking and was told that it is, since it is always permissible to pray.”
3. Expectations
Know that, even if you do everything perfectly, your survey results have limitations.
No survey or data, (this includes website metrics, etc.) can tell you what to do next. It can’t tell you what changes to make. It can’t tell you what hazards to avoid in the future or how to grow.
The only thing it can tell you are the effects of what you did.
Surveys show you the results of past actions.
For example, if you have an oak sapling in your yard and every year you measure its height; that annual data will be a snapshot of how well you’ve taken care of that tree in the past. How well you watered, fertilized, and tended it. But that data can never tell you how tall that tree will grow.
You won’t know until it’s done growing.
Same for surveys.
The takeaway?
If you manage the above three pitfalls you can gather some useful data and insights from surveys.
From there, make your best educated guess on how to proceed and blaze forward.
– Zac Smith, VC
