Does Size matter? - How many questions is too many in your online survey?

Surveys are all about collecting information. To get the right information you need to ask the right questions. While 'the right questions' is covered elsewhere, here I am going to discuss the number of questions as an aspect of design.

As with most things the answer is "it depends". However with online surveys a good rule is the fewer the better. How many times have we seen a great focussed survey get a poor response because every person in the approval chain has decided to add another question? The logic being that seeing we are asking people questions I would like to add just one more. This inevitably will be somewhat off topic and with each additional item the integrity of the survey dies a little death.

Rule 1 - Focus the survey! As soon as you wander, even slightly, the respondents will start to question why they are responding at all. I must say the most successful surveys are the 'front of mind' type topics with up to ten questions. The questions follow a logical progression to help the user think about the answers in a logical way. For example you may ask about? 1. what they thought the process was like before 2. did they feel involved in the problem identification process 3. did they feel all problems were identified 4. did the upgrade plan make sense to them 5. was there sufficient communications during the upgrade 6. did the new system live up to what they expected 7. What could have been done better Can you see how the questions followed a rational order, in this case time/process based?

Rule 2 - 10 minutes is an inconvenience, 15 minutes is an investment. When you ask users to respond to a survey you are asking them to spend some of their hard earned time for you. - What are they getting in return? For many surveys it is little if anything. Most people will spend up to 10 minutes out of charity. Anything over this and they will be expecting something in return. It may be as basic at a raffle ticket or as complex as a detailed analysis of their response against a broader population of respondents. Either way it needs to have value to them and be clear at the outset.

This brings me to my next point: No matter how long your survey is, there must be an introduction that states what is expected of the respondent and what they are going to get in return. This is usually put as how long the survey is expected to take (don't lie!) and what if anything they can expect in return. Eg. "?will take approximately 10 minutes and will help lead us to devise a system that works to your best interests."

Rule 3 - Trapped users don't care. We have all done these sorts of surveys, where you start and it seems to go on and on. You get lost as the questions seem to repeat and you are getting no closer to the end. In frustration you then start selecting the mid range response without really reading the question. Not I! I hear you saying. Can I tell you that statistics imply that this starts to happen to some extent after the first ten questions! With only limited investment users can quickly turn to prisoners looking for a way out ignoring you hard thought questions.

I want to make a last note here. A short survey is five questions of a multiple choice type, say radio button or checkbox. A long survey is only 20 questions. Think about why your respondents are investing the time. If you have come to the lower limit of your questions and you are still well over twenty, you might consider putting the more critical ones towards the front.

If you have any topics you would like covered please mail me at SteveWrightMail@gmail.com and I will include them in my growing list.

cheers Steve

Hi, after years of designing survyes I am collecting together some of the key lessons. http://surveysurvival.2leadership.com