Pitfalls in market research

Christmas tale

Account Director takes the Client for a pre-Christmas lunch. They decide to try to land a piece just after the Holidays, about how, come the January sales, everyone will be spending their Christmas gift tokens/ cash on the Client’s Brand.

AD gets back to office and briefs AM, slightly incoherently, to get some content to support the release.

AM decides to run a survey; gets the team together and “mind showers” some ideas for a survey. They come up with a list of 15 questions.

Job done, they ask Leslie, the new Assistant (who joined last week and was busy working when the mind shower was happening), to “contact ThyRule, or some other research company, and get them to do a survey on these questions.”

Leslie, with a 2:1 in Media Studies and Dunning-Kruger driven confidence, rings a random Market Research agency (usually OnePoll) requesting “A 15 question Omnibus of people who spend their Christmas gift tokens on <Client’s Brand>.”

What could possibly be wrong with that?

(You might know all these, but Leslie doesn’t. Leslie doesn’t know enough to know what Leslie doesn’t know.):

PITFALL: An Omnibus is just one sort of survey instrument – Omnibuses are, as the name suggests, shared use surveys. Typically they look at All UK Adults (or All Women, or All Shoppers, etc.).

An Omnibus is often not the right tool for the job. But, just as you’ll tend to create confusion if you wander into a car parts shop and ask for “A set of wheel bearings for a 2007 Focus” when you really wanted spark plugs for your Nissan Micra; if you ask for “Omnibus” up front, you tend to confuse the research agency if Omnibus isn’t really what you need.

For Leslie, there are two choices (there are three, really: the third is to go back to the AM and ask for clarification, but that’s not gonna happen).

Buy space on an Omnibus, filter out the people who don’t celebrate Christmas, ask the remainder if they ever get given gift tokens or cash, then ask this group the 15 questions.

Or ask for a special sample, of people meeting the criteria. Leslies generally make the mistake of picking the latter.

It’s an understandable mistake: you want to know about “people who spend their Christmas gift tokens on <Client’s Brand>” – so you start out with a sample of people who get gift tokens at Christmas. Don’t you?

PITFALL: for most research purposes, you need to compare the sub-group to the population. If you start with an Omnibus/ Nationally Representative sample, you can say how the results relate to the whole population. You can say (e.g.) “of the 10 million people in the UK who’ll get gift tokens or cash this Christmas, 4 million plan to buy < Client’s Brand>”.

If you only ever interviewed people inside that group, you don’t have a good idea of their incidence: your figures have no reference to the world at large.

Leslie, not wanting to go against the AM’s brief, is surprisingly hard to persuade to buy the correct sample.

The sample agreed, a price agreed for 15 questions, and with just 5 days to go before the market research ‘doldrums’ period around Christmas when everyone is too busy being, Leslie sends though the questions.

One of which reads:

“After Christmas, how often do you like to treat yourself to something, such as a new hat, a makeover or <Client’s Brand>?”

  • Very often
  • Fairly often
  • Not very often
  • Rarely
  • Never

So the Agency add the initial filter questions “Do you celebrate Christmas?” and “How often, if ever, do you get given gift tokens or cash for Christmas?”, generally tidy the questionnaire, and turn the muddle above into a grid question, with a row each for ‘hat’, ‘makeover’ and ‘<Client’s Brand>’.

Leslie rejects all the agencies’ changes, demanding the agency run the original 15 questions which the AM shared from the mind shower.

PITFALL: Too often, clients forget that market research agencies take an active role in the survey process.

Market researchers are answerable to their respondents, to the Market Research Society, the Information Commissioner, etc. We cannot always run questions ‘as received’.

We have to split questions, if we spot that there are mixed topics in a single question. We have to add routing questions. We have to add ‘Not applicable/ don’t know”, both to help respondents and ensure the data is reliable.

Leslie doesn’t understand this. Leslie has neither sufficient understanding of market research to appreciate the value of the changes nor sufficient seniority to approve them.

PITFALL: Don’t get the junior member of your team to buy market research. To buy something, you need a pretty good idea of what it is and how it works.

If you don’t know what a bearing looks like, how many you need per wheel and roughly what they should cost: it’s probably best to get your mechanic to buy them.

A miracle (just before) Christmas

We persuaded Leslie to get the AM on a conference call, discussed the issues and the day was saved. The client got his PR out in time and there were bonuses and mince pies all round.

Other Pitfalls Include:

“I need you to write a question to give me the stats I want.” One beauty of Market Research is we don’t know how it’s going to come out. There’s no way of knowing how people will respond to a certain question and therefore there aren’t any question formulas to “give you the stats that you want”.

“The results haven’t given me what I want, so the research is useless” – Wrong. It may not have given you the results you were expecting but there are always different ways to analyse the results which can throw up something potentially even more interesting. Think outside the box. Write the PR to the results, not vice versa.

Leading questions – Questions written to get the answer you want for your headline are not OK.

Respondent fatigue – should be considered. More questions is not necessarily better research, if you ask a question in the right way, there will be no need to have multiple questions, essentially trying to get the same answer. Respondents have as many time pressures as you do!

Not everyone celebrates Christmas – As in the main story – there are people in the UK who don’t celebrate Christmas. Questionnaires must be applicable to everyone!