Why Focus Groups and Digital Don't Mix

As a direct marketer, I am of the opinion that focus groups are where reliable statistics go to die.

This is adequately summed up in the following example: your dog does not sleep on the bed when you’re around, but as soon as you leave, that’s exactly where he heads. People behave differently when someone is watching them – it’s human nature, and focus groups are no different.

Focus groups are expensive, deficient, inaccurate and evil – and here are four arguments to support that. I love this position because it gets people all riled up and frequently spars debate on the benefits of qualitative vs. quantitative feedback. When you get right down to it, it is somewhat a matter of opinion, but hopefully I can persuade you to see the benefits of multivariate testing for digital experiences specifically with these four arguments.

1. People behave differently when under a microscope.Ah yes, the Hawthorne Effect: The alteration of behavior by the subjects of a study because they are being observed. If you know that people are behaving differently when they are knowingly under a microscope, how much integrity does the data really have? How do you know people aren’t just there for the coffee and cookies because they have a lot of free time, and have zero intention of ever purchasing your product?
    Also, if you have ever participated in a focus group, then you know that whoever is running the forum has a significant influence on the results, especially when there are creative people involved who already have an opinion on what direction they want to move forward with (think Mr. Burns commenting “excellent” and tenting his fingers at the head of the room).
    It’s human nature to look for those subtle signs and cater your reaction accordingly, which leads to tainted results. Online, you don’t necessarily know if anyone — or who — may be watching (actually, I just assume someone always is, but I work in this industry). As a result, my reactions are more native, less influenced, and therefore more actionable from a marketing perspective.
    
2. Focus group audiences are not reflective of your actual audience – especially online.Take size out of the equation as you simply can’t compare the impact/validity of the behavior of thousands of website visitors to however many focus group participants who:
    a) Want to participate;
   b) Meet the selection criteria;
   c) Fit into conference room B that you booked for this purpose.
    A focus group audience is selected, and someone had to make a decision to arrive at that selection, hence an automatic bias. Additionally, it’s very difficult to mimic the natural diversity of a digital audience.
    On the flip side, a multivariate experiment encompasses all the actual traffic that is arriving on your website via numerous channels and at various stages of the purchase cycle with different commitment levels. As an aggregate, those results are going to be a more accurate reflection of your true audience receptivity and behavior than anyone you pick off the street or segment from your customer database.
    And you don’t have to spend time educating them on why they are there to begin with!
    
3. Results obtained through focus groups cannot be tied to an actual purchase.In my experience, focus group questions always seem to ride the grey line, leaving room for interpretation in the results and further impacting the validity of the data. Additionally, my experience with or exposure to your product or service does not mean I will necessarily buy it now or ever. Wouldn’t you rather know which elements or variations your prospective audience is responding to that actually drives them to purchase? Think of how much more useful those insights would be. Realistically, why do you care what 15 people who may never buy your product think of the creative direction?
    
4. Focus groups bleed money.Brand agencies love focus groups because they are expensive and take a lot of “strategic brainpower” to plan and oversee (cough, billable hours). Plus, they can spend even more hours sitting around pontificating on the sheer simplicity (second cough, basic logic) of the findings after the fact. Oh, to live in that world.
    You can spend a lot of time and energy on focus groups only to end up marginally ahead (if at all) of where you were based on the assumptions going into it. On the flip side, you can be up and running with a fairly simple split A/B or multivariate test experiment in a number of hours, and much more cost-effectively (some basic testing programs are even free). You can also test more assumptions over a shorter time period, depending on the amount of traffic you have driving to your site. Forget about hand selecting your focus group.
    
   With the click of a few buttons, you can make all of your website traffic immediately part of your focus group segment. And you didn’t even have to buy them lunch!

In sum

Real-world data is always better. Period.