Why it should be "target & test", not "test & target"

Here is a small nugget from the Feb 5th Webinar with Josh Manion and Bryan Eisenberg. I touched on this in my little intro during the webcast because it is the source of a fascinating disagreement. It is also in the Multichannel Metrics book in chapter 3.


When an online and an offline direct marketer discuss their approach to outbound marketing, this is the one that they disagree about. I am talking about email or SMS channels for the online marketer and direct mail, telemarketing, or direct sales for the offline direct marketer.

Say that you want to cross-sell brokerage accounts to customers who have nothing but a checking account at your bank. This will grow the wallet share that you have with customers and make them less likely to leave to competitors. You design an attractive offer with appealing creative.

But who should you send that offer to? Everybody for whom you have contact information and permission to market?

Certainly not, says the offline direct marketer. The table below shows different sizes of mailings and their effect on your profits. Click the table to expand it in a separate window.

Targeting vs testing outcomes

The big apparent difference between online and offline is that offline marketing has a higher variable cost per contact, e.g. call center costs, postage and production fees for mailings or direct sales costs. In contrast, sending an extra email or SMS or serving an extra banner, mistakenly, seem close to free.

Therefore, the offline direct marketer has to take these costs into account from the beginning. 

The cost per contact is the economic pressure for targeting before testing.

  1. Look at column 1 of the table, This is the untargeted and untested campaign. Since the target group has been selected too broadly, you will have no chance to break even. The campaign is simply too costly and typical conversion rates these days are too low.
  2. Column 2 is after applying super star testing. We now increased the conversion rate to 1%, a whopping five-fold increase!!! Yet, the untargeted campaign still loses money because it is too costly.
  3. This is why the offline marketer has been obsessed with improving conversion rates by targeting the most promising segments.  Column 3 shows that. Reduce the campaign to the most interested 10%. Costs go down and conversion rate for this segment goes up. Now we have a profit.

How to Target? 

Various ways of targeting campaigns range from predictive analytics at the high end to simple common sense list selections, suppression rules, etc at the minimum end of the spectrum. There is a lot to be said about the data to use for better targeting. May I recommend a new whitepaper on just that subject: Making Web data personal.

What’s it to me, says the Email and SMS marketer

Having a negligible cost per contact for email and SMS, the default approach online has widely been to ignore targeting, especially predictive analytics.

But there is a hidden cost that we have too often ignored online, namely the opportunity costs inucrred by spamming audiences and wasting our future permission to market.

In the sense that Peppers and Rogers emphasize in their recent books we incurr a real cost when we spam. Namely, the future potential purchases by a prospect become a little less likely since they aren’t listening to us. Peppers and Rogers would say that our expected life time value probably goes down for those recipients who feel spammed.

If the online marketer takes that cost into account, the economic pressure is now very similar to offline marketers. There now is a real variable cost per contact to reckon with.

Therefore, the calculation in the table above is just as relevant to the online and shows why targeting has to come before testing.

10 Comments on “Why it should be "target & test", not "test & target"

  1. This feels like an argument of words. What you’re essentially saying is that people should “target THEN test”. If you had to choose one over the other, then I think you’ve made a strong argument for why you would target first. However, the nice thing about optimizing online is that you do have the ability to do more with lower costs, and get results in real-time! Imagine if you took the same table in your example and applied the 500% lift from the testing of all traffic to the testing of targeted traffic. Now you’re looking at a 15% conversion rate for your targeted traffic by diligently testing as well.

    Also, there are some segments that are clear cut for targeting, such as the top 10% of your contact list. But what about the remaining 90% that is more fuzzy? Perhaps we still have segments that we can form hypotheses around, but there is not a clear consensus around how to provide the most relevant content. That’s where testing with segmentation becomes invaluable, and will ultimately lead back to targeting.

  2. Hello Lily,

    Thank you for the challenge. We are in agreement for the most part. Online and offline direct marketers both endorse testing fully and aim to get to that 15%, as you so rightfully point out. It is just that they first focus on segmentation and targeting, and then look for optimization within those segments.

    After some more thought let me add a clarification: Why do segmentation and targeting come first? That is because, as the example shows, testing alone isn’t going to turn an unprofitable campaign into a profitable one. Seems like I wasn’t able to make that point as clear as I thought.

    Great addition there that you raise about the long tail! Offline direct marketers and stores cannot address the long tail economically. But online marketers should also refrain from potentially spamming the “unprofitable” segment. They are “unprofitable” because the prediction is that they don’t think the message is relevant to them.

    What online should do however is to be ready with PPC, SEO, and web site if the prospects in the long tail start the initiative themselves. And this is what online is very good at today, i’d say. I think we are better in serving inbound inquiries vs. outbound.

    BTW: The 10% is just for argumentation. Targeters actually calculate the X% at which their campaign is predicted to be most profitable. For online, that kind of calculation will however require an answer to Jacques question.

    Adding after more thought: Another great thing that is possible online for the long tail is personalizing each communication down to the 1:1 level. Even though offline direct marketer now have print on-demand available to them, for the most they work with a limited set of offers, brochures, etc. that they map to each recipient. But an online marketer can create an unlimited amount of personalized email simply by dynamically displaying the products in the email that each recipient has shown that they have been interested in. Think of Amazon or Ebay as examples.

    So this enables marketers to activate additional of the deciles that the offline direct marketer ignores. But there will always be the rock-bottom deciles that just simply aren’t interested in buying whatever you have.

    Thanks much for the thoughtful comment! I will ignore the first line though 😎

  3. Hi Jacques,

    Good question! And Peppers & Rogers get the exact same objection to their recommendations in regards to life time customer value and their ROC metric, return on customer.

    In theory this cost is:

    * Total amount of future lost business because of spaming
    * divided by
    * number of customers contacted

    to get to a cost per customer.

    Tough to calculate, for sure.
    Let me add a note here after thinking about it some more: This cost is hard to calculate. But that doesn’t mean it is not real. It is very real. We are probably not going to be able to “calculate it” because life time value is a predictive metric anyway and life time value changes every day based on external events too.

    So the question remains whether we can estimate this cost?

    Can try as follows: How much does it cost you today to gain 1 new customer contact. Then that is the cost per customer where you lose permission to market.

    That leaves us with the need to estimate how many customers we lose permission after sending them untargeted email, SMS, etc..

    That would be

    1. the number of opt-outs we get
    2. + the number that deflect all future communication to their spam folder,
    3. + the number that will ignore and not open but maybe delete future email< SMS, etc. #1 is easy to get. #2 and #3 are very hard. I think one could theoretically measure #2 + #3 by assigning a control group that won't get the email or SMS. Then you compare how many from the control group never again show up on your web site vs. how many from the test group. If I am correct here we should find that many more from the test group ignore us in the future. But perversely, it could also turn out sometimes that untargeted email works similar to the TV commercials that we hate. Even though we hate having to delete and ignore it, seeing the sender's name may keep it alive in our heads. I hope that isn't how it works, otherwise un-targeted email will never stop. Akin

  4. Opportunity cost is the most overlooked variable in online marketing, and (I believe) that is because most onliners do not use null control groups. They are always testing one thing against another, as opposed to one thing against *nothing*. Unless you do that, it’s very difficult to measure opportunity costs – do you know how much activity occurs when you do nothing at all?

    The other large factor is faulty financial models. For example, what is the cost of an unsubscribe? Do you include that cost in your campaign profitability model? At the least, the cost of an unsubscribe is the cost to replace the subscriber, or acquisition cost.

    Great example of a rigorous online test and financial model here:


    Of course, if your focus is on Sales and not Profits, none of the above matters. But ask yourself if that kind of focus makes any sense at all.

  5. Hi Jim,

    What a great and relevant article! Thank you for pointing it out, especially in the light of Jacques’ good question.

    Quick stats that stood out to me:
    * $14 cost of acquiring a new prospect which makes the mere $3.50 cost that I used in my direct marketing campaign example look really cheap
    * This particular company was able to estimate 140,000 additional lost subscribers due to increasing email frequency from 5x to 15x per month. Not only did their unsubscribes shoot up to 2% but also their undelivarables. (I am not super sure whether this metric is really as readily available for individuals as they portrayed it here.)
    Annual lost subscriber rate shot up from 14 to 46%. I’d argue that many of the remaining 54% are likely also lost. After all, don’t we all have extra email addresses that we use for subscribing but that we never actually really read. Well, those mistakenly show up as continuously subscribed.
    a.) the cost of going too far with untargeted email is very real.
    b.) Receiving permission to market to the real email address of a prospect, i.e. the one he or she really reads, is a precious thing that must be handled with care.

  6. Fabulous debat. As a ‘old traditionnal Direct Marketer’
    I know by numbers that targeting is the only way to improve ROI.
    Unfortunately, too many times (in France) online campaign are not mesured until the ROI.
    Thank you all of them to this brillant discusion.

  7. Merci Laurence! Thank you for reading and commenting. I too heard that sometimes online campaigns are treated like brand advertising and not measured. Something that will change no doubt.

  8. The other challenge is to truly account for costs. Many of my clients look at their cost of acquisition in terms of hard costs alone, and often find the payback coming by the 2nd transaction. When I point out all the soft costs (including customer service support, sales/store training and internal communications materials), the costs increase so that it takes to the 3rd purchase (or beyond) to payback the efforts.

    If marketers compared more complete costs to response, they might wind up with vastly different “winning” strategies.

  9. Thanks much for the comment and thought!

    This reminds me of a parallel anecdote that I recently learned. Nothing to do with marketing but listen:

    In the grocery stores they keep asking us whether we’d like paper or plastic bags. I of course always picked paper thinking that it is easier to recycle. But it turns out that when you add up THE TOTAL COSTS of truck’ing tree logs and paper around the toll of paper bags on the environment is worse than plastic bags.

    So contrary to “near sighted metrics”, after looking at the total cost this speaker recommended plastic. (he was from Environmental Defense, if I am not mistaken, so seemed to have done the math for real)


Leave a Reply

Your email address will not be published. Required fields are marked *