A/B Split Testing Your Email Marketing Campaigns

What's the best day to send your email campaign? Weekdays? Weekends? What time? Early morning, or late afternoon? What kind of subject line works best? Something with a hard sell? Or something more subtle and informative?

The answer is, "It depends." It depends on your subscribers, your content, and your business. At MailChimp, we've done better with campaigns sent early in the week, just before or after lunch (when people are presumably checking up on email at work). But we've seen restaurants and nightclubs do better with email campaigns sent closer to weekends, right before quitting time (when people are planning their weekend activities). We've seen some real estate marketers do well with Saturday and Sunday campaigns (when people have time to drive around and do some home shopping).

A/B Split Tests: The Only Way To Know

The only way to know for sure what works best with your email campaigns and your subscribers is to experiment. And the most common way marketers test their campaigns is with A/B split tests. If you're new to the concept, an A/B split test is where you take your customer list, split it into two random samples (A vs. B), and then you try 2 different things with it (like 2 different subject lines). Whichever one performs better, you use that subject line to send the next campaign (if you like pictures, here's a diagram of A/B split tests). Sound really nerdy, complicated and time consuming? It has been, until now. We just made A/B split testing "MailChimp Easy."

How does A/B Split testing work in MailChimp?

Begin by clicking the "Create A/B Campaign" button on the MailChimp Dashboard:

This will take you to a page that lets you specify the criteria for your A/B test:

1. Choose what kind of test to run
2. Specify the size of the sample we send groups A and B to.
3. Tell us how you want us to judge the "winner" between A and B.
4. Specify when we should analyze A vs. B's stats to pick the winner.

So let's say you've got a campaign you want to send out, promoting a "free shipping" offer for the holidays. You're not sure if a "hard sell" subject line would work in this case, or if you should go for a more subtle approach.

So you specify a "subject line" A/B test like this:

In the example above, MailChimp will take a random 10% of your list to form "Group A" and another 10% of your list to form "Group B". So we've basically taken 20% of your list to run an A/B test with.

Automatic A/B Testing

Here's the part where we save you tons of time: When MailChimp sends the A and B campaigns, it will wait 8 hours (or whatever time frame you specify---we've found that 6 hours is long enough), then automatically check the open rate of A vs. B to pick the "winner." Then, MailChimp will automatically send a copy of the "winner" campaign to the remaining 80% of your list. No need to log in to check stats. No need to generate another segment of your list to re-send a campaign. Why go through all that hassle? Isn't that what chimps computers are for?

Anyway, since we've chosen to test 2 different subject lines, the Campaign Setup screen (the next step) looks like this:

As you can see, you just enter your 2 different subject lines. Then you simply walk through the remaining steps as you always do in MailChimp. If you were testing "From-Names", you'd see 2 different from-name fields.

If you were testing delivery dates/times, you'd see this on your final screen:

Above, you'd schedule two different delivery dates, or two different delivery times. If you're testing different days, try to keep the delivery times the same (and vice versa). You basically want to limit your variables to conduct a more proper experiment.

Note the Miami-Vice-style color schemes. Group A is always hot pink (Crockett), and Group B is always cool blue (Tubbs).

This color scheme gets carried through to your campaign report, where we show you how the A/B test went, and then give you overall stats on the campaign after being sent to the entire list. A/B test campaigns appear on your Dashboard page with the new A/B Stats icon . When you click on the icon, you get a reports page that looks like this:

Always Be Testing.

The important thing about A/B testing is that you do it often. You should always be experimenting and learning what works best to optimize your email marketing. At MailChimp, we felt that the only way our customers could test often is if we make it unbelievably easy (automagically easy), so that's why we made our A/B testing tool easy to understand, and totally automatic. This is something very unique to the email marketing world (so unique it's patent pending). We hope you use it often, learn from it, and grow your business with it.

Here's an inspirational quote from the folks at MarketingExperiments.com:

"This brings up another important point. Testing yields the most valuable results only when you test repeatedly. A one-shot test will tell you very little. But when you make a consistent habit of testing, cumulative tests over time can have a dramatic impact on the success of your site." - MarketingExperiments.com

So what are you waiting for? Start experimenting! It couldn't be any easier than this. Want some ideas for what to test? Scroll down to the "other resources" section below.

Other A/B Split Resources:

• How to use A/B Split Testing to Increase Conversion Rates MarketingExperiments.com
• With A/B Testing, You don't think-you know - Jeff Campbell, Resolution Media
• How to Improve A/B Testing: Generate screenshots of your email designs in all the major email programs, test all the major spam filters and email firewalls, and scan for spammy keywords in one click.
• Subject Line Testing Scenarios - A handful of very quick, easy subject line a/b tests you can run right now.
• Email Marketing Needs A/B Testing: A fan of ConstantContact.com discusses the need for A/B testing
• The Most Important E-Mail Marketing Tactic of All: Bill Nussey of Silverpop discusses why A/B testing your email marketing is so important, and gives some suggestions on what to test.
• Testing and Email Marketing: A list of resources about testing your email marketing campaigns
• 10 Quick Wins for Email Marketing: #7 - Testing (tips from iMedia Connection)
• Incorporate Testing Into Your Game Plan: Blog post from David Baker at MediaPost
• Online Copywriting 101: The Ultimate Cheat Sheet: Read the tips for writing headlines (use them for your subject lines---after testing, of course).

Anti-Spam Policy & Terms of Use  |   Privacy Policy  |   About  |   Contact  |   Help  |   Search  |  Site Map

Email Marketing Resources  |   Free HTML Email Templates  |   Partners  |  Find an Expert  |   Add-ons  |   API  |   Blog  |  Demos  |  Surveys  |   Webinars  |   Subscribe to our free Email Marketing Tips Newsletter

New Features:  A/B Split Testing Emails  |   Google Analytics Integration  |   Email Authentication  |   Segmentation

© 2001-2008 MailChimp. All Rights Reserved.  |  Toll-Free: 1-866-284-2541