A/B Split Testing – Does it Help Email Marketing?

Ben — Tue, 26 Aug 2008 17:07:37 +0000

Ever since we launched our automatic email optimizer, we’ve been tracking all this wonderful A/B testing data on our system.

We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we only included campaigns sent to over 500 recipients).

Overall, we found that A/B split winners did better than non-tested campaigns. By how much?

Before we give you the exact numbers, test your email marketing aptitude and pick the winning subject line from these real life A/B email tests conducted by our customers:

Winning Subject Lines by Open Rate
Test Group “A”	Test Group “B”
Don’t Forget Mom…	Gifts Mom Will Love!
Protect Yourself From the Sun This Summer…	The Best Sunscreen, Hats, Rash Guards and More…
[COMPANY]: March [COMPANY] Mail	Spring Update from [COMPANY]
[COMPANY] March Newsletter	March is National Crochet Month
[COMPANY]: France River Cruise & Golf – Last Call	[COMPANY]: Last Call for France River Cruise & Golf
Computer Security Quarterly – Spring ‘08	2007 Record Year For Data Breaches
Coupon Enclosed – Save on Printer Ink, Toner, Paper, Storage Media Now – HP, Xerox, Canon, Brother, Dell & More	News & Discounts from [COMPANY] – Savings on Printer Supplies from HP, Xerox, Brother, Canon and More
[COMPANY] Easter Newsletter – Free Shipping	[COMPANY] Easter Newsletter

Think you know which ones were the winners? They’re all the subject lines in the first (A) column. In general, we found that:

Subject lines with company names in them did better, and they did better when the company name was near the beginning of the subject line (perhaps it was the increased recognition factor)
Shorter subject lines seemed to work better than long subject lines (the difference in open rate was more noticeable when they differed by 30 characters)

How Much Better Are A/B Tested Email Campaigns?

We compared the email marketing stats of A/B testers against the average email marketing stats across all MailChimp customers.

Average open rate across all lists: 21.7%
Average open rate for winning A/B groups: 24.1%

Average click rate across all the lists: 4.7%
Average click rate for winning A/B groups: 5.5%

*We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we filtered down to emails sent to over 500 recipients).

As you can see, those who take the time to A/B split test their email campaigns get better overall results than those who don’t. For some extremely large lists, those few percentage points can make a huge difference in eyeballs, clicks and conversions.

Microsoft Word and Outlook 2007 – Bad Mix for Email

Ben — Thu, 20 Mar 2008 15:29:42 +0000

We just had a customer create her email content in Microsoft Word. Then, she pasted from Word into her HTML email, and sent a test to herself. That’s normally when most people learn how bad Microsoft Word is for making emails (it adds a whole bunch of code that breaks your email design). But this user never caught that, because she checked her test email in Microsoft’s Outlook 2007 (which uses Microsoft Word to render HTML email). So to her, everything looked just fine. So she sent the broken email to her entire list. Ouch.

In our experience, Microsoft Outlook has always been a very “forgiving” email app. Almost anything will work in Outlook, including videos in some cases. So if you want to do some serious email campaign testing, download a few other email programs. Mozilla Thunderbird is a good start. Of course, if you don’t have the time or desire to download a bunch of email programs and sending lots of test campaigns to each one, you can always use our built-in Inbox Inspector.

MailChimp Email Marketing Blog » email testing

A/B Split Testing – Does it Help Email Marketing?

How Much Better Are A/B Tested Email Campaigns?

Microsoft Word and Outlook 2007 – Bad Mix for Email