This isn’t the first we’ve seen of PR email spam…

Chris Anderson, editor-in-chief at WIRED magazine, pissed off a lot of people in the PR world back in 2007 with: Sorry PR people, you’re blocked. Ironically, his own PR firms have been accused of sending PR spam to his benefit.

Clearly, PR people are in a catch-22. They want to get the word out to the press, and email seems like a fast, cheap distribution method. It’s just that when you send unsolicited email in bulk, that is the very definition of spam.

One more time: Sending a personal, one-to-one email to a journalist is fine, even if it’s unsolicited. That’s called “doing business.” But when you cross the line and send in bulk, a whole new set of netiquette (and law!) applies. You just can’t assume you have permission.

To send email in bulk, you have to 1) earn permission, then 2) send your PR emails responsibly (here’s how one company, in the anti-spam industry, sends PR email responsibly).

It’s that first part, “earning permission,” that’s hard. Some people call this “growing your subscriber list,” but the problem is a lot deeper than that. Tactics like “include a link to your signup form everywhere!” are fine and dandy, but they aren’t going to make a huge enough impact on your email list size.

Be an expert. Or, just be more interesting.

Fundamentally, it’s about making yourself and your content actually useful to others. If we think you have interesting insights, connections, or news, we’ll subscribe to receive your emails. Wait. Honestly, no we won’t. That’s admittedly a stretch. I would never sign up to receive email newsletters form a PR person.

But if you say interesting stuff on twitter, people will follow you there. I think Jim Caruso, from MediaFirst, does this well. He’s been at every single technology event I’ve ever attended in Atlanta for the last 10 years. He knows what’s going on. He’s a technology geek at heart (who needs to stop hacking at his website and just hire a web designer, for pete’s sake! 3/6/10 UPDATE: Looks like Jim recently did! Sorry, Jim!). And he’s on twitter, tweeting about local startups, global technology news, and of course, his own clients.

I follow him on twitter.

Full disclosure: MailChimp was once Jim’s client (and a happy client, too) but we ultimately stopped doing PR after deciding that the email industry is still very young, and all our press releases were just being read by other ESPs.

Anyway, twitter is perfect for PR professionals, because you can tweet about anything you want, and only interested people will actually follow you. Then the question becomes how to get found on twitter. Yeah, we’re back to the “be more interesting” part. It just takes time. But there are tools out there, like Journalistics, that can help you use twitter in a non-spammy way. Go do that. Not mass email.

How PR professionals should use email marketing

Okay, so besides Jim, I’d probably never, ever, ever subscribe to any PR firm’s list. Ever.

I’d more likely subscribe to receive emails from a company that a PR person represents. So the sequence of events goes like this: PR person represents Acme Global, and tweets something interesting about that company. It gets re-tweeted, and re-tweeted, until I see it in my own twitter stream, thanks to someone I respect and follow. I click the link, learn more about Acme Global, and if they seem interesting (i.e. offer some kind of useful content), I sign up for their emails.

So whenever a PR person asks me “Can I use MailChimp for my business?” I tell them, “If you mean sending unsolicited mass emails to journalists on your clients’ behalf, then no. For the love of all things holy, no no no. That’s a violation of our terms of use. If you mean sending newsletters to your own clients, then sure. And if you mean helping all your clients setup double opt-in email subscription forms on their websites, so that you can help them responsibly (and consistently) send out truly useful email press releases to journalists who actually cover your clients’ industries and who opted-in to hear about your clients, then absolutely MailChimp can help you.”

Furthermore, these features might actually be useful for you and your clients:

• Google Analytics integration with MailChimp helps you measure ROI from email to website purchases.
• Twitter re-tweet tracking for your emails, and more email-to-social integration
• Geomap and click overlay for MailChimp emails

Okay, back to the topic at hand. If you’re the stubborn or paranoid kind of IT person who really, really, really wants to build your own email delivery engine, and you don’t want to use a service like MailChimp, that’s cool. But setting up a mass email infrastructure (with great deliverability) is hard, and there are things you’ll need to know about selecting your MTA, pitfalls in cloud-computing IPs, selecting the right hardware, proper bounce handling, ISP rate limiting, security concerns, abuse monitoring, blacklists, reputation services, and on and on.

You’ll need to get your hands on some kind of super-secret, industry-insider, reveal-all kind of guide. Our new Deliverability Engineer, Brandon, just wrote that guide…

Email Delivery (for IT Professionals) – 1.4MB PDF

Nice work, Brandon. Your first project after joining MailChimp is expose all our delivery secrets?

Seriously, this stuff is really hard. If you’ve gotta do this on your own, we want you to do it right. We’ll tell you the knobs and switches you’ll need, but we’re not going to tell you the exact settings (c’mon, that’s the fun part!). And hopefully, some people out there will come away with an appreciation for how complicated mass email delivery really is, and just use MailChimp.

For example, I sent this MailChimp Newsletter to a little over 10,000 recipients a week ago, and 48 people have since unsubscribed.

To see the reasons, I go to my campaign’s report, click on the “unsubscribes” link:

and you’ll see everybody that unsubscribed after receiving that campaign, and why.

Notice these stats are downloadable to Excel spreadsheet.

Hmm, when I downloaded my unsub-reasons, I sorted by the “Member Rating” column. This is our measure of engagement, and I just think it’s weird to see that my most engaged subscribers would choose to unsubscribe.

Turns out they were also some of my most enRAGED customers. Check out some of their reasons for leaving:

this, of course, prompted me to check our customer service ticketing system to see what might’ve happened. Nothing there.

So I checked our Abuse Desk ticketing system. Yep, turns out we had to say goodbye to them for some abuse related issues. Guess they weren’t happy with that.

Here’s one where someone was really angry, and said he switched to one of our competitors:

Again, abuse related issues. Backtracking through the tickets, I saw that their campaigns caused problems, our Omnivore system flagged their account, and our abuse desk staff reached out to him and tried to help him resolve the issue. But in the end, the account was closed. Wish we could’ve kept these customers, but we can’t please everybody.

And as you can imagine, when they get a MailChimp newsletter about how happy our little family is over here, they’re not very happy for us. I’m tempted to ask our abuse staff to just unsubscribe anybody from all our newsletters if we have to shut down their account. That would certainly prevent anyone from maliciously reporting us to their ISPs or the spam cops of the world.

In fact, when I search for these members across all my other lists:

I see that these very same (now angry) people had also signed up to receive some of my other newsletters (my MonkeyWrench Newsletter is published on a different schedule than our company newsletter).

Dilemma. I’m planning to send my next MonkeyWrench newsletter within the next few days. Surely, they’ll complain again when they receive that one. They’d almost as likely complain if I unsubscribe them w/out their permission though. Hmm, what to do.

Luckily, MailChimp keeps records of opt-in IP address, along with a timestamp, so we have plenty of proof that they opted in on their own free will. When I click into their profiles, I get their opt-in proof:

This is exactly why we always advocate using the double opt-in method for subscriptions. They can prevent false spam complaints from ruining your deliverability.

As you can see, unsubscribe “exit surveys” can tell you a lot more than whether or not subscribers like your content.

But I did find this comment from Laura very relevant to something we’re doing at MailChimp:

“Quite frankly, I am unsurprised by this. My impression of Goodmail has always been they never really understood the role of a certifying agency. For any certifying agency to be successful, they must continually monitor certified customers and enforce standards. Goodmail’s initial certification process was fine, but they never seemed to follow through on the monitoring and enforcement.”

That part about how “they must continually monitor”? I can’t blame Goodmail. That’s extremely hard to do! In a way, MailChimp tried to do this sorta thing ourselves…

We used to give preferential treatment to reputable accounts who passed an internal gut-check. But we learned the hard way that even good users can go bad, and then taint your best IP range.

To truly monitor a sender and protect your infrastructure’s deliverability from bad behavior, you have to watch all their activity, 24/7. And that’s hard for an ESP in the self-serve business like us. We have a  human abuse desk team. But continually monitoring 260,000+ users sending close to 20 million emails a day? Hiring more human reviewers is simply not scalable.

This is the very reason we started the MailChimp Omnivore project.

In late 2008, MailChimp Labs began Project Omnivore. Our goal was to build a massively scalable tool for our abuse team that could predict bad behavior.

The experiment started with an nVidia Tesla supercomputer, then grew to a cluster of Amazon EC2 servers running a genetic optimization program for 2 weeks nonstop, running over 61 trillion email data comparisons.

This article shares some of the results of our experiment, and where the technology is taking us…

Why Is Omnivore Needed?

You know what the hardest part of running an Email Service Provider (ESP) is? Detecting ignorant spammers. They’re very different from evil spammers. See, it’s pretty easy to detect “evil” spam. You know, the pharmaceutical appendage enhancing stuff, phishing scams, and Nigerian prince (419) junk. Spam filters actually do a really good job of catching the evil stuff nowadays (not perfect, but pretty darn good, all things considered). And most ESPs employ some kind of spam filter (usually a variation of SpamAssassin) to scan outgoing emails in their queue. Either to prevent evil spam from tainting our reputation, or to “grade” the spamminess of a message.

But those spam filters aren’t designed to detect when an ignorant marketer doesn’t realize he’s spamming, and sends a mass email without permission (remember, the definition of spam is “unsolicited bulk email“).

Lack of permission, in an otherwise perfectly legitimate looking business email, is very subtle and much harder to detect.

I’m talking about when a well-meaning small business owner just wants to get the word out about his new store, and “blasts” an unsolicited email to a list he obtained from his local chamber or from a tradeshow. He didn’t mean harm, and he thinks he’s “just doing business,” but he’s actually spamming. While it’s a different flavor of spam, it’s still spam (again, see: definition of spam). This kind of spam is hard to detect because the content is often perfectly fine and doesn’t contain the normal keywords or traits that spam filters are trained to look for. But this flavor of spam can cost an ESP dearly, because they tend to generate the bad kind of engagement (high complaints, high bounces, high unsubs) that can get our IPs blacklisted by email gateways and ISPs.

How exactly does one detect the lack of permission in someone’s account? Across over 230,000 accounts? Sure, we’ve got a well-trained compliance team who can review a new user’s account, and in the blink of an eye, judge whether or not they’re going to cause trouble for us. But as good as we are, a human review team is just not scalable enough to deal with hundreds of thousands of senders. Not to mention that someone we might approve as a “good sender” can eventually become a “bad” sender. Rigorous, 24/7 account review becomes a necessity.

So our abuse desk decided long ago that we had to change the way we think about handling abuse. We began experimenting and analyzing massive amounts of data in 2008, which led to our list activity score feature. The idea here was to stop classifying customers as good or bad (and giving them access to special IP ranges for better deliverability), and start looking at their list management practices instead.

This then led to even more granular analysis: subscriber engagement tracking. We now treat email delivery differently, depending on the engagement level of your subscribers. Which is nice, considering ISPs are also looking at engagement to decide whose emails show up in the inbox or not. As a sender, you can segment your campaigns based on subscriber engagement, or clean out the inactive members.

But it was when we came up with the idea for our freemium plan that we knew we needed a completely automated, intelligent abuse detection system in place. Without a scalable abuse prevention system, there’d be no (scalable) way to protect the deliverability of our servers from the abuse that comes with free. So we stepped up our research and created Omnivore.

What Omnivore Does

Omnivore is a program that runs in the background and analyzes email campaign and user account data. Non-stop.

When it finds anything suspicious about a MailChimp user or his campaigns, it’ll do one of two things:

1. Send the user a warning for something that looks problematic.
2. Suspend a user’s account for something bad, send them a warning, and alert our abuse team to investigate the account.

What Omnivore Doesn’t Do

Most important of all, Omnivore doesn’t replace or reduce our human abuse desk team. And despite what some angry people out there might think (or tweet), Omnivore doesn’t shut down “totally innocent, opt-in users” with “absolutely no warning.” Humans review reports from Omnivore. If an account’s been suspended or flagged by Omnivore for problems, our team investigates. So long as the user is not obviously an evil spammer, we attempt to contact the sender with some advice or instructions for account reinstatement. If you’re curious about how our abuse team makes its decisions, check out these compliance tips.

How Omnivore Works

Chad, our lead engineer, headed up the Omnivore Project. I’ve asked him to provide some technical insight into how it all works.

Ben: Without revealing too much of the secret sauce, how does Omnivore work? I heard the team discussing something about “genetic optimization?”

Chad: Yes, in a nutshell, genetic optimization is a method of determining the best option from a large set of possible choices.  When the universe of possibilities is large enough, it isn’t practical to just try all of them and pick the best – you have to use an optimization algorithm to narrow down on the best choices.  Genetic optimization uses a process that roughly mirrors how natural selection processes can incrementally produce the fittest candidate over many generations, hence the name.  You create a population of possible options, then breed and mutate the top performers until you get a good enough solution to stop. Assuming that choices that are similar to each other will perform similarly, this can get you to a good answer relatively efficiently.

Ben: So how’d you apply that to email marketing and spam?

Chad: We took every bad campaign that had ever been shut down by our human reviewers as well as every bad campaign that managed to get through, and started looking for common patterns.  We know a lot about every campaign that goes through our systems, as well as every list we manage and customer we sign up.  Our human experts had a laundry list of the traits that scream “bad campaign”, but for this thing to scale we needed to be absolutely, mathematically certain.  So we used a series of large scale genetic optimization tests running against every campaign we’ve ever sent to confirm which traits were predictive, and how predictive they were.

We did this for both negative reactions (bounces, unsubscribes, abuse complaints) and signs of engagement (opens, clicks) to give our team a complete picture of the likely results of any campaign, before the campaign is ever sent.  If Omnivore sees something that it’s certain will be bad, it alerts the abuse desk to review the campaign before it’s let through the system.

Ben: I hear you tried this on the machines at the office and they were too slow?

Chad: Right – even early small-scale tests would run for weeks before giving good results. The full tests would have taken years to complete. We ended up getting an nVidia Tesla and writing the process in highly-optimized C code, which was able to give us our preliminary results in a couple of hours. After we knew our algorithm was pretty close to what we wanted, we converted the process to a giant Hadoop Map/Reduce program running on a cluster of Amazon EC2 servers for about 20 days to get the final results for the first version.  Smaller optimization processes still run continuously to test new ideas and refine the model.

Ben: So this is totally different than just checking all outgoing campaigns with a spam filter?

Chad: Yes. It’s using the detailed sender information that we have as an ESP to look for that permission “gray area” mentioned above.

More importantly, we needed to be sure that Omnivore would continue to be efficient and predictive as our customer base grew and morphed after the free program was put into place.  Unlike static rules or blacklist-based methods of detecting spam, all of the major Omnivore systems are learning algorithms that keep up with changing user behavior without losing their predictive power.

Ben: After all is said and done, any fun or surprising observations to share?

Chad: Some traits and keywords that we thought we should focus on were actually poor predictors of bad behavior. For example, highly-targeted campaigns don’t do much better than other campaigns when it comes to abuse or unsubscribe rates.  Other things that you’d think are totally irrelevant at first glance turned out to be effective predictors, like the length of the subject line.

Ben: So a subject line that’s too short, or um — too long — would be a sign of trouble?

Chad: Something like that. Keep in mind it takes a combination of traits that add up in order for Omnivore to determine “this looks like lack of permission.”

Ben: Any other interesting observations?

Chad: When we started this process, we went straight to our team of human reviewers to show us the patterns that they were looking at when evaluating a new customer.  A lot of it was right on the money – particular industries definitely have a profile, and the language used when describing where permission came from is crucially important. However, some of the patterns turned out to be less predictive, like having a mailing address displayed prominently in the content and some of the other details of CAN-SPAM compliance.  It was also a bit surprising to discover exactly how bad most spam filters are at predicting permission issues.  Whether or not a campaign passes any of the free or commercial spam filters generally has little impact on its predicted outcomes.

Results So Far

As MailChimp scales and sends more campaigns, Omnivore will collect more data and adapt. It’s by no means complete. There are switches and knobs we haven’t even turned on yet. We’re currently running some of Omnivore’s scanning in “observation mode,” and not letting it act on anything. As it gets smarter, we’ll gradually activate more functionality and grant it more decision-making power.

But so far, here are some of the results:

• As of January 6, 2010, Omnivore has automatically sent 19,581 warnings to 9,349 users for exhibiting bad behavior. Of course, we also include tips and pointers on how they can change their ways.
• Omnivore has automatically suspended 2,249 users since September 1st 2009.
• 861 of those users ultimately had to be shut down. We hate losing customers (because we love money), but no customer is worth jeopardizing the deliverability and reputation of our entire system.

Looking ahead (literally)

The reason we built Omnivore was because we wanted to change the way we think about abuse. The project involved so much data crunching that it resulted in some interesting byproducts. Our subject line suggester is one example, as well as the engagement ranking and segmenting tools we mentioned earlier.

But Omnivore is learning more every day, and is actually getting good at predicting not just bad behavior, but good behavior too. Here’s a snapshot from our internal dashboard:

As you can see, Omnivore’s predicting open and click rates for this particular campaign, along with the “bad” stuff. As we feed it more data, the margin of error narrows, making it a powerful new feature we could be offering to our customers one day.

Omnivore’s predictive reporting is changing the way we deal with abuse, but might end up changing the way we think about email marketing in general.

of course, the comments on youtube are pretty mean.

Viral project with some complex results.

Now, they’re shifting their attention to measuring “engagement.” They defined engagement as opens, clicks, and having an email moved out of the spam folder. This is similar to Gmail’s approach to leaving images on if the recipient knows the sender.

How Does This Change Things?

Hmm. If ISPs are starting to look at how engaged your subscribers are, how could an email sender use this to their advantage (beyond simple list  segmentation)? Perhaps you could send email a little differently through your delivery servers, based on your subscribers’ engagement activity? For example, if you knew half the people on your list were active users, but the other half not so much, wouldn’t it be smart to deliver the campaign to the engaged people first, then the others last? It would really suck to only get a small portion of your list delivered before an ISP decided you have poor list management practices, and blocked the remainder of your message.

Yes, MailChimp does all that. Automatically, and behind the scenes. That’s the reason we launched the List Activity Score back in March. We rank every single user on your list by their engagement, then we prioritize email delivery through our network based on overall list activity score. One of the many ways our nerds in the lab keep striving to improve email marketing.

Learn more about our List Activity Score

Related:
How sending to old lists will kill your deliverability

To be honest, I’ve never seen that warning, and have no idea what exactly triggered it. As you can see, the email was also sent straight to gmail’s junk folder.

On the surface, nothing about the campaign looks bad. The general content of the campaign is fine. The sender is not in a risky business (it’s a church). Their email delivery infrastructure (ahem, mailchimp) is fine. So what gives?

We ran the campaign through our inbox inspector, and got the following “spamminess” score:

Notice it failed Barracuda, Cloudmark, and Postini. It also triggered one rule in Spam Assassin (which, btw, is used in some way, shape, or form by just about all the other spam filters) that got 2 whole points. By now, we should all know how spam filters generally work, and that you shouldn’t use “trigger words” like “FREE!” or “BUY NOW!!!” in your content. But even when you do, those words usually only get assigned a few fractions of a point. Go to this list of spam assassin rules and CTRL+F for the word “FREE!” to see what I mean.

But when you see something getting 2 whole Spam Assassin points like this, something’s very wrong.

The rule that was triggered? The message contained a URL listed in the URIBL Blacklist. Upon closer inspection, it turns out they were using a URL shortener (you know, something like tinyurl.com). I’m not going to name names, but this URL shortener wasn’t quite as well known as most of the others I’ve heard of. No idea if it has a bad reputation, but if it’s new on the scene, chances are high that it doesn’t have enough of a reputation.

In general, URL shorteners are great tools that serve a good purpose, but spammers have abused the heck out of them to disguise their (already blacklisted) links.

In response, some spam filters make a habit out of “clicking” all URLs in an email, just to follow redirects from URL shorteners, and analyze the landing page they’d take you to. Which, btw, can lead to some unintentional unsubscribes, but that’s another topic.

If this is all new and fascinating to you, check out this article from Laura Atkins at Word To The Wise: Failed Delivery of Permission Based Email. She covers a few other seemingly innocent but oft-abused URLs that can get your messages blocked.

But it’s not just URL shorteners at risk. Any domain name with a bad reputation can get blocked. For example, there’s this article from yours truly:

Is Your Domain Name Getting You Blocked?

Finally, if you’re a MailChimp customer be sure to check out our built-in, one-click email checker: Inbox Inspector. It can help you prevent renderability and deliverability problems before you send your campaigns.