How to Run AB Email Tests in Marketo

So you discovered “AB testing” and realized statistics was useful after all.  The good news is Marketo does all the hard calculations. The bad news is you have to tell it what to calculate!

Marketo enables you to compare versions of your emails to see which ones result in higher Opens, Clicks, or whatever else you want people to do when they read your email. The key metrics in most cases are Delivered, Opens, and Clicks and their corresponding percentages vs. Delivered. (This is how Marketo shows your data in the Email Performance Reports).

While Marketo enables you to do email AB tests, doing so is much different than landing page AB testing. In this section, we will walk through your options and show you how to run an actual email AB comparison.

What I talk about when I talk about AB testing

While I hear much talk about performing AB testing by marketers, not as many of us are doing these tests back at the office. Let’s change that right now, with this chapter. First, let’s review what AB testing is

1. The comparison of two or more emails to the same group of people, randomly selected;
2. The comparison of responses to one email by two or more groups of people;
3. The system designed to build, send, and report on 1 or 2.

To keep this guide simple, we will discuss comparing two different emails or two different groups’ responses. We will not explore multivariate testing although you can try to do so on your own.

Key Concepts:

• Test the Null Hypothesis (H₀) which is always that there is no difference between Open Rate or CTR for the Control and the Experiment.
• A: Control: this is the original email. You may already have data on it too.
• B: Experiment or Test: this is the new design with a single change to make it different than the Control.

To setup a test properly, you need to have two things to compare. Your comparisons must be consistent, with one change at a time to be a valid test. Here is a table you can use for ideas. Note that you can only compare one cell at a time here. That is, “Compare open rates of different subject lines at one time of day,” or “Compare open rates of one subject line at two different times of day.”

For testing ideas and visuals, take a look at Anne Holland’s famous WhichTestWon.com or the Testing with the Stars (2012) Marketo Summit presentation.

I finished the 43 page chapter on How to do AB Email Testing in Marketo the other day. Instead of showing you all the goodies, let me point you to some useful resources already out there. When the Marketing Rockstar’s Guide to Marketo launches, you will see several great examples of AB testing options fully built inside Marketo.

Until then, sign up for the Marketing Rockstar Guides email list for the latest updates.

AB Testing in Marketo Resources

• Using Random Sample [Nation]
• Using Random Sample example of splitting lists. [Nation]
• AB Email Testing Options in Marketo [Official Docs]
• Using Champion Challenger AB Email Testing (for Triggers or Engagements only) – [Official]
• Forward to a Friend Link {{system.forwardToFriendLink}}
• Calculator Spreadsheet for Statistical Significance [Avinash Kaushik]
• AB Testing FAQ –  includes a calculator! [HubSpot]
• HubSpot’s AB Testing Intro [HubSpot]
• Increase Conversions Cheat Sheet [Mixergy]
• Whichtestwon.com –
  Case Studies – Tools Guide [WhichTestWon]
• Sirius Decisions’ Email Quality and AB Testing – about the number of inquiries related to target audience.

[Note: links updated on July 29, 2017]