Load Balancing in Marketo and Marketing Automation

As Marketing Automation Platforms mature, so do their customers become more skilled in the use of the platforms. And that means some customers push the limits of what the system is designed to do. Whether it is running dozens of data correction flows, scoring, and lead lifecycle or accumulating workflow detritus, an older system is often a slower system.

To keep your leads flowing fast, there are some tricks and best practices for running a system efficiently and effectively. These tips are helpful for new system builds (do it right the first time) as well as rebuilds or cleanups. These tips are considerations and each situation may require different designs.

Why do systems like Marketo slow down?

Like any computer program, MAPs are limited by hardware as well as software. Marketo is designed to handle marketing focused activities like scores, syncs, and email sends. Marketing tech people like me have taken advantage of other functions like data normalization, personalization, and lead lifecycles to perform tasks that the system can do, but maybe wasn’t expecting to be used heavily.

There are three key components that can slow down a system:

• Number of trigger campaigns
• Volume of Leads
• Complexity of smart lists

There is not a hard and fast rule for the number of campaigns or leads that a MAP can handle well. Based on my experience, slow downs can occur with as few as 150 triggers and 100,000 leads. With regular cleaning and good systems design, it is fairly easy to keep your system running fast.

Increasing speed, reducing load

Trigger campaigns are always on, always listening. That means for every lead that changes, the system must check if it meets those conditions. If you import a lot of leads, or change a lot at once, the system will have to check all of those at once. According to Marketo, each lead is set serially into the flow, further slowing down the system.

Reduce the number of Triggers!

• Convert triggers to batches.

Does every score have to happen immediately? Probably not. Behavior based triggers are prone to frequent hits and complexity, so move these to a nightly batch. Data normalization can also help. Batches also run all the flow steps for every lead at once, instead of serially, which reduces total processing time.

[Update: May 5, 2016]: I recently learned something I can now share, it’s a special, insider secret: The real reason triggers cause problems is there are two queues. Whenever a change to a lead happens, it hits every potential Trigger in the backend – the Trigger Evaluation Queue. You cannot see this and Marketo Support won’t show most of you what is going on there. The key thing to know is that you should multiply the Change x # of Leads.

100,000 Leads Imported x 20 Lead is Created Triggers = 2,000,000 evaluations

Yes. It gets big, fast. The fewer triggers you hit, the faster Marketo can send the lead to the correct Trigger that’s visible in the Campaign Queue. Some big offenders could be your SalesOps team or Product team making batch changes on their end, only to let 1,000,000 records sync to Marketo to hit that Evaluation Queue. This Queue will slow down your processing and routing inside Marketo because it takes longer for the lead to hit the correct set of Triggers and then be processed as you expect.

• Reduce the Qualifications Rules from Every Time to something else.
• Reduce Triggers like Change Data Value and Lead is Created to single points of entry or batches.
• Reduce the number of leads that can flow through with filters
• Leverage the auto-deactivation after 6 months.
• Clean up quarterly by using the Campaign Inspector.
• Compress flows by adding flow steps to one campaign based on the same trigger. Eg: Interesting Moments and Scoring can often be on the same flow.

Smart Lists Should be Less Complex on Triggers

Depending on the situation, you may want to reduce the complexity on batches and triggers. Instead of Marketo looking for the list and running it, just put the filters in the trigger itself. Or reduce the number of nested smart lists called in a smart list. The more complex, the harder it is for the system to figure it out, which increases backend processing (which you do not see on the Campaign Queue) and even creates campaign failures from timeouts.

Change the Time of Day of Batches

While this may vary with your business and location, batch data flows should occur at night, or the time of day where leads aren’t on your site frequently. This helps give priority to email sends that may start around dawn or through the day, as well as triggers that route leads to sales on weekdays.

Use Wait Steps in Processing

There is a much longer discussion on this regarding Lead Lifecycles and Ed Unthank’s lead processing articles. Many times, some campaigns are subject to a race condition between themselves or the CRM. To mitigate this, where the lead should have been updated, but wasn’t in time for the next campaign to process properly, you can do three things:

• Add Wait Steps at the start of the flow to let the other flows catch up. Sometimes 5 to 30 minutes are needed.
• Add Wait Step after a Sync to CRM step to let the CRM assign a Lead Owner. May require 10 – 30 minutes if the system is complex.
• Request Campaigns to control the order of operations

Marketo Campaign Queue and Priority of Campaign Type

Here’s Marketo’s secret campaign priority queue order.

According to presentations shared by Scott Nash, Marketo’s VP of Product Platform, you will also see any batch or trigger fall in the priority order if the Wait Step is activated and the Wait Step is 5 minutes or more. There’s a new one for 2018 if you’ll be at the Summit.

• MUG Slides with information (2016)
• Campaign Sync Queue Information
• More Marketo Docs (2016)
• Campaign Queue Icons 
• Order of Operations Advice (Justin Norris)

Is a Slowdown the Vendor’s Fault?

Yes, and no. The vendor often designs their infrastructure for a shared load, with instances running on servers, pods, or unique servers at data centers globally. Depending on your contract, the software is often designed for SMB variances. Since each organization has different needs and different growth rates, some firms may outgrow the expected load bands.

Some customers also may underestimate their needs and buy a system that underperforms because it wasn’t the right match. Thus, it is incumbent upon you, as the customer, to work with your vendor to determine what they can do for you, and what you can do to improve efficiency. And be honest–your needs may outgrow a vendor that only is ready for SMB, not enterprise scale systems. Honesty also works the other way — you may be asking more of the vendor or doing inappropriate things with the tool.

There are a few other tricks I’ve learned over the years as well. And maybe I’ll share those soon. With the tips in this article, you should be able to keep your marketing automation platform running smoothly.

[Updated: March 23, 2018 with new links and details]