In general, we don’t recommend A/B testing as a primary metric for evaluating the performance of send time optimization.
Because send time optimization usually increases email health overall, the benefits of using this approach compound over time. Split tests take none of the long term benefits of using send time optimization into account. At best, a split test will only tell you how much lift STO gave to a specific email.
That said, many users will still choose, or are required to “A/B” test send time optimization. This guide will help you to avoid some of the most common pitfalls and get the most accurate results possible.
Allow the system time to learn before evaluating
Seventh Sense uses machine learning to build profiles on your subscriber’s engagement. To give the system a head start, the system builds initial profiles based on your historic engagement data from HubSpot or Marketo.
These initial profiles are biased towards times that you’ve historically sent your emails and you won’t see a full degree of optimization until the system has had more time to learn.
For example, if you’ve previously sent all of your emails at 9am on Thursdays, 9am and the hours after it on Thursday will appear to be better times than they may be in reality.
Fortunately, Seventh Sense is designed in such a way that this bias works itself out relatively quickly. But it can take a few months of consistent mailing with STO before the system is fully optimized.
For this reason, we generally recommend you wait at least 90 days before split testing Seventh Sense.
Be aware split testing can reduce the benefits of throttling
Hardware spam filters, such as Barracuda, often enforce rate limiting. This means that when a certain amount of messages hit the server at the same time, the spam filter may kick in.
By using STO, your messages are less likely to trigger these filters because they’re not sent all at once. If during an A/B test, the blast cohort is sent first and rate limited, the STO messages sent afterwards will likely be blocked as well.
For this reason, we recommend that when split testing STO, you always send the blast last.
Take into account that the blast cohort can negatively affect STO’s performance
When you split a list in two, and send half the mailing with STO and half as a blast, email service providers (ESP’s) don’t see it as two separate messages. They see it as one message being sent over time.
This can introduce a number of biases in the test which almost universally benefit the blast cohort and lower the performance of the STO cohort.
Why would the blast benefit and the STO cohort be hurt?
Think of it this way: remember when you worked on group projects in school? There was always that one kid who didn’t do any work, but managed to pass based on the other kids’ hard work? That’s the blast.
Email service providers view email messages holistically. Rather than evaluating the STO cohort and the blast separately, to the ESP, they are working on a group project.
This means any increase in engagement brought by STO, will increase the chances that the blast cohort makes it to the inbox.
What if I put the blast first? Will that fix the problem?
Not really. As we discussed earlier, putting the blast first will eliminate the throttling benefit of STO.
In addition, in the same way that STO will lift the open rates on the blast cohort, the blast cohort can lower the performance of the STO cohort.
When deciding how to folder your email, ESP’s sometimes hold back part of the messages to see how they perform. If they get a high open and low complaint rate, they will release more messages to the inbox. If the performance is lower, more messages will go to the promotions/clutter or spam folders.
Putting the blast first is like sending the kid who hasn’t studied up to do your group presentation. It will be obvious they didn’t study, and their poor performance will drag down your grade as well.
To reiterate, when split testing STO, you should always send the STO cohort first, and the blast after all the STO mailings have been sent.
There will likely still be some positive “priming” effect from the STO cohort that will affect the blast cohort, but overall your results will be closer to accurate.
Be careful when testing over multiple days, or overlapping other campaigns
If you’re used to all your emails going out at once (blasting), it’s easy to end up scheduling overlapping emails when using STO.
Currently Seventh Sense doesn’t automatically take into account what other campaigns may be running in your HubSpot or Marketo instance. If you schedule an A/B test, and the STO cohort’s emails are sent around the same time that another email blast goes out, your subscribers’ attention will be reduced.
For this reason, we generally don’t recommend scheduling one off campaigns to contacts who are currently in nurture workflows.
Make sure your segments are truly random when A/B testing
This one should be obvious, but make sure your list is randomly split. Because HubSpot / Marketo doesn’t have a native way to split lists, users are often left exporting spreadsheets and splitting outside of HubSpot / Marketo and then importing this back which alone can increase the chances of user error.
If a list is split in a way that is not truly random, this will introduce bias into the test. We’ve seen cases where an operator has “split” a list, but one cohort is clearly made up of older contacts than the other.
The easiest way to do a random split in HubSpot is by using the Seventh Sense Split test automation action in a workflow. For Marketo users, you can follow these instructions on how to split a list using Seventh Sense.
Wait at least 48 hours after the last mail has been sent before analyzing your results
Out of eagerness, or anxiousness, many users start tabulating their results as soon as the email has completed sending. Because not all of your emails went out at once, this will give a major advantage to whichever segment was sent first.
We recommend you wait at least 48 hours, if not longer, before analyzing the results of any email A/B test.
Take into account Holidays, Weekends when scheduling mailings
Seventh Sense will pick the best time within the delivery window that you give the system. However, if you send the STO cohort from Sunday morning to Monday morning, and your blast cohort on Monday, Seventh Sense will be put at an obvious disadvantage especially for B2B audiences.
If contacts have shown a stronger pattern of afternoon engagement than morning engagement, the system will schedule them on Sunday afternoon rather than Monday morning. After all, that is the only window you gave it to choose from!
While Sunday’s can be a high performing day for some subscribers, in general weekends will see lower performance than weekdays as a whole.
The same effect can be seen if the STO cohort is scheduled on a Friday or over a Holiday.
For this reason, unless you are sure weekends are a higher performing time for your audience, we generally advise only sending on weekends by using a 7-day STO option. This will allow Seventh Sense to choose which contacts should receive weekend mailings, and which shouldn’t based on their engagement history.
Recognize the limits of open rate/click rates as performance metrics
Seventh Sense generally increases open and click rates, but ultimately opens and clicks are not why most of us are sending email. We send email because we want people to engage with our content and business.
Many longtime Seventh Sense users report higher engagement among the key groups that they find the hardest to reach. This makes sense if you consider that decision makers are often the busiest members of your audience.
When split testing, look beyond open rate. Many users see increases in conversion, read times, and time spent on site from email generated traffic.
Conclusion
Like many other elements in a marketing stack, it’s often hard to isolate the effect Seventh Sense is from all the other parts of your email program. Whether or not you choose to split test, we recommend you look at your email program holistically.
In order to get the most from Seventh Sense, you’ll want to make sure your lead generation strategies are sound, your audience is segmented when needed, and your messaging relevant to your audience.
For a more in depth guide on how to improve your email program, check out the Seventh Sense Guide to Email Optimization.
