How to Effectively A/B Test Emails in Kentico

A/B testing is a tactic that compares how two or more variants of a given feature affect outcomes within a test group. Based on the results of an A/B test, the more effective variant is determined and used from that point. This technique helps marketers increase their conversions in less time than if they didn’t use A/B tests. Though this post is specifically about effectively A/B testing emails in Kentico, it is worth noting that A/B tests can also be performed against pages on your website. Both of these features are available in Kentico with the Enterprise Marketing Solution.
 

What Tests to Perform and Why

There are four main aspects of emails that you can test in Kentico:

1. The subject line
2. The sender (name and email address)
3. The template (design)
4. The content

While it is possible to test differences in all four areas within a single test variant, the best practice is to focus your tests on only one of these at a time. By focusing on a single item during the test, you can clearly analyze and trust the results of the test since all other objects are equal. Deciding which area you are going to alter in your test should depend on the metric you are looking to improve.
 
Variations in the subject line or the email sender will have an impact on the open rate of your emails. Users typically scan these two items very fast and make a quick decision on whether or not to open the email. If the subject line is not enticing or the sender is not someone they trust, they will most likely ignore or delete the email, which is obviously not good for you.
 
Variations in the design of the email or the content within the email will impact the click-thru rate. In this case, you may test the layout and aesthetics of the email, or you might test using different wording, different calls-to-action or different images. The goal of these tests is to find the variation that engages your subscribers the most and entices them to click thru to your website. These results will have a direct impact on the success of your email marketing campaign.
 

Creating the Test Variants

When creating your email marketing campaign issue, you will see a button at the top that says Create A/B Test.  As you would expect, clicking this button is how you create a new test variant. You will be asked to give the variant a name so you can decipher it from the other variants and you will also be asked whether or not you want to create an empty variant or copy the content from one of the other variants. In most cases, your test variants will not be significantly different from each other so I find it easier to copy the content from another variant and just make minor changes.
Once the new variant is created, you will notice a new section above the buttons that contains the relevant controls for the test. There is a slider that is used to switch between the different variants as well as 3 buttons that allow you to either add a new variant (+), delete the current variant (trash can) or edit the variant name (pencil).

You must make your edits for this variant based on the strategy you are going to test. Any changes you make to the selected variant will only apply to that variant. If you decide to test the subject line; simply update the value in the Subject text box. If you want to test the sender, click the Show Advanced Options and UTM Tracking Settings link to view the additional fields. Then you can edit the sender name and/or the sender email address.

If you decided to test variants in the content, you can update the content in the main window as necessary. Testing variations in design requires a few additional steps. The design is controlled by the email template, so you first need to create a new template for the design in the Template Management section of the email marketing application. Remember to assign this new template to your campaign. Once have created the new template, select it from the Template field drop-down. With any of these updates to your variants, you must click the Save button to commit them. 
 

Sending Your Campaign

Now that the variants are completed, you are ready to launch the campaign which begins with the test. To do this, click on the Send tab.
 
There are three main configuration settings you need to set before you click the Send button. The first is the size of the test group.  The best practice is to send your tests to about 20% of your total subscribers. This 20% will then be divided by how many test variants you have. For example, if you only have 2 variants, 10% will receive variant A and the other 10% will receive variant B.

Next, you will schedule when to start the tests. You can set each variant to send at a different time or to send them both at the same time. My recommendation is to send all variants at the same time because otherwise the send time becomes a variant in itself and can skew the results.
Last you will set how to determine the winning variant. This setting should be established based on the metric you are trying to increase: Opens or clicks. You can optimally set it to manual, but that will require someone to monitor the results and select the winner themselves. In addition to determining which condition picks the winner, you also need to set how long to wait before determining the result.

All Done!

Once you have everything configured correctly, click the Send button and launch the campaign!

That's it! Now you know how to effectively A/B test emails in Kentico. I hope you found this helpful and will use this knowledge to improve your email marketing campaigns.