Do you ever have a problem deciding between a few different subject lines? Or maybe you were sure that one was better, but a co-worker disagreed? Well, maybe you’re both wrong. With subject line A/B splits, you can take the guess work out of fine tuning your messages. This Brontoversity Blog topic will walk you through the basics of setting up a subject line A/B split test and help you understand how this powerful feature can help improve your email marketing campaign.

In its most simple form, a subject line A/B split test starts with two groups chosen randomly from all of your targeted contacts for a delivery. You then send a message to those groups with a variable subject line in each. After all the responses have come in, review the results and identify which version performed better. You can then use that information to drive future marketing. A/B split tests within the application allow you to automate the tedious steps of testing different message types and provides access to valuable testing data with a minimal investment of your time.

There are two main types of subject line A/B split tests that you can use:

A/B-Winner

A/B-Winner split tests involve testing messages with two or more groups to see which is most effective. After a set period of time, a winner is chosen based off of preselected criteria, such as highest open rate or highest number of clicks. After the winner is selected, the winning message is sent to the remainder of your targeted contacts.

Champion/Challenger

Champion/Challenger is quite similar to A/B-Winner, the difference being that you don’t have a winner group. In this type of test you have an established way of doing something, called the Champion, and you want to test if it is still the best by introducing a Challenger. When creating a Champion/Challenger split test, it is common to send most of your contacts the champion, while setting aside a smaller portion for the challenger. Once the test is run, there is no winner group who receives a message, but you will have a better idea about whether or not your champion is as good as you thought it was.

The Setup

1. Click the Send A/B Split button on the Message Overview page for particular message in Bronto.

2. Choose Subject Line from the Split Content pull-down menu in the Perform Test On section.

3.  Choose whether you want to perform an A/B – Winner or Champion Challenger split test.

4.  Next you need to specify your groups.  Remember that the groups are randomly chosen from your targeted group of contacts, which you’ll be selecting later. What you are setting here is just the size of those groups, which is done in percentages of the whole group. For A/B-Winner, it’s a good idea to try and keep the total for your test groups to under 50% of your list. If you are just using two groups, then 20% and 20% is good; you should reduce the size of the groups as you add more. As you create groups and specify their sizes, you will see that the Winner group is automatically updated to contain the remainder of the percentage.

For Champion/Challenger there is no winner group, so you need to allocate a full 100% amongst the groups you create. You can create a maximum of 5 test groups.

5.  On the next page you will assign the content for each group.

6.  After you enter the content for each test group or select an alternate message version, you will be brought into the familiar delivery scheduling page. In delivery scheduling, you will need to specify the Sender Options, select the lists and segments that you would like to have receive this message, and pick a time for the test groups to be sent out. For an A/B-Winner split test you will also need to pick a time for the winner group to be sent. You should give enough time for your message to get out and for people to open, click, and even convert. Remember that if you throttle the message, it may cause it to take more time for the message to go out. Even with a throttle of 2-4 hours, sending the winner 4-6 hours later should be fine.

7.  Once you have selected the send times, you will be brought to a verification screen where you should double check all of your selections. Click the Schedule Delivery button and your A/B split test is ready to go out.

Please feel free to share any experiences with Subject Line A/B split tests and what results you noticed in your campaigns.

Rob Slade
Client Support Specialist at Bronto

Related posts:

1. Challenging the Status Quo: Champion-Challenger A/B Splits We’ve talked about A/B splits a number of times –...
2. Adding Dynamic Subject Lines For quite some time, Bronto users have had the...
3. Dynamic Content In Text Version Subject Lines Although you can’t directly add Dynamic Content to the subject...

So What Are A/B Content Splits?

With A/B Content Splits, you can quickly compare multiple versions of a message’s content against each other in a way that lets your contacts choose the best performing option.

Setting Up A/B Content Splits

1. Compose two or more version of the same message and make minor changes to the areas of the message content you want to test.
2. Select the message you want to perform the A/B Content Split on.
3. Click Send A/B Split from the Message Overview Page.
4. Choose Full Message Content from the Split Content drop-down box.
5. Choose the type of split you want to perform and then designate split groups.
   • A/B Winner – A/B-Winner splits involve testing messages with two or more groups to see which is most effective. After a set period of time, a winner is chosen based off of preselected criteria, such as highest open rate or highest number of clicks. After the winner is selected, the winning message is sent to the remainder of your targeted contacts.

   

   • Champion/Challenger – Champion/Challenger is quite similar to A/B-Winner, the difference being that you don’t have a winner group. In this type of test you have an established way of doing something, called the Champion, and you want to test if it is still the best by introducing a Challenger. An example of this is testing a new template design or header image compared to your current standard. In creating a Champion/Challenger split, it is common to send most of your contacts the champion while setting aside a smaller portion for the challenger. Once the test is run, there is no winner group who receives a message, but you do have a good idea about whether or not your champion is as good as you thought it was.
6. Click Next.
7. The message you choose in step 2 will automatically be used for split group A. You will need to choose the alternate versions of the message you created using the message selector for the other split groups.
8. From here, you need to specify the sender options, verify the selections you have made, and then schedule the actual delivery.
9. Later on, you can view a report for that delivery that will show you which version of the message garnered the most favorable reception.

Important Things To Consider

With A/B Content Splits, remember not to get pulled into the trap of using two drastically different versions of a message. Rather, use this type of test to examine how simple design tweaks, such as order and positioning of links and images, affects the amount of clicks and conversions your message receives. You may be surprised to see how simple changes, like moving your call to action to the area of the message above the fold, will help drive clicks and conversions in your message.

Have you successfully implemented the A/B content split feature in Bronto? Please share your comments with us below.

John Gunther
Technical Writer/eLearning Specialist at Bronto

Related posts:

1. Challenging the Status Quo: Champion-Challenger A/B Splits We’ve talked about A/B splits a number of times –...
2. Can’t Decide On A Subject Line? Split ‘Em! Do you ever have a problem deciding between a...
3. Testing, Testing 1, 2, 3: Sending Test Messages As a Bronto user, you have probably been sending test...

Ever have an argument with a co-worker about which subject line would be better? I know we have them here at Bronto. Heck, we’ve even had company-wide contests to drum up fresh ideas for subject lines – anything that might take it to the next level.

When you have that many ideas about how to hook people into reading your message it can be hard to know which to use. Even when everyone agrees on which to use, it’s still just a guess. This has been brought to light in a recent post by DJ about first name personalization. Some of our clients have taken DJ to task and are doing the tests themselves (see the comments on the above linked post).

The new A/B split feature that we introduced in the Bronto Fall 2008 Release has come to the rescue. This is just what A/B splits are all about, testing and learning something new. One of the best part about A/B splits is that they are so easy to do.

Almost any time you send a message you should be doing an A/B test. If you always assume that you are doing what’s best for your email marketing program then you aren’t getting any better. Trust me, there’s always room to improve. In Bronto, we make it very easy to send an A/B Split. It is right there (look!) waiting for you to click it.

A/B splits don’t just come in one flavor. You can run A/B/Winner or Champion/Challenger tests.

A/B/Winner

A/B/Winner tests send out emails to two or more test groups with the best performing option being sent out the remainder of your list at some designated point in the future.

Champion/Challenger

Champion/Challenger tests just split up all of your list amongst two groups. The Champion/Challenger tests are great for trying something new with a small percentage of your group.

At Bronto we make this quick and easy, with powerful reports that allow you to see where each version excelled. But if you don’t have that built into your email marketing tool, that doesn’t mean you can’t run A/B splits. Just do two different deliveries and print out the results for each.

A/B splits can be really fun. Place a friendly bet, or maybe just your pride, on your subject line. But don’t bet too much, you may be shocked by the results. Just ask DJ!

Adam Covati
Product Manager at Bronto

Related posts:

1. Challenging the Status Quo: Champion-Challenger A/B Splits We’ve talked about A/B splits a number of times –...
2. Can’t Decide On A Subject Line? Split ‘Em! Do you ever have a problem deciding between a...

When you have that many ideas about how to hook people into reading your message it can be hard to know which to use. Even when everyone agrees on which to use, it’s still just a guess. This has been brought to light in a recent post by DJ about first name personalization. Some of our clients have taken DJ to task and are doing the tests themselves (see the comments on the above linked post).

The new A/B split feature that we introduced in the Bronto Fall 2008 Release has come to the rescue. This is just what A/B splits are all about, testing and learning something new. One of the best part about A/B splits is that they are so easy to do.

Almost any time you send a message you should be doing an A/B test. If you always assume that you are doing what’s best for your email marketing program then you aren’t getting any better. Trust me, there’s always room to improve. In Bronto, we make it very easy to send an A/B Split. It is right there (look!) waiting for you to click it.

A/B splits don’t just come in one flavor. You can run A/B/Winner or Champion/Challenger tests.

A/B/Winner

A/B/Winner tests send out emails to two or more test groups with the best performing option being sent out the remainder of your list at some designated point in the future.

Champion/Challenger

Champion/Challenger tests just split up all of your list amongst two groups. The Champion/Challenger tests are great for trying something new with a small percentage of your group.

At Bronto we make this quick and easy, with powerful reports that allow you to see where each version excelled. But if you don’t have that built into your email marketing tool, that doesn’t mean you can’t run A/B splits. Just do two different deliveries and print out the results for each.

A/B splits can be really fun. Place a friendly bet, or maybe just your pride, on your subject line. But don’t bet too much, you may be shocked by the results. Just ask DJ!

Adam Covati
Product Manager at Bronto

Related posts:

1. Challenging the Status Quo: Champion-Challenger A/B Splits We’ve talked about A/B splits a number of times –...
2. Can’t Decide On A Subject Line? Split ‘Em! Do you ever have a problem deciding between a...