The Beginner’s Guide to A/B Testing

How many compliments did you get on your shirt today?  How many compliments did you get on your shirt yesterday?  What was different about the two shirts in question?  Welcome to the world of A/B testing. It’s simpler than you may expect and an incredibly powerful tool to improve your store’s performance.

Of course, the aforementioned analogy is an exaggerated example, but in it’s simplest form, A/B testing compares the engagement from two different variables when all else is equal.  Think of the comparison less as apples to oranges and more as Granny Smith to Macintosh.  

A/B testing is everything in marketing and advertising as it allows us to test the effectiveness of our campaigns on consumers at the micro level, tweak based on the results, and optimize for future performance.  Google was at the forefront of this process, which started about 12 years ago and used it to determine how many search results they should lay out on one page.  To give you an idea of how much they utilize A/B testing, in 2011, they ran over 7,000 tests.    

The beauty of A/B testing is we can use it for almost anything.  Take, for example, landing pages.  About 48% of marketers do not test varying landing pages, which means, their ability to optimize conversions on that webpage goes out the window.  So, whether it’s testing your landing page, banner ads, email subject line, button color or yes, your t-shirt, the technique works incredibly well for driving optimal engagement.  The only things required are time and an audience.  

Here are a few A/B testing best practices that will help improve your marketing performance  

1.  Start Small

If your focus is ad creative, start as simple as possible.  Begin with the color scheme.  Test a palette of colors or single background and then a completely different color set on your next batch of ads.  Launch each campaign at the same time, with the same number of ad units, in the same sizes, with the exact same targeting and flight dates. By keeping all variables except one equal you will be able to isolate what is really driving performance. As a general practice, don’t try and test multiple variables at the same time as it will be challenging to determine which variable was the actual driver of change.

2.  Choose what Key Performance Indicator (KPI) matters most

It’s critical to know what KPI you want to measure with your test. It’s best to have one main KPI that you are looking to improve then working towards it with methodical testing. In the example of ad creative, if your main KPI is your Click-Through-Rate (CTR), run your test with the goal of improving this one metric. Test different background colors or Call-to-Action language (eg “Shop Now” vs “Shop the Sale”) to see what moves the needle most.

3.  Test then optimize, then test again

Once you complete your first A/B test you’ll have a baseline for how the two variables performed. So, for example, if you ran a test to see if a blue or red background increased engagement on ads and blue was the clear winner, your next step will be to iterate on your blue ad. Maybe on the next test you can try sky blue vs cobalt blue. From here, you can drill down even further. Once you have a clear winner on background color, use that winning color scheme and perform an A/B test on the call-to-action (CTA).  When you have your CTA winner, perform an A/B test on your headline copy, logo color, etc.  Small incremental changes can have a dramatic impact on your business and your bottom line.

4.  Time is money

There are countless permutations you can choose from and plenty of things to A/B test, so be careful not to get carried away.  Once you’ve found something that works and drives the performance you desire, stick with that method for the time being and work on managing your new traffic.  After you’ve developed strategies around managing the influx in traffic, work on scaling up again using A/B testing as a tool.

In advertising, it’s easy to see how this trend started with the digital age and how it will keep expanding.  Now that ad space is cheap, and emails are free, what was once unfeasible financially for smaller companies is now attainable. Thank you technology.

This practice even extends outside of the eCommerce industry to politics.  During his 2008 campaign, President Obama tested call-to-action buttons on his display advertisements.  What resulted was a 40.6% increase in sign-ups, $60million in funding, and 288,000 additional volunteers all from the “Learn More” call-to-action his campaign adopted.  

By using the outlined A/B testing best practices, you’ll be on your way to marketing and advertising much more intelligently.