Why predictive personalization is replacing A/B testing

Website Personalization

Personalization correctly applied always outperforms “one size fits all” approaches.

Performance marketers today face a conundrum. On the one hand, you’re tasked with creating marketing campaigns that appeal to wide—and sometimes very different—groups of prospects. On the other, those campaigns and efforts must deliver results. Why is it a conundrum? Because a single strategy, no matter how refined and researched, cannot optimally appeal to all of your visitors.

Performance marketers have turned to A/B testing to identify the single best page for all visitors. While A/B testing has helped marketers better quantify the impact of their ideas, trying to find the single “one size fits all” best page to show all of your visitors leaves money on the table and wastes time and effort.

Let’s explore an example to illustrate this. Say you are responsible for optimizing the conversion rate of a website and the current messaging that performs best (which we’ll call the baseline) converts 3% of your visitors. You decide you want to see if different messages can perform better, so you create two new test variations. In this example, you have three different, equally sized audience segments visiting your site: existing customers, prospective customers and competitors’ customers.

Conversion Rates for 3 Different Variations to 3 Audience Segments (with equal traffic) 

Variations
Baseline 1 2
Audience Segments A: Existing Customers 3% 2% 1%
B: Prospective Customers 2% 4% 1%
C: Competitors’ Customers 1% 3% 5%
Average conversion rate 2.0% 3.0% 2.3%

Pick one to show to all: 3% conversion rate overall

Personalize to each segment: 4% conversion rate overall (33% lift)

If you could only show a single variation to every site visitor, you would select Variation 1 because it delivers the best overall conversion rate across all customers (3%). However, if you could personalize the experience for each group and show each group the variation that performs best for them, you would show Baseline to existing customers (3%), Variation 1 to prospective customers (4%) and Variation 2 to competitors’ customers (5%). This personalized approach would result in a 4% conversion rate from the same ideas and the same group of visitors — a 33% improvement in performance.

No matter how individual variations perform personalization correctly applied always outperforms “one size fits all” approaches.

While rules based personalization is better than a “one size fits all” approach, you are required to set up static rules to deliver specific experiences to predefined segments of your audience. Predictive personalization uses machine learning to present the best experiences to each individual visitor to your site. Additionally, both A/B testing and rules based personalization optimize for a point in time, irrespective of how your visitors’ behavior changes or how your marketing efforts change in the future. Predictive personalization automatically adjusts to changes in visitor behavior over time, shifting traffic to your best performing experiences.

There are three advantages predictive personalization offers marketers over A/B testing:

Faster Results: Predictive personalization begins optimizing without waiting weeks or months for statistical significance, unlike A/B testing. Predictive personalization also allows you to test more experiences and variations at the same time than A/B testing so you can see results across more ideas sooner.

Better Results: Personalization outperforms “one size fits all” approaches by serving the best performing experiences to each user. Additionally, as visitor behavior changes over time, predictive personalization adjusts accordingly to deliver the best performing experience. A/B testing, on the other hand, picks a winner once and does not adjust again.

Less Work: A/B testing requires your ongoing attention, monitoring experiments, deciding when to call winners, and managing potentially large matrices of separate test cells. Predictive personalization automates experiment management and execution, freeing marketers to spend more of their time understanding prospects and developing new ideas to drive conversion.

In the year since we’ve launched our predictive personalization system, we’ve seen that these benefits are real. Perkville, a referral and rewards program increased customer referrals by 70% in its first few months. Chime, a bank focused on helping consumers lead healthier financial lives, saw a 79% lift in new account signups four weeks after introducing their second round of ideas.

Any performance marketer who has a frustratingly long backlog of ideas to test and wants to see results quickly should be investigating predictive personalization. A/B Testing, gave us a way to understand the best single experience for all visitors at one moment in time, and predictive personalization now gives us a way to deliver the best experiences for each and every visitor as visitor behavior changes.

How predictive personalization is changing marketing

The average campaign running on Intellimize optimizes across 94 million possible versions of the page. Our clients are effectively personalizing down to the individual level.

In the year since we launched Intellimize, we’ve had the privilege of working with innovative marketers like Chime, Stella & Dot, LendingTree, Segment and Perkville. They’ve become active users of our predictive personalization system that tailors their web sites for each visitor automatically using machine learning. We’ve seen how they’ve put predictive personalization to use day-to-day, and we want to share some of what we’ve learned.

One thing that surprised us was how quickly marketers embraced the differences between predictive personalization and A/B testing approaches. There was pent-up demand within our clients to try many new ideas in order to drive more conversions. Our customers were able to meet that demand testing, on average, 44 different experiences containing 127 variations across 5 different pages on their mobile and desktop websites. Because predictive personalization allows marketers to run many individual tests simultaneously, the average campaign running on Intellimize optimizes across 94 million possible versions of the page. This means our clients are effectively personalizing down to the individual level.

What does this mean for marketing going forward?

Marketers are doing more in less time

The ability to test and learn faster reduces both cost and risk, allowing marketers to experiment with more ideas and invest more effort in developing the ideas that prove effective. Had our customers used A/B testing, it would have taken, on average, 25 years for them to do what they’ve done with us to date.

One of our clients, Chime, an innovative banking app, uses Intellimize to increase new account signups from their homepage. In the first six weeks of using Intellimize, the digital marketing team at Chime tested nine different headlines, one new video, and three variations of body content. The result? 8% lift.

After analyzing the initial results, the team added new photography, headlines, and body content to their campaign. Over the next four weeks, they saw a 79% lift in new account signups. Overall, they tested 21 different ideas, which enabled 216 versions of their page, in ten weeks. Testing these ideas with A/B testing would have taken Chime more than nine years.

Thought leading marketers are using this increase in speed to test more ideas and deliver more insights, which together then deliver better results.

Tools are freeing marketers up to focus on prospects and how to drive them to convert

Predictive personalization automates most rote experiment management tasks, giving marketers more time to focus on understanding their prospects and creating personalized experiences that drive customers to take action.

Perkville, a referral and rewards program that helps companies drive revenue and loyalty, uses Intellimize to increase customer referrals for its clients’ businesses. The marketing team personalized calls to action, social elements, and the layout of their referral pages. They did no manual management of variations, test matrices, or audience cells, and were able to test 62 ideas that resulted in hundreds of thousands of possible versions of their referral pages. The result? 70% increase in referrals.

We’ve seen how the combination of faster testing, learning, and iterating with less rote work helps marketers drive impressive results. It’s an exciting time to be a marketer as machine learning tools like Intellimize are enabling us to focus more on understanding and delighting prospective customers. It’s been an amazing first year, and we’re looking forward to seeing the amazing and creative ideas our customers come up with in year two!