Blast Analytics and Marketing

Analytics Blog

Supporting Leaders to EVOLVE
Category: Digital Experience

Are You Using the Right Ecommerce A/B Testing KPI?

May 18, 2018

image of woman looking at a performance reviewFor the past several years, ecommerce and other organizations have invested more resources into running A/B tests on their websites. A/B testing platforms, like Optimizely, have made it easier for digital marketers to try their hand at running A/B tests, and these platforms allow teams to conveniently track transaction and revenue as goals. If this is the case for your organization, it’s important to ensure that you’re making the best decision based off your test results.

Below we’ll identify 3 KPI’s that are commonly used in ecommerce A/B testing (i.e. conversion optimization) and discuss why Revenue Per Visitor (RPV) is the best KPI to rely on for ecommerce testing, and more importantly, for making business decisions.

A/B Testing KPI: Transaction Rate

icon representing transaction rate

For ecommerce and other companies where transactions are involved, it makes sense to track transaction rate when conducting A/B tests on your website. Calculating transaction rate is simple:

Total Transactions / Total Unique Visitors = Transaction Rate

It’s reasonable to believe that a test that improves transaction rate should be good for the bottom line of the business. In fact, it makes your paid marketing efforts (e.g. PPC, paid social) more cost effective because increasing transaction rate means you’re converting more of your current visitors. In theory, by increasing transaction rate your business should be getting more revenue from every dollar spent on paid marketing efforts.

How Transaction Rate Can Be Misleading

However, this is not always the case. For example, let’s say an ecommerce company decides to run a test on their product line-up. Specifically, they want to use a banner that calls out the “most popular” item.

image representing a/b testing the most popular item

After running an A/B test, the company finds that the variation successfully increased overall transaction rate by 10%.

a/b testing transaction rate kpi calculations

Based on these test results, it appears that implementing the variation design is the best business decision. However, transaction rate doesn’t tell the whole story:

a/b testing AOV kpi calculations

What the transaction rate KPI doesn’t reveal is that even though more of the visitors were converting (resulting in increased transactions), the “most popular” label drove some visitors to choose one of the lower priced products ($25 and $50 items) instead of the highest priced product ($100 item). This in turn led to the variation having a lower AOV ($42.05 vs $47.50). In this case, the decrease in AOV ended up having a significant adverse impact on revenue.

If the company implements the “winning” design based solely on the transaction rate performance, ultimately they would be losing revenue — not the best business decision.

A/B Testing KPI: Average Order Value

icon representing aov

Given the last example, it’s easy to see how Average Order Value (AOV) plays an important role in impacting an ecommerce company’s bottom line. As a result, teams may choose to use AOV as the primary KPI with the belief that if they can see an improvement in AOV, then this will lead to the best business decision. Calculating AOV is simple:

Total Revenue / Total Transactions = AOV

How Average Order Value Can Be Misleading

For example, let’s say the ecommerce business wants to run another test on their site, specifically focusing on the impact of providing free shipping to customers. Instead of providing free shipping on all orders the company wants to set a spend threshold. The test hypothesis is that the spend threshold will encourage visitors to buy the higher priced item to qualify for free shipping.

image representing website with free shipping on orders over $50

The company runs an A/B test and has the following results for AOV:

chart representing results of running an a/b test for aov

Based on the test hypothesis this would be considered a winning test because the free shipping offer did increase average order value by 12%. However, similar to transaction rate, this does not provide the full picture:

table representing how using aov as main kpi affects transaction rate

What the A/B test results show is that even though the “Free shipping over $75” led to a higher AOV, it also caused a decrease in transaction rate. One reason for this could be because there were frustrated customers who wanted free shipping, but didn’t want to spend the extra money to qualify. As a result, instead of choosing a lower priced item (where they would have to pay for shipping) they chose not to purchase. Thus, for this test the transaction rate had a far greater impact than AOV, which caused a decrease in the company’s bottom line, revenue.

The most significant risk to tracking only one KPI is that your team may be blind to what’s really happening with performance.

So far we’ve discussed two popular KPIs that are commonly used for ecommerce A/B testing. We’ve seen the potential pitfalls that can occur when you focus on just one of these metrics. The most significant risk to tracking only one of these KPIs is that your team may be blind to what’s really happening with performance. That leaves your team vulnerable to making business decisions that will actually hurt the bottom-line, instead of improve it.

A/B Testing KPI: Revenue Per Visitor

icon representing rpv

What the previous A/B testing KPI examples highlight is the need for a KPI that accounts for transaction rate and AOV, since both play an important role in determining revenue.

graph demonstrating revenue is impacted by both transactions and aov

Revenue per visitor is that KPI. It’s a composite metric that measures the amount of revenue generated each time a user visits your site. Like the other two metrics, it’s simple to calculate:

AOV x Transaction Rate = RPV

By accounting for both AOV and transaction rate, RPV removes potential blind spots and provides your team with a better picture of overall performance. Using the previous examples, here is what the test results would have looked like if RPV was the main KPI:

A/B Test #1 (Most Popular Label)

a/b test results if rpv was main kpi

A/B Test #2 (Free Shipping Over $75)

a/b testing results if rpv was main kpi on orders with free shipping

In both instances, RPV would have accurately reported that the variation had a negative impact on the bottom line, ensuring the company would make the right business decision not to implement the variation designs. Essentially, RPV is the KPI that provides reliability for your ecommerce test results, something that transaction rate or AOV alone cannot offer.

Using RPV Beyond A/B Testing

icon representing how to use rpv beyond conversion optimization

In addition to using RPV for A/B testing purposes, it’s also worth utilizing to monitor sales performance.

Specifically, if your RPV is trending downward, this could be the result of an increase in unqualified users to the site (e.g. your team launched a new social campaign) or potential site problems (e.g. issues with the payment page), which negatively affects your transaction rate.

Alternatively, site visitors may be converting at the same rate but are spending money on lower value items (e.g. higher priced product is out of stock), which negatively impacts your AOV.

Monitoring RPV provides you with another way of diagnosing potential issues with the site’s user experience, and allows you to take meaningful action.

Understanding A/B Test Performance

If you are conducting A/B tests on your ecommerce or other transaction-based website (e.g. SaaS), it is imperative to understand how test performance is being measured in order to determine if the right business decisions are being made.

If you find that your team currently isn’t tracking revenue per visitor, and are interested in changing to this better ecommerce KPI, have them reference this blog post that outlines how to leverage Google Analytics for user level revenue data. The post also introduces the free Revenue Per Visitor (RPV) Confidence Calculator, which calculates statistical significance for RPV data. It also provides a way to ensure that the A/B/n test results you’re seeing are having a meaningful and sustainable impact on your organization.

Thanks for reading! I’d love to hear your thoughts on this post, as well as the Revenue Per Visitor (RPV) calculator we created. If you have any questions, please post them in the comments.

Roopa Carpenter
About the Author

Roopa Carpenter is Vice President of Digital Experience (DX) at Blast. She leads a team of talented, DX consultants responsible for helping brands to better understand, optimize and measurably impact their digital experience. With many years of experience, Roopa offers a high level of knowledge and guidance to clients across a variety of industries regarding testing and personalization strategy and execution, user experience research and closing the empathy gap through Voice of Customer. Her data-driven approach focuses on impacting customer conversion and driving desired business outcomes.

Connect with Roopa on LinkedIn. Roopa Carpenter has written on the Blast Digital Customer Experience and Analytics Blog.