Google Website Optimizer Test: We Didn’t Implement the Winner

December 14, 2008Kayden Kelly

Google Website Optimizer Multichannel Ecommerce Case Study

Why was this Website Optimizer test so important?
This test makes a great case study for the value of follow up tests, how strong conversion rates can still be improved substantially, and most importantly provides a critical lesson in why the Google Website Optimizer results cannot be taken at face value. You must truly understand a website’s underlying business goals and sometimes make small compromises on the volume of leads in turn for higher quality leads.

Let’s start from the beginning…
We identified several deviations from lead generation best practices on the AudioEditions.com ‘Request a Catalog’ page that we believe were adversely impacting the conversion rate. Thus we ran a test to see if taking out the left and right side columns (most importantly removing the call to action for “Request a Free Catalog” since the user is already on the page to do this), synced up the headline with the offer to reinforce the “Free” expectation, adding a larger cover image (to set clearer expectations of what they are requesting), and simplified the form fields down to the minimum info needed for the catalog request (especially since some of the fields were difficult to understand).

Our Experiment Hypothesis
You must always start any test out with a hypothesis. The hypothesis process is key in determining whether or not this is a test that should even be run. In this case study, our hypothesis was that we could gain at least a 25% increase in conversion with the new variation that will remove the apparent obstacles.

Website Optimizer Test #1

In our first test we ran a simple A/B test. Here is the original page (A)

URL: http://www.audioeditions.com/audio-books-catalog.cfm

Here is the revised page variation (B)

URL: http://www.audioeditions.com/audio-books-catalog-v1.cfm

Which version of this page won; A or B?

Here is the Google Website Optimizer Report Screen

Valid Concerns from the Client
While this first ‘Request a Catalog’ test demonstrated a 67.4% increase in conversion we received valid concerns from the client that the improvement was likely just related to the layout/content changes and not the elimination or modification of form fields.

Follow Up Website Optimizer Test #2

So we ran a follow up test to validate that the conversion increase was not just a result of the design, layout and headline changes but also substantially impacted by the form field changes. This way the client could properly gauge the business value of adding each critical form field and decide whether or not the conversion impact was worth it.

Tip: Google Website Optimizer provides a link on your completed tests for easily running a follow up test as in this example. If we had stopped with the first test and implemented the improved page variation you can see below the important lessons that would have been missed.

Here is the follow up test using the original (A) against 3 (B) page variations

URL: http://www.audioeditions.com/audio-books-catalog.cfm

Here are the three (B) page variations

URL: http://www.audioeditions.com/audio-books-catalog-v1.cfm

Why didn’t we implement the winner?

After discussing the follow up tests with the client it became clear that even though variation B1 had the most dramatic 41.7% improvement; the variation B3 with a 32.4% improvement was the best choice since it gave the client much more (customer data) value and the ability to digitally deliver marketing in the future (which is key). The final conversion rates were not that different and understanding the client’s underlying business goals it was the best decision to make a small compromise in the volume of leads for the higher quality of leads.

Just because you have a good conversion rate doesn’t mean it can’t be better!

The initial form conversion rate was 22.1% and the final conversion rate was 32.5%. Many people may think that such high conversion rates aren’t worth bothering with but clearly we demonstrated that there was still considerable room for improvement. High value pages on a site should always be the focus of your efforts and continuously analyzed for further testing and improvement.

About the Author

Kayden Kelly

Kayden is the Founder and CEO of Blast Analytics & Marketing. He leads overall strategy and execution in positioning Blast as a leading analytics and digital marketing consulting company. An industry trailblazer who stays ahead of trends and opportunities, Kayden has spent nearly two decades building his expertise in all areas, with a deep focus on analytics, search engine optimization, and user experience. He is an evangelist for “the customer,” helping clients create sustainable competitive advantage by developing, optimizing, distributing, and connecting valuable content to customer needs. Connect with Kayden on LinkedIn. Kayden Kelly has written 48 posts on the Blast Digital Customer Experience and Analytics Blog.