Analytics & Digital Marketing Tips

Google Website Optimizer Test: We Didn’t Implement the Winner

Google Website Optimizer Test: We Didn’t Implement the Winner

December 14, 2008           Conversion Testing

Google Website Optimizer Multichannel Ecommerce Case Study

Why was this Website Optimizer test so important?
This test makes a great case study for the value of follow up tests, how strong conversion rates can still be improved substantially, and most importantly provides a critical lesson in why the Google Website Optimizer results cannot be taken at face value. You must truly understand a website’s underlying business goals and sometimes make small compromises on the volume of leads in turn for higher quality leads.

Let’s start from the beginning…
We identified several deviations from lead generation best practices on the AudioEditions.com ‘Request a Catalog’ page that we believe were adversely impacting the conversion rate. Thus we ran a test to see if taking out the left and right side columns (most importantly removing the call to action for “Request a Free Catalog” since the user is already on the page to do this), synced up the headline with the offer to reinforce the “Free” expectation, adding a larger cover image (to set clearer expectations of what they are requesting), and simplified the form fields down to the minimum info needed for the catalog request (especially since some of the fields were difficult to understand).

Our Experiment Hypothesis
You must always start any test out with a hypothesis. The hypothesis process is key in determining whether or not this is a test that should even be run. In this case study, our hypothesis was that we could gain at least a 25% increase in conversion with the new variation that will remove the apparent obstacles.

Website Optimizer Test #1

In our first test we ran a simple A/B test. Here is the original page (A)

URL: http://www.audioeditions.com/audio-books-catalog.cfm

Here is the revised page variation (B)


URL: http://www.audioeditions.com/audio-books-catalog-v1.cfm

Which version of this page won; A or B?

Here is the Google Website Optimizer Report Screen


Valid Concerns from the Client

While this first ‘Request a Catalog’ test demonstrated a 67.4% increase in conversion we received valid concerns from the client that the improvement was likely just related to the layout/content changes and not the elimination or modification of form fields.

Follow Up Website Optimizer Test #2

So we ran a follow up test to validate that the conversion increase was not just a result of the design, layout and headline changes but also substantially impacted by the form field changes. This way the client could properly gauge the business value of adding each critical form field and decide whether or not the conversion impact was worth it.

Tip: Google Website Optimizer provides a link on your completed tests for easily running a follow up test as in this example. If we had stopped with the first test and implemented the improved page variation you can see below the important lessons that would have been missed.


Here is the follow up test using the original (A) against 3 (B) page variations


URL: http://www.audioeditions.com/audio-books-catalog.cfm

Here are the three (B) page variations

URL: http://www.audioeditions.com/audio-books-catalog-v1.cfm

URL: http://www.audioeditions.com/audio-books-catalog-v2.cfm


URL: http://www.audioeditions.com/audio-books-catalog-v3.cfm

Which version of the page won; A, B1, B2, or B3?

Again the original (B) page variation won. Here is the Google Website Optimizer Report Screen showing the dramatic differences that the form fields made:

Why didn’t we implement the winner?

After discussing the follow up tests with the client it became clear that even though variation B1 had the most dramatic 41.7% improvement; the variation B3 with a 32.4% improvement was the best choice since it gave the client much more (customer data) value and the ability to digitally deliver marketing in the future (which is key). The final conversion rates were not that different and understanding the client’s underlying business goals it was the best decision to make a small compromise in the volume of leads for the higher quality of leads.

Just because you have a good conversion rate doesn’t mean it can’t be better!

The initial form conversion rate was 22.1% and the final conversion rate was 32.5%. Many people may think that such high conversion rates aren’t worth bothering with but clearly we demonstrated that there was still considerable room for improvement. High value pages on a site should always be the focus of your efforts and continuously analyzed for further testing and improvement.

 

  • Great Title for this post, indeed you showed your client what the best options were but really that extra data for newsletter and customer data is worth more that just sheer volume of less detailed responses. Great walkt hrough on the value of split testing and looking at value as more than just conversions but going after higher quality conversion data.

  • Great Title for this post, indeed you showed your client what the best options were but really that extra data for newsletter and customer data is worth more that just sheer volume of less detailed responses. Great walkt hrough on the value of split testing and looking at value as more than just conversions but going after higher quality conversion data.

  • Fantastic stuff. If clients would focus more on deploying these kinds of experiments and less on fluffy flash animations they might also see such results. We’re making an attempt in 2009 to simplify reporting and your layout of this post stimulated a few thoughts. I wish GWO had some more creative reports that use thumbnails… perhaps a good 3rd party app idea.

  • Great break down. It’s good that they saw far reaching benefits of Variant 3 as opposed to the “results now” you got with Variant A.

    I liked seeing a case study broken like that, very well done.

 

Analytics Blog

We're here to help with tips and insight on the following topics:

Subscribe to RSS


Optimize your website and marketing campaigns

Get a constant flow of Google Analytics help and digital marketing tips, case studies and more from Google Certified Partner Blast Analytics & Marketing.




Connect with Blast Analytics & Marketing