Blast Analytics and Marketing

Analytics Blog

Supporting Leaders to EVOLVE
Category: Data Management

Optimizely X vs Classic: Top 8 Reasons to Upgrade

August 24, 2017

If you’re familiar with the landscape of A/B testing tools, you probably noticed that toward the end of last year, Optimizely rolled out a new conversion optimization platform: Optimizely X.

Optimizely X is a big step forward and we strongly recommend optimizers and testers start using it as soon as they can. Optimizely X — and Optimizely X Web in particular — builds on the foundations laid by their original A/B testing tool, Optimizely Classic, and adds many more features.

A Far Superior Conversion Optimization Platform

Below we’ve listed the top 8 reasons why the battle of Optimizely X vs. Classic is an easy win. If you run your company’s testing program and need to convince others to upgrade, this post should arm you with the information you need to present the case for the switch to Optimizely X.

#1) Performance gains

performance gains iconThe dreaded A/B test “flashing” (when a user sees a page flick between different test variations) should now be a thing of the past. With Optimizely X, the Optimizely team has significantly reduced the size of the snippet you have to load to your site in order to run A/B tests.

Optimizely has achieved this by making a number of updates:

  • jQuery is no longer required to run experiments, meaning no jQuery dependency needs to be built into the snippet
  • draft and paused experiments are no longer included in the snippet
  • experiments and events have been moved from cookies to local storage, thereby reducing browser cookie bloat and improving performance when tracking events are sent to Optimizely log servers

Also, for those larger sites looking for even more performance gains, Optimizely X now supports custom snippets. This means teams can build snippet payloads that include only the experiences relevant to the page a user is on, great for reducing snippet size and managing experiments efficiently.

This should mean pages and experiments load faster, resulting in happier users and site engineers!

#2) Built for modern web and your future testing

future testing iconThis is a big reason to upgrade. With the Optimizely X conversion optimization platform, Optimizely has looked beyond web A/B testing. The new suite of products in this conversion optimization platform includes:

Optimizely X is ready to grow with your testing program.

As your testing experience deepens and your optimization program grows, you will naturally begin to run out of things to test. If you continue to stick with testing only aesthetic changes to web pages then your ability to impact your business’s bottom line will become increasingly more limited.

Optimizely has seen this happen with their customers and have built their Optimizely X suite to enable you to broaden the types of tests you can run.

With Optimizely X you can ensure your testing program continues to have a positive impact on your business well into the future.

Perhaps you want to test different price structures, complex changes to the back end of your website, or start serving users personalized and recommended content based off of their browsing history.

Each of these would have been challenging with Optimizely Classic, as its primary focus was on allowing you to test smaller changes to the front-end look and feel of your site.

Optimizely X provides a variety of products to support you in these more complex testing and optimization scenarios. By upgrading to Optimizely X now you’ll be ready to seamlessly start integrating the best conversion rate tools into your testing program.

We see personalization as the next big step in optimization efforts, once you have mastered A/B testing.

Increasingly customers are coming to expect you to serve them personalized content.

If you make the switch to Optimizely X vs. Classic now, you’ll be giving your organization a head start for when you’re ready to progress your optimization efforts and start focusing on personalization using the best conversion rate tools.

#3) Visual Editor & Preview have been simplified

Eye iconOptimizely has made its Experimentation tool even easier for the non-developers among us. A simplified interface means all visual changes to an experiment can be made in the left-rail navigation.

The Visual Editor can also load your website faster and more consistently with the addition of Optimizely’s Desktop App. You’re now able to drag and resize the editor window to see how your page resizes at common breakpoints.

No need to preview mobile versions to test this anymore!

The screenshot below shows the simplified left-rail navigation that you’ll see for any visual element you click on to edit:

example of new optimizely x left rail navigation

The next screenshots shows how you can easily test your variations’ responsiveness at common breakpoints from within the Visual Editor:

example of optimizely x visual editor

Preview itself has also been updated so that when previewing you can quickly and easily evaluate all aspects of your experiment, including:

  • how variations are looking,
  • if audiences are being selected correctly, and
  • whether variation code and event tracking are firing correctly.

No need to load developer tools and fuss with force parameters anymore (although these options are still available with Optimizely X if you wish to use them)!

#4) New Results page and updated Stats Engine

icon representing resultsAlong with updating the look and feel of the results page, Optimizely has made a couple of adjustments to the Stats Engine that powers the results page and helps you understand which of your testing variations are the best.

These updates are designed to help companies make better business decisions, rather than simply getting a winning test.

Instead of showing absolute differences between original and test variations, the Optimizely X results page now shows “confidence intervals.”

This means the stats shown to you are based on the relative difference between variations and the original. So an improvement interval of 5% means your variation is converting 5% better than your original.

This change should help you reach a quicker understanding of which of your variations are proving most effective.

Optimizely has made updates to how their Stats Engine accounts for the False Discovery Rate in your experiment (i.e. the chance that your experiment returns a false positive). Part of this update means that you can now rank your experiment goals, and doing so is quite important.

Here’s why:

Imagine you’ve set up an experiment with 10 goals. There’s 1 goal that you think of as your primary goal that will determine the success or failure of the experiment, 4 goals that you think of as secondary goals, and 5 goals that can be considered diagnostic goals. (You use the diagnostic goals to debug the experiment and provide an overall picture of site usage during the experiment.)

The primary goal is the most important for your business, but the original False Discovery Rate checks that Optimizely Classic had in place treated all goals as equals.

This meant that your primary goal was often slower at reaching statistical significance because Optimizely was also equally considering the significance of all the secondary and diagnostic goals you had in place.

Conversion Goal Ranking

Now, in Optimizely X you can rank your goals in order of importance:

  • Goal 1 is your primary metric and will be the one that reaches statistical significance quickest.
  • Goals 2-5 are secondary goals and will reach significance more slowly than your primary goal.
  • Any goals after goal 5 are diagnostic goals and will reach significance even slower than your secondary goals — they should only be used to help you debug the test, rather than waiting for a “winner” to be reached based on their results.

The screenshot below shows an example. For this experiment you can see that “Overall Revenue” is set up as the primary metric — ultimately the main indicator as to how well the test variations are performing.

Following “Overall Revenue” are 4 metrics that we see as being good indicators as to how well the test variations are performing, though not as important as “Overall Revenue.”

They include: “Visit Checkout”, “Add To Cart”, “Product Selected” and “Visit Product Page.” These are our secondary goals.

Below the secondary goals we have metrics 6 – 9, diagnostic goals. These goals will help us get an overall picture of how users are interacting with the site during an experiment, and may prove useful if we need to debug the experiment.

However, they’re not going to provide an indication as to whether experiment variations are proving successful or not. The diagnostic goals in this test are: “Log In”, “Log Out”, “Scroll 50%” and “Visit Blog Page.”

example of optimizely x new results page

These updates mean that ranking your goals in Optimizely X is an important step, and ultimately one that should help your testing results support relevant and impactful business decisions.

#5) Easier to use on Single Page Applications

icon of single pageAs Single Page Applications continue to grow in popularity, Optimizely has stepped up their support for SPA experimentation. If you’re reliant on listening for hash changes to activate an experiment then Optimizely X supports this by allowing you to build URL targeting and experiment activation conditions one time and use them throughout experiments.

There are also a host of “page-level conditional activation” options that allow you to specifically control when and where experiments activate based off of on-page interactions.

For example, if you want an experiment to trigger after a user clicks on a specific menu, or on a state change within an SPA, then in Optimizely X you can set a conditional activation mode for this at the page level to ensure the experiment triggers when you want it to.

There are three different conditional activation modes Optimizely X provides: polling, callback and manual.

Polling is ideal when you want your experiment to trigger when a certain element is present on a page. If polling is set as your activation mode, Optimizely will poll your site every 50ms for 2 seconds to check if the set condition is true.

screenshot of optimizely x activation polling

If you want your experiment to trigger on a specific event, such as a click on a feature of a state change in an SPA app, then you’ll want to use the callback mode.

screenshot of optimizely x activation trigger

If you select the manual activation mode then you’re required to add code to your site telling Optimizely when you want your experiment to start — more information about this can be found on the Optimizely developer site.

screenshot of optimizely x manual activation

#6) Custom Code Editor

code editor icon

The big transition here in Optimizely X vs. Classic is no more dependency on the JavaScript library jQuery.

In Classic, any changes you made in the Visual Editor generated jQuery in the Code Editor, sometimes making it difficult to keep your code organized when using both the Visual Editor and Code Editor.

The screenshot below shows how a few small adjustments made in the Visual Editor quickly start to fill up your Code Editor with lines of jQuery that aren’t always targeting elements in the cleanest fashion. Now, unless you want to use jQuery in your custom code, there’s no need to load the jQuery library for Optimizely X, helping performance.

example of optimizely x code editor

Compare with how variation code now executes immediately upon experiment activation, and if you want greater control of timing then you can make use of Optimizely X’s utility functions.

Updating the styles of your variations has also gotten easier as you can now write CSS directly for your experiment in the new CSS code editor:

example of optimizely x css code editor

Another great way in which Optimizely X can increase the efficiency of your testing is through the use of Extensions. These allow developers to build custom features such as carousels, light boxes, or banners as reusable templates in the Visual Editor.

Once an Extension has been built, non-developer testers can reuse them in as many experiments as they want. This should really reduce the amount of developer time needed for your testing program and allow your non-technical testers to more efficiently set up and run tests.

Below shows a simple banner extension that can quickly be deployed across multiple experiments by your team, without the need for developer support:

example of simplicity of optimizely x code editor

Optimizely X also offers developers a new suite of tools to help them build and debug tests, including a JavaScript Utility Library, JavaScript API, REST API and Optimizely logs.

#7) PCI Compliance

pci compliance iconOptimizely X was recently updated to meet the demanding security standards specified by the Payment Card Industry Data Security Standard (PCI DSS).

This means if you test on a PCI-compliant website you will now be able to use Optimizely X to run tests on pages in which users enter credit card information — meaning you can test the whole checkout funnel, something that was not previously possible on PCI-compliant sites.

#8) They’ll force you to migrate eventually!

optimizely x iconExisting Optimizely Classic customers have the option to upgrade to Optimizely X Web, or continue working with Classic. Whilst no indication or timeline has been given for the day all existing Optimizely users will be forced to migrate to Optimizely X, it seems likely it will happen in future.

Given the upgrades available with Optimizely X vs. Classic, it’s unlikely that Optimizely would want to indefinitely keep supporting both conversion optimization platforms in parallel.

It’s best to start getting familiar with Optimizely X now so that you’re ready for the day when Classic is switched off.

Optimizely X Upgrade FAQs

Now that we’ve covered some of the main reasons why switching to Optimizely X is a good idea, here are answers to common business questions we hear when discussing the upgrade.

Do we have to pay for a new plan to use Optimizely X? Are there additional associated costs with making the switch?
There’s no extra cost involved when you’re making the switch from Optimizely Classic to Optimizely Web Experimentation. However, if you wish to use Optimizely X Personalization or any of the other new products included in the X suite then you’ll need to pay for new plans. Reach out to your Optimizely Rep for details and pricing.

What if I’m not ready to completely switch to Optimizely X yet? Can I run Optimizely X and Classic at the same time?
Yes you can. There is nothing to stop you running Optimizely X and Classic at the same time, you could leave existing Classic experiments running whilst learning to build new experiments in Optimizely X. In fact, this “Phased Rollout” is recommended by Optimizely if you don’t have a dedicated QA environment and build, QA and run all experiments in your production environment.

See below for where you need to go in Optimizely X to enable both X and Classic in your snippet:

The negative to this is that your bundled Optimizely snippet is going to be around 50kB larger than a snippet for either Optimizely X or Classic. This could have speed implications for your site so it isn’t a viable long term solution — you’ll want to fully switch over to Optimizely X as soon as you’re ready.

Do I need to add a different Optimizely code snippet to my site to make the switch?

No. If you’re taking Optimizely’s phased approach to transition to Optimizely X and will be using it concurrently with Classic for a period of time then you simply need to enable the Optimizely X bundled snippet in your Implementation Settings.

When you make the full switch to Optimizely X and completely stop using Classic then you will need to again make a change in your Implementation Settings and check the ‘Use only Optimizely X’ option in the Snippet Configuration settings. The snippet you have added to your site stays the same.

What about all my Classic integrations? Will I need to re-integrate any products I want to use alongside Optimizely X?
As Optimizely X is effectively an entirely new product offering, you will need to redo your integrations. The process to setup integrations is similar to the one in place on Classic so shouldn’t prove too much trouble. Optimizely also provides great supporting docs for common integrations with Optimizely X.

One of the most common integrations we come across is between Optimizely and Google Analytics Universal. If you have this integration setup in Classic and are upgrading to Optimizely X than you can expect the new integration to be even simpler to complete. With Optimizely X the Optimizely and Google Analytics snippets can be placed on your pages in any order, and you no longer need to add an activateUniversalAnalytics API call to your pages. You’ll need to enable the Google Analytics Universal Analytics integration from within your Implementation menu (much the same as for Classic) and assign a Google Analytics Custom Dimension to receive experiment data – and that’s it!

At this point Optimizely X does not have as many Optimizely nor partner built integrations available as Classic, so integrating some tools that you’ve traditionally used alongside Optimizely may be challenging and is something you’ll want to consider before making the switch.

Do my goals and audiences migrate over from Classic or do I need to re-create them?
Generally you won’t need to recreate Audiences as these are shared between Optimizely X and Classic and vice versa. The only time you would need to recreate an Audience when upgrading is if your Classic Audience contains custom tags. This Audience type has been deprecated in Optimizely X, as the new pages feature replaces this functionality. Also, Optimizely X Audiences that use Visitor Behavior will not work in Classic as this is a new feature available only through Optimizely X Web Personalization.

However, you will need to re-create goals when transitioning to Optimizely X. Goals have been upgraded to events which are more powerful and can be used across multiple campaigns.

If you want to migrate all of your Custom Events from Classic to X then you can contact the Optimizely Customer Support team who can help you do this programmatically.

Will I retain my archived experiments in Classic if I completely switch over to X?
Yes! They won’t be going anywhere so you can always look back into Classic to view your archived experiments and relive the good old days.

Will I be able to copy and paste my campaigns from Classic into X?
No, campaigns and experiments exist either in Classic or Optimizely X and will not show up in both unless you build duplicates. If you want to duplicate a Classic experiment in Optimizely X you’ll need to build out the new pages, events, integrations and variations in Optimizely X from scratch. As long as you have jQuery loaded on your site you should be able to reuse your variation code by copying and pasting it into the Javascript editor that Optimizely X provides.

Optimizely X vs. Classic: No Contest!

Hopefully this has helped you compare products, understand some of the improvements Optimizely X has made over Classic, and learn how it could help you and your testing program.

Make the switch!

Alex Molineux
About the Author

Alex is Associate Manager of Optimization at Blast Analytics. He leverages his expertise in analytics strategy, implementation and optimization to help clients translate data into actionable insights, and is an expert in a wide variety of analytics and testing tools with a focus on Google Analytics, Google Tag Manager, Tealium IQ & AudienceStream, Optimizely and Google Optimize. Whether you’re looking to build out an analytics strategy to gather insights into your core business questions or you are looking to run an effective A/B testing or personalization campaign, he is able to use his expertise to guide and support you.

Connect with Alex on LinkedIn. Alex Molineux has written on the Blast Digital Customer Experience and Analytics Blog.