a/b or multivariate testing | clicks and clients

CREATIVE Dept too many secrets memo:

At the Creative labs here at Clicks & Clients we do a lot of web page CRO testing. But even for folks who do this a lot it isn’t always so obvious which kind of testing to start with.

Most likely an A/B test or Multivariate test is just what a page needs–but which is the right choice for a given site?

Hopefully this post will help shed some light…

A/B testing

With an A/B test we typically put up two identical web pages and change only one element on the page that we think will make it garner more leads, usually in the way of a form submit:

In this example test we think a different call-to-action button (CTA) will net more form submits. 

This is an especially good test for those of you who might already have a pretty strong landing page that 1) loads fast, 2) is usable and intuitive, 3) has a nice content-flow going from left-to-right and top-to-bottom with 4) a relevant headline and 5) an easy-to-find CTA, which is to say the page is already probably pretty well optimized and it’s time for the kind of fine-tuning that usually only an A/B test can determine.

Some pros to consider if you’re considering an A/B test may be:

  1. Because you’re typically only testing one element on the page at a time in an A/B test, it’s easy to design and setup
  2. You can determine which page is the winner in a short amount of time since the results don’t often require a lot of traffic
  3. You can move on to the next element you want to test fairly quickly

However, some of the cons to A/B testing are you’re 1) very limited in the number of changes that can be tested in an A/B experiment and 2) it can be difficult to determine how the different variables on a given page may impact each other.

Multivariate Testing

When we do multivariate testing, however, we can do several A/B tests at once. 

In this example we’re testing two elements on the page for a total of 4 variations using 2 headlines and 2 buttons. 

A couple of the pros of multivariate testing are 1) we learn a lot more about how different elements on a given page may impact one another (something A/B testing doesn’t reveal as much) and 2) we get a more powerful way to test variables (here we may even be permitted to try complete redesign).

Some of the cons of a multivariate test are 1) they’re typically more complicated to design, require more complex hypotheses and take longer to setup and 2) they require a lot of traffic (and if you’re lacking in volume, it’ll require more time to determine a winner).

So which do you choose–an A/B or Multivariate test?

If you’ve read this far and if you know your web page at all you probably already have a pretty good idea, but keep things simple (K.I.S.S.).

Maybe try jotting down the first few things that come to your head that could be made into tests.

For instance, when I’m analyzing a new page I like to load the mobile-version of it onto my mobile browser and then flip back and forth to my notepad app

While studying it I can usually spot a few things on the mobile-page that catch my eye that may be detrimental and should probably be tested (e.g., is the CTA a unique color that’s different from the branding colors? Is the featured image out-of-place or possibly covering or competing with the headline or submit button?)

Does the CTA/submit button have a good call to action, or is it generic (e.g., “Submit” or “Send” vs “Get Started Now.”) 

Is the headline conversational and/or does it relate to my audience?

Are there other buttons or elements on the page that are competing for the CTA’s attention, of which often should otherwise be the main point of focus?

Try something like that and see how many quick testing ideas you can come up with–you may be surprised by how much jumps out at you.

My web page is already pretty solid

As stated before, some of you might indeed already have a landing page that checks all the boxes: loads fast, intuitive, content-flow going left-right-top-bottom, et al.

Others may have some combination of the above but believe they could optimize other things, but aren’t sure what, why or how (maybe you’ve tried adding a discount offer or a coupon or have emphasized the quality of the customer experience you get or a money back guarantee if not completely satisfied (or something else unique to your business)).

These are all powerful steps that can and will help.

But if you’ve tried most of these things and have seen only sluggish increases in lead interest over time there’s a good chance an A/B or multivariate test is the next logical step. Because there’s a good chance you’ve gotten as far as you can get without one.