Multivariate testing (also called MVT, multivariable testing, or experimental design) is one of the three common techniques for landing page and onsite optimization (A/B testing and onsite targeting being the other two) though it can be used effectively for ad testing as well. While multivariate testing does not usually provide the performance lifts of A/B testing or the ability of onsite targeting to deliver relevance when used properly it is an essential part of ongoing and iterative online marketing testing and optimization.
MVT provides incredible intelligence and learning. The greatest benefit of MVT is what is referred to as element contribution. Element contribution informs the marketer of the percentage contribution each tested element has on conversion rate. This learning is highly valuable as it provides an understanding of the triggers that influence behavior and can lead to numerous iterative test ideas that can provide further learning and improvements in conversion.
There are two main design experiments that marketers use and multivariate testing vendors base their technology around. These are fractional factorial design and full factorial design. Let’s examine the pros and cons of each.
Fractional factorial testing requires the marketer to choose two or more elements to test using one or more variations of the selected elements. This method is called fractional because not every page element is tested. The marketer selects what they believe are the most important elements based on what they are testing.
The benefit of fractional factorial testing is that less data is required in order to reach statistical confidence in the results. This means that with a solid test design results can usually be achieved in shorter time periods. Usually weeks. This is highly beneficial for marketers that like to “fail faster” and take more iterative approaches to testing.
Since not every element is being tested certain interactions are coded. This means that while the results may achieve statistical confidence full interactions between all elements are not measured. If the marketer selected the most important elements these interaction effects can be mitigated. Of course, it's not always easy to determine the most important test elements.
Full factorial testing uses every element of the page to create a test array measuring all interactions. Because so many interactions are measured this method requires a great amount of data (thus time) to achieve results.
As mentioned each interaction is accounted for so when confidence in the results are achieved there are no questions about possible interactions skewing the results.
Since every element must be considered and tested a large amount of data is necessary to achieve results. Incorporating variations of each element can also be a taxing creative effort. Because of these reasons full factorial tests need a much longer period of time to gain results, usually months. SInce online behavior can change (for many reasons) over such long periods results may tend to benefit certain temporal swings.
Successful testing starts with clear goals. It is these goals (metrics) that will inform your strategies for element selection and variation. Being that we are interactive marketers our metrics tend to be an action of some sort or another. This action (sometimes referred to as conversion) can be a click, it can mean a sale, it can refer to any outcome. For example, if I am trying to increase Average Order Value (AOV) or Revenue per Visitor (RPV) I may select different elements (and most certainly create different variations) than if I’m optimizing Order Conversion Rate. Making a decision what success means before you start test design is important but it is essential before you start this part of multivariate test design.
Once we begin testing a large part of the value in MVT is the element contribution reporting. The best tests will answer questions for you around your goal but the element contribution reporting will raise many additional questions that you’ll want to find the answers to. Therefore the next consideration we want to make with our test design strategy is to select elements where we desire to achieve greater understanding of their influence on the outcome.
The nature of MVT and the learning created lend itself to ongoing and iterative testing cycles. In Part 1 I discussed that the nature of fractional factorial testing requires us as marketers to select the elements that we believe are the most important factors to the metric we are trying to optimize. What influences an action? It can vary quite a bit. There are no defined best practices for element selection for each type of test array. Generally however we look to the following elements as those that have the most influence on people’s experience.
There are some strong opinions in optimization around having or not having navigation on landing pages. The best thing, of course, is to test.
Headlines are often the most influential element on the page. Many elements may not be noticed on your page but more often than not the headline is noticed as people have been trained to look for and scan headlines for clues to relevance.
Offer testing can lead to huge learning. How does your audience best respond to offers and how does this affect your bottom line results? Does 10% or 25% Off make a difference for AOV or Conversion? Do more act with a dollar offer or a percentage. Do gated offers ($10 off orders of $50 or more) get better metrics than straight percentage off offers?
How do you present your benefits and unique selling propositions? Are bullet points enough or do you need more copy? In what voice are you speaking to visitors (more on this in element variation). What is your mix of content?
Sometimes nothing makes more of an impact than price. If you have the ability to sync price testing with your back end reporting and analytics this is essential to understanding the market value of your products and services.
Those little buttons and links really matter a lot. I love doing MVT on the buttons themselves however as an element in an array you can often get a quick win testing a button or call to action link.
Element variation is the most creative aspect to MVT. It requires a mix of marketing acumen, an understanding of creative and some jigsaw puzzle skills.
The marketing aspect to element variation presents itself in the themes that you create to test against one another. For example the headline can be constructed to be emotional, aspirational, direct, inquisitive, helpful and any number of other themes. These messages can also be presented with a certain voice like peers, experts, underdogs or others.
Creative differentiation is the primary key to testing success. The best test variations will look nothing like one another. Something I do regularly with test designs is, stand 5-10 feet back from my monitor and look at the variations to ensure they are distinct from one another. I’m not talking about background color or font style, I’m talking about radical creative differentiation.
Since we are creating an array these radically different pieces will have to come together in their different combinations and variations and need to present themselves as a unified experience. This is one reason it’s helpful to understand what your array is going to be prior to element creation. Puzzles are fun, just make sure you are not missing any pieces.
Here are some tactics for element variation that I recommend:
Test things being present or not present. There is no better gauge of what factor of influence they have on performance. More often than not you will find that addition by subtraction is a way to improve your results.
How much information is needed? How should it be presented? What should it be associated with? Should it link out, have icons, bullets, images? Should it be customizable? Does any of it matter? These are all questions a good MVT can answer.
The paradox of choice is a rule that I’ve seen proven countless times in testing. Sometimes it takes a test to prove the value. Other times there has to be more than one choice presented. MVT can help determine how many choices, the order of selection, where or if defaults should be present and any number of other questions around selection.
At RAMP Digital we believe simple is the new sexy. That ajax scroll bar is cool but is it more effective than a static image? How about your navigation, buttons, images. What looks good and what performs well are often two different things. Rich media can be incredibly useful and improve digital experiences and performance immensely but the only way to know is to test.
As mentioned, there are multiple messaging themes that can be tested. This provides really valuable feedback that can be used not only on the web but in other media to find the “voice” that best connects with your target audience or within certain segments. Segmentation is very important when analyzing results based on theme. The classic example is the dating site where the image of a couple performed better with women but the image of a woman alone performed better for men.
Multivariate testing can be a tremendous amount of fun and get you great results but it requires highly dedicated marketers and great creative methodology. Matt Roche the founder of Offermatica once shared three learnings from his time building the most successful multivariate testing tool. I'll end with his great advice for digital marketers.
Content marketing guru at Mailmunch. I’m passionate about writing content that resonates with people. Live simply, give generously, stay happy.