This post discusses the series of Conversion Conference Sessions I attended.
Why (and How) to Run 250+ Online Experiments a Year
Ton Wesseling, CEO, of Testing.Agency
Ton is a web analyst who became a growth hacker, an optimization lunatic and entrepreneur. He founded the biggest blog in the Netherlands on Analytics and Optimization (webanalisten.nl) and organizes many events on this subject in the Netherlands. He started well known optimization firm Online Dialogue five years ago and is now the CEO of Testing.Agency – an A/B-test development and management machine. His life is all about growth optimization and bare foot presentations to inspire the world to get more value out of their digital efforts.
During the recently-concluded Conversion Conference, he gave a very enlightening presentation on Why (and How) to Run 250+ Online Experiments in a Year. Now, the number alone seems daunting, but Ton makes it work. Here are some important points from his presentation:
The reason a lot of people are not testing is because they lack resources, budget, and strategy, among others. As a result, they lose opportunities to find out how they can make their websites more profitable.
A/B testing is hot, with everyone pushing to test more. It is one of the most commonly-used and proven methods to improve conversion, staying ahead of the pack at a whopping 67%. Keep in mind, however, that it is not an end-all and be-all solution – it is a method. It isn’t just testing, but a continuous optimization program.
With 1000+ conversions a month, your test capacity is at 20+ tests a year maximum, with a statistical power of 80% on predicted uplifts of 15%. On average, 1 out of 3 tests have measureable impact. This means you’ve got impact every 7 or 8 weeks.
Testing seeks to provide answers to these two questions: Why are these users here? How can you help these users’ brains to fulfill their needs? You also need to keep in mind that testing takes time, so best to use it to learn.
Some more tips:
*Use this process: Find -> Analyze -> Create -> Test
Analyze -> Conclude -> Tell
*Know the site/app that need to be optimized thoroughly
*Apply the ease/effect matrix and test schedule
*Know your users, as well as their journeys – there are three ways to go about this, first through data, then by asking them why they are on your site, and finally, by listening to their needs
*Use your analytics data
*Use event flow reporting
*Create heat maps and screen recordings to find out the critical locations
*Apply psychology. Remember, your team needs an economic/consumer Psychologist and a UX designer
*Using that persuasion knowledge, design your new dialogue. You will need a good front end developer for this.
*Test, measure in analytics, analyze event tracking variation behavior, then calculate results
*Analyze your results. If it’s not a winner, stop. But it doesn’t mean it’s a losing one. An average of 33% of your tests will have winners
*Choose what to analyze up front. Go and dig until you find something
*Don’t stop the test until it is significant, and don’t stop the test too soon
*Your test has a fixed length, so be aware when your test takes too long
*Don’t test broken stuff – always check browser compatibility and don’t break dynamic stuff
*Slow down the control down – add the same code to the control
*Leave out some visitors – only test on those who can be changed
*Only test on fresh people – start with control for a full purchase cycle then start sending traffic to your variations, setting the control to 0%
*Give your audience time to buy! Stop getting fresh visitors after your test time Send new traffic to the control, but give your tested people time to finish converting. Take it easy.
*Conclude. Your psychologist should conclude and your conversion lead should tell.
This is where you decide your next step: Do nothing, retest, implement, or scale and continue or change test roadmap?
Never stop testing, because A/B testing is your ultimate resource. Minimize regret, run all the experiments per year that your site or app can handle.
Take my word for it – you should NOT miss out on the next event. It’s the least expensive it will ever be. It is definitely worth the trip to Vegas. The rooms and airfare were actually super cheap too. Grab your tickets for MAY 3-5, 2016 @ THE RIO IN LAS VEGAS at