How do you know when your AB test, split test, multivariate test or any other conversion rate optimization task is complete? When do you know you've found a winning experience that you can confidently apply to your web experience and expect the conversions to come rolling in?
Well the key thing is statistical significance of course, where essentially the volume of visitors to your test have voted with their feet, you've separated signal from noise and so on...
BUT, and it's a big but, have you allowed your test to play out through key trading times?
In my current company we have a number of trading factors that mirror many other eCommerce businesses:
1. Day of week
2. Visits by hour
2. Visits by hour
Once a CRO test has lived through several of these, plus a number of other factors specific to our industry (seasonality) we can safely say the new experience has done it's tour of duty and is fit to implement.
Reaching a statistically significant result in itself is therefore not enough. Conversion optimization experts must ensure that their test audience takes into account critical factors that impact their tests. This is important because one variation may appeal to one season or segment more than others and ultimately misguide the result.
