A/B Testing for Landing Page Optimization: What to avoid

A/B testing is a scientific way to optimize landing pages. It tells us plainly what we need to do to ensure more conversions in our landing pages.

An optimized A/B testing strategy is the key to running a successful test. Your hypotheses could be substantial, have good results, and things could seem orderly, yet the test could be full of blunders.

Less emphasis on the execution of the experiments than the design could result in flawed analysis.

A proper method for running your A/B tests is key to successful optimization. Let’s examine three mistakes that could skew your test results.

Source: Pixabay

Three Things To Avoid in A/B Testing

If you don’t want to invalidate your test, you need to avoid these three things:

• Too many variations in the test
• Changing the settings of the experiment in the middle of the test
• Incorrect post-test segmentation

Your test should not have too many variations because:

1. More variations do not equal more insights.
2. More variations need more traffic and longer testing times.
3. Longer testing times can mean sample pollution.
4. More variations indicate a lower significance level.
5. More variations mean increased chances of a false positive.
6. The Bonferroni correction shows how multiple comparisons can skew data.
7. It is difficult to pin down the experiment variation that is the winner because of no statistically significant difference between the winner and runner up.

Don’t change the settings of the experiment in the middle of a test because:

1. Mid-experiment changes in control, variation design, test goals, experiment settings, or traffic allocation result in skewed test results.
2. Traffic split changes between variations in an experiment result in the Simpson’s Paradox, where trends in different data groups disappear when combined.
3. Mid-test changes in traffic allocation alter returning visitor sampling.

Do correct post-test segmentation

1. Don’t stop the test when you reach statistical significance.
2. Stop the test only on getting the calculated sample size.
3. Use a large enough sample size.
4. Use stratified sampling – divide the group into homogenous, mutually exclusive samples.

Conclusion

Get the most out of your A/B testing for high conversion optimization by:

1. Not changing the settings mid-test
2. Reaching the desired sample size before stopping the test.
3. Correcting your significance level for multiple variation tests if your testing tool doesn’t adjust for the multiple comparison problem
4. Comparing segments that are significant and have a large size