Implementing an ab testing tool is the painless step in embracing Conversion Rate Optimisation (CRO). Perhaps you have a new shiny ab testing tool implemented on your site and you don’t know where to start. Or maybe you’ve heard everyone else is doing it (not to mention the awesome benefits), and you want to kick off the conversation with your team/boss.

Whatever tool you have chosen or you are exploring, it is important to understand that it is just an enabler. Without people and process, the tool is useless. So here is a quick guide to Conversion Rate Optimisation (CRO).

What is AB Testing?

Serving multiple versions of a web page or experience at the same time. This allows you to:

  • Test hypothesis and new ideas
  • Quantify your decisions
  • Measure website performance
  • Provide insights about your visitors to feed into future features and designs

 

Why a Testing Culture is Important

Opinions are like noses…every one has one. Whether this is what features you should be developing or the look and layout of your platforms; ab testing challenges these opinions and lets your audience decide through their behaviour. By implementing the right processes you can continually optimise to make your product(s) better.

(For more detail please see my post How AB Testing and Product Development Work Together)

What to Test

Knowing where to start can be difficult and sometimes overwhelming. If you work on a website or mobile app that has thousands of pages/screens, where do you start? Firstly you should have some clearly identified (micro) KPIs for every page on your site, which impact your more high level (macro) KPIs, so you know which metrics to focus on page by page.

I always start by looking at the 2–3 highest volume landing pages and/or the top of a funnel. I measure the performance of the page(s) next to the relevant goals. How is the page performing? Could it do better? What elements of the page could be impacting the success of the page? From here you can begin to build out some test ideas.

Here is a few scenarios where you might want to run a test:

  • A Hunch — when there is an element on a platform that you believe isn’t working and you want to change it. Having a hunch is fine and is a good basis to run a test, but it’s important to back up your hunch with data, to make sure you’re ab testing the right page(s)
  • Data — when you spot a problem in the data. For example, you notice that a high drop off from step 1 of the sign up funnel. The data tells us WHAT is happening, but it doesn’t tell us WHY. This is where you create a hypothesis and then test it
  • A New Feature — when a new feature is developed, there should always be clear KPIs, so you know exactly what metric(s) you are trying to impact. You can use this KPI to run tests and let your users decide whether it is the right decision or not
  • Iteration — sometimes when you run a test you will see results you are unable to explain. This might create a new hypothesis which you might want to test further, to find answers

 

AB Testing Process

When running multiple tests, it can get messy. Here is 7 steps to consider to keep your ab testing program as effective as it can be:

1. Hypothesis

Every test MUST have a hypothesis, so you establish why you’re running the test in the first place and then it can be proved or disproved. A hypothesis should have a problem, solution and result

2. Design

  • Create variations
    Create as many variations for test as makes sense, based on the assumptions you have made in your hypotheses and the traffic volumes you have available
  • Tracking
    How are you going to track your goals? What custom events do you need to implement in your testing tool and/or analytics tool?
  • Audience Segments
    What segments do you want to test on? For example, is there an specific country or mobile device you want to target?
  • Approval
    Get internal approval for the changes

3. Build & QA

  • Using your preferred ab testing tool, build the test inside the UI.
  • QA every variation in all the major browsers

4. Launch

  • Push the test live

5. Analyse

  • Review the results of the test in your chosen ab testing or analytics tool
  • Don’t forget to segment your data. You may see zero uplift in the data when looking at the high level data, but when you segment by – for example – traffic source / device / country, you might see an uplift. Don’t treat every user the same

6. Action

  • Make recommendations off the back of the data:
  • Is there a clear winner? If so, should you roll out the new experience or run an iterative test to build on the success of the test and try to drive even greater uplift?
  • Is the test generating more questions than answers? This is ok. These questions may point to more assumptions that you want to test (even if you don’t see a clear winner, this could still apply)

7. Document

  • Record test results somewhere organised and accessible.Test results are another source of data insights, like some one logging into an analytics tool to find insights

Below is an infographic that helps to illustrates the process above for running ab tests, designed by Seb Wals

Summary

  • It needs planning and sound methodology
  • It needs a hypothesis, so you have something to prove or disprove
  • It should always be data driven
  • It needs clear KPIs, so you can clearly measure success
  • It should not have too many KPIs. Keep it simple
  • When thinking about success metrics, also think about whether your test could cannibalise other KPIs on the platform — these may be worth measuring as well
  • You can’t fail when testing — there is always insights
  • AB Testing must be statistically significant before any results can be acted upon

ab testing process

Growth Lead at Dailymotion, Richard has a passion for improving user experience and ROI through data and experimentation.

262 thoughts on “AB Testing: A Quick Guide for Getting Started

  1. Pingback: viagra 5mg price
  2. Pingback: cialis 20 mg price
  3. Pingback: cialis 5 mg
  4. Pingback: how much is cialis
  5. Pingback: goodrx cialis
  6. Pingback: viagra for sale
  7. Pingback: ed pills
  8. Pingback: ed pills
  9. Pingback: sale cialis
  10. Pingback: Cialis in usa
  11. Pingback: vardenafil usa
  12. Pingback: order vardenafil
  13. Pingback: generic sildenafil
  14. Pingback: personal loan
  15. Pingback: instant loans
  16. Pingback: real money casino
  17. Pingback: instant loans
  18. Pingback: viagra 100mg
  19. Pingback: cialis 5 mg
  20. Pingback: generic for cialis
  21. Pingback: cialis 20
  22. Pingback: cialis 20
  23. Pingback: chumba casino
  24. Pingback: viagra for women
  25. Pingback: sildenafil dosage
  26. Pingback: viagra pills
  27. Pingback: generic for viagra
  28. Pingback: viagra sample
  29. Pingback: brand viagra
  30. Pingback: buy cialis
  31. Pingback: viagra canada
  32. Pingback: best ed medication
  33. Pingback: buy cheap viagra
  34. Pingback: jueriy.com
  35. Pingback: order Viagra 200mg
  36. Pingback: buy cialis online
  37. Pingback: Viagra 200mg nz
  38. Pingback: Cialis 40 mg uk
  39. Pingback: cialis prices
  40. Pingback: buy Cialis 20mg
  41. Pingback: Cialis 10 mg nz
  42. Pingback: buy Cialis 40mg
  43. Pingback: viagra coupons
  44. Pingback: tadalafil 20mg nz
  45. Pingback: viagra price
  46. Pingback: amaryl 1 mg usa
  47. Pingback: online viagra
  48. Pingback: cialistodo.com
  49. Pingback: buy cialis toronto
  50. Pingback: benicar 20mg otc
  51. Pingback: cialis dosage
  52. Pingback: buspar 10mg uk
  53. Pingback: buy cialis bangkok
  54. Pingback: tadalafil generic
  55. Pingback: viagra sold in usa
  56. Pingback: casodex tablets
  57. Pingback: buy ceclor
  58. Pingback: celebrex 100 mg nz
  59. Pingback: celexa nz
  60. Pingback: casino game
  61. Pingback: rivers casino
  62. Pingback: gambling casino
  63. Pingback: best casino online
  64. Pingback: play casino
  65. Pingback: casinos online
  66. Pingback: accc car insurance
  67. Pingback: cialis 20mg pills
  68. Pingback: payday loan today
  69. Pingback: tadalafil cialis
  70. Pingback: quick loans
  71. Pingback: cialis 10mg
  72. Pingback: hims viagra
  73. Pingback: 5mg cialis
  74. Pingback: viagra boner
  75. Pingback: how to take viagra
  76. Pingback: buy cialis canada
  77. Pingback: tadalafil 5 mg
  78. Pingback: viagra women
  79. Pingback: viagra costs
  80. Pingback: online viagra
  81. Pingback: cbd massage oil
  82. Pingback: viagra and alcohol
  83. Pingback: cost of cialis
  84. Pingback: Discount viagra
  85. Pingback: viagra twitter
  86. Pingback: atorvastatin and
  87. Pingback: buy essay
  88. Pingback: clomid prices
  89. Pingback: clonidine pills
  90. Pingback: Testogen
  91. Pingback: Buy cheap viagra
  92. Pingback: female viagra sex
  93. Pingback: tadalafil
  94. Pingback: cialis price india
  95. Pingback: viagra prices
  96. Pingback: cialis online
  97. Pingback: cheap cozaar 100mg
  98. Pingback: crestor 5mg cheap
  99. Pingback: thesis editing
  100. Pingback: write my paper
  101. Pingback: cost of ddavp
  102. Pingback: doxycycline otc
  103. Pingback: elavil tablet
  104. Pingback: Testing60
  105. Pingback: flomax 0,2mg pills

Comments are closed.