Many are talking about growth and product analytics, but how many well established companies are leaving behind their old ways of design by committee? In this post I want to highlight a common scenario and promote the modern approach to product development.
Once upon a time…
I received a heads-up from a UX team about an AB test they wanted to run for a new page they had designed. My first reaction was “great, you don’t want to just roll out a new page, you want to make sure it has the desired outcome.”
So what was the desired outcome? By digging a bit deeper, I discovered that…erm, there wasn’t one. The “KPIs” had been plucked out of thin air and they were not relevant to the page in question.
For example, one of the success metrics for the test was bounce rate. This suggests there was a high bounce rate to begin with, which in-turn prompted the redesign. But this page is rarely an entry page, so why were the team redesigning a page to reduce bounce rate?
It occurred to me that the team had no idea why they had created this new page and had no clear objectives for the redesign. They were trying to solve a problem that didn’t exist…or did it?
The page in question, was the top of a funnel. By digging into the data, we discovered that the drop off rate from the top of the funnel to the bottom was significant.
Funnel
Ok, so now we have a problem to solve. And what a great place to start optimising — at the top!
So why were certain decisions made for the redesign?
In my experience this example of product development is not uncommon. Its the difference between having HIPPO (highest paid person’s opinion) led decisions, and data led decisions.
It is a deeply routed cultural issue in many organisations, where areas of the business lose sight of why they are creating something, especially when it involves jumping into a web analytics tool they don’t know how to use, and putting opinions (and sometimes egos) to one side, by listening to the data.
I’m sure if I asked you to post some thing on Facebook that will receive lots of ‘likes’, you’d know what to post. This is because you have learned over time what resonates with your friends, by looking at the numbers. Ok, this is a very basic example, but my point is data is now so deeply engrained in our every day lives — sometimes without even realising it.
The New Approach to Product Development
This is why multi disciplined growth teams (designers, analysts, developers…) are now being formed across the tech industry — to fulfil a desperate need to understand audiences and make smarter decisions.
The importance of a correct web analytics implementation and test & learn function cannot be understated. Without them, product teams are just shooting into the wind, in the hope that something sticks, without ever really knowing if it does in-fact stick.
Going back to my original example, of course there is nothing wrong with testing a new design just because you feel a particular page needs a refresh, as long as every one remains focused on why they are changing/updating certain elements of the experience and what metric(s) they are trying to shift.
Maximise the impact of your changes
If you launch a test where you have changed multiple elements on a page, without any prior iterative testing, then the chances are you won’t see the positive results you’d hoped for. And if you are fortunate enough to see an uplift you will have no way of knowing which changes have been the drivers.
It’s a great headline to share with the business: “We tested this new shiny page and we saw an increase of 2% in sign ins”. High fives all round… But how do you know which element of the new design drove this uplift? Was it the new title/new CTA on the button/larger fields/more emotive images…? These are all valuable questions you could and should have answers to.
Your web analytics data gives you insights into how your users are behaving on your site or app. Are they behaving in the way you intended? If not, then you have identified a problem. Make some assumptions and test them.
Continuous experimentation cycle
So if your team wishes to refresh a page / product development, break down the various elements you think will impact your specific KPI for that page, and test them individually. Slowly you will gather valuable insights that will feed into your redesign as well as future changes. With a more iterative approach you are likely to see even bigger wins.
Finally
Whether you’re a designer, developer, analyst or marketer, we are all striving to see our companies succeed.
In order to succeed we need sustainable growth. To achieve sustainable growth we need to understand our audience. To understand our audience we need data.
Key takeouts for upstream product development testing
- Know why you are redesigning a page / an experience — what problem are you trying to solve?
- Have data to back up your new design
- Test whatever you want to test, but know what you are testing and why
- For gathering insights, break the elements of the page down into bite size pieces
- Make assumptions with a sound hypothesis
- Treat your testing program like any other analytics tool. Use it in conjunction with your web analytics, to find answers quickly, to important questions. Testing tools are effectively web analytics tools
For further reading see my infographic 7 Steps for Effective A/B Testing
Feature image Designed by Freepik
Growth Lead at Dailymotion, Richard has a passion for improving user experience and ROI through data and experimentation.
2,305 thoughts on “How AB Testing and Product Development Work Together”
Comments are closed.