Creating a testing culture in an organisation that is unfamiliar with the concept is not easy, especially in large companies. However, when you do eventually make an impact, which can be extremely rewarding, it can begin to snowball. And by this I mean lots of people getting wind of the benefits of testing and wanting a piece of the pie. It is fantastic to reach this point, but it can come with its own problems.
Depending on how you’re structured, very quickly you can have a queue of business owners wanting to run tests on their areas of the product. If you and your team are in charge of testing, then it is your responsibility to maintain law and order. It’s all well and good implementing a shiny new testing tool, but your job should not just be about becoming the gatekeeper for this tool — it is so much more than that.
It is very easy to get carried away and just launch a test when another one has finished, because you have a Business Owner bugging you, asking “when will my test be prioritised”. Before you know it, you have run a multitude of tests without any time to breath before or afterwards. This may not be the most effective way for you to run your testing program. Here is a few points to be aware of:
Testing Culture Maintenance
Build a roadmap
You hear this alot in the world of AB testing and MVT, but I think it is easy to get carried away and forget the importance of careful planning and maintenance. Ensure you have a roadmap on a single document or wiki page, which gives you a clear overview of all the tests you want to run over the next quarter or half year. This plan will not stay the same, it will evolve and grow as you start testing.
If you don’t maintain a roadmap, then before you know it, there will be no strategy and you will just be answering ad hoc requests from around the business.
Prioritise
Once you have built a roadmap with lots of test ideas, you then need to prioritise them. I find that a good way to do this is to rank your ideas in order of desirability and feasibility. It might be that the easier tests with quick, small wins is where you want to start. Or maybe you want to run with a test that has huge desirability and low feasibility, and you have the allocated resource to develop the experiences and launch the test. This is individual to you and your business, but the point is, you must prioritise your tests to maintain law and order
Iterate
Testing can raise as many questions as it answers, so It is not wise to just analyse the data, form a conclusion and move onto the next test. It may be that you don’t have a winning experience but there could still be some valuable insights you can take from the test in order to build out an iterative test, and help find answers. Or you might have a test result which is close to reaching statistical significance and needs a bit more exploring.
This would not be possible if you just jumped onto the next test because a Business Owner is banging on your door
Create traffic groups
When a user enters a test, the last thing you want to do is contaminate the data by allowing them into another test that’s running at the same time. This is where traffic groups are extremely necessary. All 3rd party testing tools have their own terminology and ways of creating traffic groups, but the outcome is the same — creating rules in the UI that says only this group of users are allowed to enter this test.
Providing your site has the traffic volumes; traffic groups allow you to run multiple tests at the same time. If you have enough traffic to the site, you could allocate a certain percentage of your traffic to those adhoc test requests. For example, a redesign that’s due to be pushed live (and needs to be tested) tomorrow, that nobody has communicated to you. This happens alot in some organisations, especially large ones. It’s difficult to get a handle on everything that is going on across the business, but ultimately the testing should become part of the design process, so it can be built into the roadmap and scheduled as best as possible. However, for when that doesn’t happen, having a dedicated traffic group could prove useful.
Hypothesise
When teams get the testing bug and they start chucking tests they’d like to run at you, they often forgot to hypothesise or they are unaware of the importance of forming a solid hypothesis before running an experiment. This is where strategy and best practices really need to be communicated internally. It might even be necessary to run a training session to explain why we need a solid hypothesis and how we can form our hypothesis.
In Conclusion
Ultimately, it is your responsibility to structure your plan so you can either accommodate those ad hoc “emergency” requests or have the processes in place for prioritising them amongst your other planned tests. The key to success is to maintain a fluid plan and ensure all areas of the business it impacts, are aware of it and your processes. You need to take charge, so you can be strategic about your test and learn programme. This is only going to benefit your team and the business in the long run.
Growth Lead at Dailymotion, Richard has a passion for improving user experience and ROI through data and experimentation.
1,212 thoughts on “How to Maintain A Testing Culture”
Comments are closed.