A Guide to A/B Testing Implementation for Product Teams
If you want your A/B testing implementation to work, you have to introduce it in the right way—and the earlier the better. Get buy-in from all departments (especially the leadership), set company-wide vision, select the right software, and stick to a methodology until every team understands how A/B testing makes their work easier.
How to Implement an A/B Test
A/B testing implementation is simple: Draw out your funnel and pick a stage where you’d like to increase conversions. When in doubt, begin at the top. Brainstorm test ideas to improve conversions at that stage. Create an A/B test backlog. Prioritize those test ideas based on the effort-to-benefit ratio. Then, begin running tests.
A Five-Step Plan to Make Your A/B Testing Implementation Work:
1. Get Buy-In
A/B testing has the potential to benefit every single department in your company. Apps and sites that satisfy users and drive revenue are what makes the business grow. But A/B testing’s impact isn’t always clear, so you’ll probably need to get verbal buy-in from each department that they’re at least willing to give testing a try.
Gather case studies of how companies with similar products or in similar verticals hit goals you want to hit with A/B testing. Similarity is a critical factor. A best practice isn’t necessarily best for you unless you face a similar situation. If it’s a success story from an unrelated company, it can lead you astray, and this happens a lot. Plenty of companies have invested in retention efforts, for example, based on a famous statistic from Bain & Company. Few of them know that it’s based on a study with a sample size of one—a single credit card company that’s no longer in business.
Resources on A/B testing outcomes:
Sell the vision of A/B testing differently to each department, based on their interests. Marketing will want to know that it’ll increase top-of-funnel conversions. The product team will want to know that it’ll help you quantify product quality. The engineering team will want to know that it’ll lead to shorter development cycles and more exciting work.
2. Establish Goals and Vision
Companies don’t arrive at product perfection through random acts of testing. They get there by having a strategy replete with goals, vision, and an iterative testing plan to reduce risk. You can ensure that a culture of A/B testing takes hold in your organization by defining what you want the A/B testing program to accomplish before you purchase a software license.
Our foundational A/B testing principles:
- Test to the funnel: Every A/B test should improve at least one measurable conversion in your marketing funnel (or buyer journey). Teams that don’t test to the funnel waste time and resources improving things that may not matter to users or the company.
- Start small: Break big ideas down into small tests to validate them iteratively. Instead of launching a recommendation feature, first see if users enjoy using a “top picks” button.
- Share what you learn: Help your entire organization learn. Record knowledge in a test log so others within the organization commit fewer mistakes and make cumulatively better decisions.
Your vision for your A/B testing strategy should clearly describe the ideal state for your site or app. For instance, to become the number one gaming app in the iTunes app store, or to drive at least 10 percent of the entire company’s sales. That vision is a goalpost, and it doesn’t come from research—it comes from intuition and knowing your business.
Companies new to A/B testing often make the mistake of thinking A/B testing alone can solve high-level design questions in their product. It can help, but you won’t generate good testing ideas if you don’t have a product vision.”
– Matt Aranoff, Co-founder, Lead developer & UX designer, Logical Animal
Optimize for a North Star metric
The most precise indicators are lagging ones, like revenue. Be careful that you’ve chosen the right one to follow. The history of failed enterprises is replete with stories of smart people working exceptionally hard on the wrong goals. When in doubt, measure revenue, or a KPI tightly related to revenue, like sign-ups or net order value.
How far should you plan in advance for A/B testing?
Every company is different. Fast-moving mid-stage startups might be thinking 6 to 12 weeks in advance, whereas large enterprises think 6 to 12 months out. The ideal time horizon is not too short—you should know what you’re trying to achieve this month and the next–-but not so long that you can’t adapt. You need flexibility for your team to apply what they’re learning, otherwise they’ll spend all their time re-writing the plan, or worse—sticking to it after its outdated.
3. Purchase Software
Choose A/B testing software that supports all your must-have use cases and channels. For instance, will Amazon FireTV be important in the next year? Put it on the list. Will combining web visitor data with app user data for a full picture of user behavior be important? Add it.
Important features of an A/B testing platform:
- Runs A/B tests on most of your desired channels and platforms
- An intuitive interface that allows for coding, but doesn’t require it
- Easy to install SDK (or no-code install option)
- Breadth of integrations
- Flexible data export options
- Flexible segmentation
- Analytics and reporting tools
On top of that, you may want to add a data visualization platform, data repository, or additional testing tools.
4. Set Up and Install
A good A/B testing platform removes a lot of the work of installation. If the interface is intuitive, it requires less training and leaves less room for error. If the vendor’s professional services team is highly experienced, they’ll help handle the setup and tagging so it works right the first time around.
The A/B testing software implementation process:
- Schedule a kickoff call with your team
- Create a shared services project plan
- Establish roles: Who participates? Who owns the backlog? Who schedules check ins?
- Schedule a quarterly check-in with bi-weekly progress updates
- Analyze existing funnels and test results
- Implement the software and set up tracking
- Get started
How do I practice my AB test?
Always start your A/B tests with phased rollouts: Run the test with only 10 percent of the sample population to make sure that everything is installed and firing correctly. Then, roll it out to larger and larger samples. Phased rollouts give you the opportunity to correct errors before they go live.
5. Stick to the Methodology
Keep the spirit of your A/B testing methodology alive with recurring training sessions. For the first year of A/B testing, bring everyone back into a room with your most experienced testers a few times to ensure they’re sharp on the basics.
Celebrate Your Culture of Experimentation
Your culture of experimentation will take hold when the positive results of A/B testing are clear to all. When your tests produce rules and knowledge that product, marketing, and engineering teams come to rely on, it becomes a vital part of work—and actually works.
Start A/B testing fast with Taplytics — Learn more