Back to all posts Back

A/B Testing Ideation: Perfecting Your A/B Test Design


Like any tool, there are two ways to use an A/B testing platform: The right and the wrong way. You can either build up to a big feature change with incremental quick wins or you can commit random acts of testing without regard to your learnings or resources. To enjoy Netflix-levels of test success, you have to know a few things.

Here at Taplytics, we have three A/B test design principles:


Test to the funnel: Tests should map to your buyer journey. Every A/B test you design should aim to improve conversions at some point in the funnel, ideally, starting top-down. (Because, why optimize for purchases if nobody’s browsing the site yet?)

Start small: Start by testing quick wins that don’t require heavy design or engineering support. This helps you conserve company resources. You won’t go fixing things that aren’t broken, and you’ll build a case for enlisting those resources if test results do turn out positive. For example, if you increased purchases by moving buttons around the checkout page, you have good reason to believe a minor checkout redesign is in order. And if that works, perhaps a major one.

Share what you learn: Record what you learn in a test log. Share it with your team to multiply that knowledge throughout the organization, and so nobody duplicates your efforts.


What is A/B Testing in Digital Marketing?

A/B testing in digital marketing is a process for testing which variant of an image, feature, or button users like best. Testing can occur in the product itself, as with an in-app message, or out of the app, as is the case with emails or push notifications. The goal of testing is always to improve funnel conversions.

Thinking of some test ideas? Start by looking at a diagram of your marketing funnel.

Draw Your Funnel

Every company’s buyer journey is different, but most follow the common flow of stages from acquisition to activation, retention, revenue, and referral. Customize this to your own situation, and start thinking about the type of tests you’d like to run.

Funnel Graphic - AB Testing


Important: Focus on just one, two, or three stages to begin. Ideally, these are stages that roll directly up to your quarterly key performance indicator (KPI). This reduces noise—if everyone on the team is testing something different and not coordinating, it could create confusion. Starting small also prevents your team from feeling overwhelmed, and allows them to get in some quick wins, build confidence, and expand their testing program to other areas of the site or app.

Where to Begin Your Tests, and What to Measure

Acquisition: Most companies begin testing their acquisition. It’s the first and most important step because it fills the rest of the funnel. Without decent conversions here, there won’t be enough data or users to analyze at lower levels.

Key Metrics:

  • Sign-ups
  • Logins
  • Opt-ins


Activation: This is where ah-ha moments happen, and where users begin to explore the app or site’s features and build habits (both good and bad) that will impact the rest of their journey. It’s a good place to start if you have lots of traffic but low sign-up or purchase conversions.

Key Metrics:

  • Feature usage
  • Number of sessions
  • Time spent in app
  • ‘Favorites’ added


Conversion: The point where users actually make a purchase. Tests here typically focus on optimizing the checkout experience to be as easy, convenient, and secure as possible. This is a good place if lots of users aren’t completing their checkouts or sign-ups.

Key Metrics:

  • Items added to cart
  • Completed purchases,
  • Basket size
  • Average order value (AOV)
  • Time to purchase


Retention: At this stage, you should be focused on keeping users around (or bringing them back) after a sign-up, visit, or conversion. The higher your retention rate, the lower you cost of acquisition (CAC), because each user retained grows more and more valuable over their lifetime. This is a great place to test push notifications, which can reach users outside of the app to bring them back.

Key Metrics:

  • Number of sessions over X day period
  • Inactive users
  • Lifetime value (LTV)


Growth: This stage is about tuning your high-functioning app to make it even better. Say, by finding ways to encourage users to share it with their friends, buy more, or buy more often. This stage includes social sharing, referrals, and all manner of word-of-mouth advocacy initiatives.

Key Metrics:

  • Referrals
  • Social shares
  • Lifetime value (LTV)


Brainstorm Your Tests

Now that you know what stages you’ll focus your tests on, brainstorm the tests you can run to move your key metrics. Good testing relies on a diversity of ideas and creativity—the more you know about your buyer, how they think, and what motivates them, the better ideas you’ll generate.


Here are A/B testing examples:

Acquisition: Test app layout, reorder items in dropdown menus, copy, images
Activation: Test in-app message copy, button copy, button placement
Conversion: Test checkout layout, recommend items, payment options, rearrange form fields
Retention: Test email copy, push notification copy
Growth: Test push notifications, test sharing buttons, test recommendations


How Zappos Runs its Ideation Sessions:

The footwear brand Zappos is legendary for its customer service, but it also has a strong culture of experimentation. (Perhaps they’re related?) When the Zappos team ideates tests, they try to suspend judgement and get all their team’s ideas on paper, then decide how much effort each idea will take to support.


If they have a high-value idea that’s also high-effort, like, say, redesigning the mobile app’s checkout page, they oftentimes work backwards to ask, “How can we validate that notion with less effort in a quick win fashion?”


Instead of redesigning features, they move buttons around
Instead of launching a wish list feature, they rename one of their existing lists “wish list”
Instead of having a holiday-specific section on the site, just add it to the departments tab in the app


Then, when a quick win yields positive data, they dive deeper into that experiment and invest more resources from design and engineering. In the case of the wish list feature, they added holiday-specific categories in the departments tab and learned that it increased sales, so they launched more tests and eventually built it out as a feature.


How Do You Conduct a Multivariate Test? 

Like the name implies, a multivariate test tries multiple combinations of variables at once. For example, if you’re sending a push notification and have two images and two versions of the copy, you can run a multivariate test to determine which combination of them (four versions in total) performs best.


Interested in learning more? Read our “Ultimate Guide to Mobile A/B Testing” today .