Sean Oliver
A/B testing has long been a critical driver of growth on the web, but its role in mobile app development is still beginning to take shape. It has been less than a decade since Apple first introduced us to the iPhone and induced the massive proliferation of mobile apps and the huge community of mobile app developers who support them. It’s easy to forget that the first modern mobile app came out only seven years ago, and many of the mobile app teams today are still trying to figure out the perfect formula to acquire, retain, and monetize their users.
One of the tools app developers and mobile product managers are increasingly adding to their toolkits is A/B testing. Having a discipline around A/B testing enables mobile app makers to deliver engaging, high-ROI experiences by optimizing elements like signup flows, checkout funnels, and navigational elements, to name a few.
Here are three simple yet effective tips for those just getting started with mobile A/B testing from the mobile product managers who own testing at Pinterest, CheapOAir, What to Expect, and Credit Sesame. (These three companies spoke on a panel at Opticon. You can watch the video recording at the end of this post or here along with the rest of the sessions.)
1. Deconstruct your funnel to build your plan.
kyle humphries
“Map out all of the micro-interactions a user must complete as part of a funnel or flow in your app. Then use use that to plan out which elements to test first.”
– Kyle Humphries, Senior Product Manager at Everyday Health (What to Expect)
One of the main reasons mobile product managers (PMs) are not currently running a testing program is that they simply don’t know where to begin. There is a vast universe of things to test in any mobile app, and it isn’t always obvious which of those things will yield the greatest impact.
Kyle Humphries from Everyday Health, the company that makes apps such as What to Expect, Calorie Counter and Diabetes in Check, has a strong recommendation for every first-time mobile A/B tester: Map out your funnel in excruciating detail. Most apps have some type of funnel or flow that is important to their key business metrics such as a checkout funnel, a signup flow, or some other type of multistage interaction.
Once the stages of your funnel are mapped out, you can determine the conversion rate at each stage, which provides an immediate insight into which stage of the app is the most important to focus on. Then, you can use this list to plan out subsequent tests, until the whole flow has been optimized.
2. Use testing to reduce release risk.
Raman Bhatia
“A/B testing allows us to try things out in our app without hurting our conversion funnel. Nothing goes out without an A/B test.”
– Raman Bhatia, Director of Mobile at Fareportal (CheapOAir)
Any mobile app developer will tell you that releasing new features in an app or changing aspects of the underlying architecture can be risky business. Apps are different from websites in that they are compiled programs installed locally on a user’s device whereas a website lives forever in the cloud and is always up to date with the latest version.
This means the stakes for app updates are high and PMs should do everything possible to ensure they are high quality. If something isn’t caught, an app update could include bugs or interaction issues that damage the user experience or unexpectedly tank business metrics.
Andrea Burbank, Data Scientist at Pinterest, advises that testing to ensure that you’re not breaking anything is just as important as testing for improvements to business goals. A/B testing allows you to feature flag new functionality and then gradually roll those features out to your users. If a negative impact is observed, you can turn off the test and revert everyone back to the original experience before things get out of hand.
If you push a buggy update, the only resolution is pushing out another update and waiting patiently while users upgrade. This process can take weeks, and in the meantime, users may uninstall the app out of frustration and leave negative reviews which hurt future installs. It’s a nightmare scenario that is solvable with A/B testing.
Even if you haven’t yet mapped out a testing plan, every mobile app team should be thinking about how they can mitigate the risk of new releases through feature flagging and phased rollouts.
3. Don’t lose sight of the long term.
Andrea Burbank
“We make sure to track the impact of our tests over time. The question is not, ‘Will they convert right away?’ but ‘Will they continue to convert in one, two, three months from now?’”
– Andrea Burbank, Data Scientist at Pinterest
Humans are creatures of habit, so whenever an experience we’re already familiar with changes, we react. There’s a period of time during which we need to adjust to the new normal. Depending on what that change is, it can take us anywhere from a few seconds to a few weeks or even longer to determine a new habit that works in this new environment.
When you change the experience in your app, especially if that change is a large one, there is typically a big positive or negative spike followed by a period of time where the results stabilize to some less extreme value.
Advanced testers, such as Andrea Burbank at Pinterest, recommend a longitudinal approach to testing to eliminate these changes over time. Even after a test has achieved statistical significance and the winning variation has rolled out, Burbank’s team will maintain a holdback for a small percentage of traffic so it always sees the original variation.
Implementing a holdback not only allows you to optimize for retention over time, it can also help alert you to new testing opportunities. For example if the conversion rate for a winning variation begins to converge with the conversion rate for the original, it might be time for your team to consider running a new test on this feature or experience.
Watch the full recording
If you couldn’t make it to Opticon and want to learn more from these mobile testing experts, check out the full recording below:
(164)
