3 Tips to Improve Your A/B Testing Process

In the long-term, your A/B testing process is one of the most important factors driving performance of your conversion rate optimization efforts.

I want to share three tips that will make your A/B testing process more impactful. We use these techniques on daily basis to convince companies to invest in CRO, spot real customer problems, and run long-term optimization programs more efficiently.

The three techniques described below enabled us to finish the process quickly, bring conversion rate uplift, and keep ourselves well-oriented through the multitude of tests.

1. Start testing as soon as possible

The problem we’ve struggled with in the past was over-analyzing data before running the first A/B test for a particular website. It was taking us two months on average before we could show results of our work. Sometimes after this long time we delivered conversion rate drop – you can imagine how disappointed was the client. The same applies to your boss, who invests money in CRO.

Your goal should be to start a test as soon as possible, because every hour you lose traffic you could have tested. Not to mention the opportunity cost of running less performing version of your website.

Goal flow report in Google Analytics is one of the places where you should head first in order to quickly find elements worth testing.

In many cases you can spot severe problems with your website quite fast. A look into standard Google Analytics reports should give you a couple of ideas worth testing. Landing Pages, Goal Flow and Funnel Visualization reports are good places to start looking for low-hanging fruits.

Use guerrilla user testing to identify most cumbersome usability issues with your website. Show your site to someone who is not working with it on daily basis (it could be your friend or even a random stranger in the nearest coffee shop) and ask him to complete the most important user scenario.

Another way to gather ideas for A/B tests is interviewing your sales and customer success departments to learn about your customers’ most frequently asked questions. We’ll talk more about this in tip #2.

Remember, don’t over-analyze in the beginning. Try these ideas to come up with a few quick ideas and start your test. It’s the most effective way to use your time when you’re starting conversion optimization process.

2. Do interviews with sales and customer care departments

We are an agency, so client interviews at the beginning of a project are standard protocol for us. If you are a part of an internal conversion optimization team though, it’s easy to think that you know everything about your customers. Be careful! I know a lot of marketing specialists that were completely detached from reality before they spoke with customer care department (CC).

You should constantly keep in touch with members of sales and CC departments. They talk with customers on daily basis and have the most knowledge about their fears, doubts, problems, needs, and motivations. This knowledge is the foundation of effective conversion optimization.

Here is a list of questions I always ask sales / customer care specialists:

• What are the reasons customers look for a product like this?
  
• What are the expectations of end effect of using the product?
  
• What are the main doubts about the product?
  
• How do you tackle these doubts? Which answers work best?
  
• What features and details are most important for customers?
  
• Which emotions do people express when talking about the product?
  
• Which incentives work best on customers?
  
• What problems are most common?
  
• How do you respond to these problems?

Why do we put emphasis on this particular part of analysis? This story explains it: We were working for a dating site. A customer care specialist told us that users often asked about how the payment for the dating service would appear on their bank statement. Why? Apparently people felt nervous that someone would be able to see they were using dating site by looking at their account history. It’s an abnormal doubt, but a crucial one because it concerns users who are about to convert. Fixing it was an important step towards higher profits.

3. Keep track of your A/B testing

You should keep a record of every single A/B test. You may not think about it in the beginning of your A/B testing process, but the more tests you conduct, the longer the list of lessons becomes.

With this A/B testing record on hand, you can look back and quickly check what you have optimized before and the impact. Without one, you may end up forgetting what you’ve tested and find yourself testing the same hypothesis after a couple of months.

It’s an example of an A/B test note, that we use to keep track of our conversion optimization activity. The one above describes a test of removing unnecessary elements from product page for traffic coming from price comparison engines. We ran it on awa24.pl an e-commerce site for electronics.

It’s also an invaluable asset to have when you want to add another person to the conversion optimization team. You simply grant the new teammate access to the spreadsheet or wherever you’re keeping your record, and they can instantly get up to speed on the historical testing and results.

An A/B testing record should include:

• domain, number of test, dates
  
• test hypothesis (+ metrics you will use to verify them)
  
• result
  
• screenshots of original website and all the challengers (mark your treatments)
  
• test analysis
  
• lessons learned from the test (what can you improve in future tests?)

I also encourage you to create additional spreadsheet where you can list lessons learned from all the tests. It’s a valuable resource to recap your experience, as well as for new team members to get on board effectively.

After working with dozens of clients we’ve learned, that these three tips stand out as the most helpful tactics for a solid conversion optimization process. Start with basic analysis to get your first A/B test up and running quickly, spend time with your customer success and sales team to really learn the problems customers face, and keep a track record of your tests. Committing to these steps will help you secure executive buy-in, run more successful experiments and build a culture of continuous improvement.

(54)