3 Steps to Experiments with Business Impact

Julie Ritchie — December 30, 2016

The Hotwire website serves millions of visitors per week, offering a lot of opportunities to experiment. For Hotwire, experimentation is more than just trying different button colors; it empowers their organization to learn about their customers and improve the product experience.

During her session at Opticon Virtual Summit, Experimenting for Maximum Business Impact, Josephine Foucher, Hotwire Product Manager, recommended focusing on three key pillars to help drive business impact from experimentation:

• A robust prioritization process based on your company’s strategy
  
• A backlog of diverse ideas from different sources
  
• Comprehensive analysis of multiple data sources
  
• Clear and constant communication around test results, insights and impact to the bottom-line

Prioritize Ideas with Your Company Strategy in Mind

It’s important to understand the organization’s strategy. While there are many ways to optimize a site, to deliver business value, experimentation needs to deliver on business goals. Spend the time to identify in which direction the company wants to grow and the most important challenges to achieving that goal. Then figure out how experimentation can help the company get there. Use tests to validate customer needs, estimate the business value and identify the risks. Doing so helps build a coherent and focused experimentation program, and ensures investments are allocated to the most impactful initiatives. To that end, make sure you understand:

• The competition: Who is the competition and how does the organization’s site compare?
  
• Industry best practices: What does the market leader do?
  
• Opportunities: Is anyone trying to disrupt the market?
  
• Your brand’s strengths and weaknesses: Where does your organization fit in past and present?
  
• 3-Year Vision: Where does your company see itself in 3 years? 10 years?

Not sure where to start? Identify and meet with an executive within your organization that has a vision for the company and insight into how it will get there over the next 3 years.

Get Everyone Involved in Experimentation

Now that you understand your company’s strategic priorities, the next step is to create a backlog of experiment ideas. When creating your backlog it’s important to pay attention to the quality, diversity and richness of the idea backlog. As the saying goes, “garbage in, garbage out.” In other words, without solid ideas, the experimentation program will not deliver value to the business.

To that end, engage the whole company to contribute test ideas. Leveraging ideas from marketing, design, and engineering teams can dramatically improve the quality and diversity of ideas. Facilitate idea submission by having an easy submission process that everybody is familiar with. Other possibilities are workshops, guided ideation sessions and competitions that incentivize people to submit ideas. These options can yield high-quality ideas, but it’s important to be mindful of people’s time, and close the loop to show the results of those efforts to maintain interest and momentum. Other ways to request people’s input include posters on tables, in bathrooms and elevators, or even in front of meeting rooms.

A good experimentation leader will be able to generate solid ideas without anyone else’s input, but a great experimentation leader will educate and leverage the whole company to generate a wide range of rich ideas.

Analyze Data to Pinpoint the Best Ideas

The next step is understanding the current state. Each company faces unique challenges, and knowing the organization’s website’s strengths and weaknesses is paramount to identifying opportunities. This step requires effort and certainly takes more than jotting down a “top 10 test ideas” list. However ideas that come from analytics and the voice of customers will drive much better win rates and lifts than a list of tips.

A good rule of thumb is to not start experimenting until the following have been clarified:

• Analytics – what does the data tell you about your customer’s journey?
  
• Funnel performance from entry to conversion, and drop-off rate at each step
  
• Error analysis – Where do site visitors struggle?
  
• Top customer behavior, and how they differ from average site visitors

Sources of data:

• Voice of customers – Employees tend to be biased so its best to call upon feedback from real users, whether gathered via the website, on social media, or through customer operations agents or online forums
  
• Surveys – This allows the organization to collect structured answers to specific questions rather than just get the viewpoint of frustrated customers
  
• Watch customers – Whether through field tests, labs or online testing tools, see how real users interact with the organization’s website for true insights

Note: Find the sweet spot when it comes to research; use your judgment and don’t get stuck in analysis paralysis.

Putting the Approach into Play

One of the biggest winners this year for Hotwire came from listening to how site visitors talk about the company’s Hot Rate hotels. Hot rate hotels are the flagship Hotwire product, where users save more by booking a hotel without knowing its name.

Hotwire’s naming convention used to only focus on the neighborhood, for example, “Fisherman’s Wharf area hotel”. However, most of the users that understood the Hotwire value proposition would add the star rating when talking about the hotels. This gave Hotwire the idea of changing the name to “4-star hotel in Fisherman’s Wharf area”.

This seemingly mundane change made a big impact, and the idea would never have come from inside since all employees understand Hot Rate hotels.

(63)