Minimize Potential Website Launch Risks by Testing New Features and Pages First

Lisa Frank

Imagine this scenario: You help run the website for a successful e-commerce company. You listened to your customers and redesigned a key feature on your site. Based on their feedback, you thought that streamlining the search process would lead to increased conversions. But you were wrong and conversions were flat. What happened?

This scenario happened recently to one of our customers. The company received feedback that it was difficult to use the search functionality when accessing the site from mobile devices. To remove this roadblock, the web team redesigned the functionality to meet the needs of the mobile user. In this instance, they condensed a multi-field search engine into a single search field. After several iterations, the design was ready to launch and the redesigned search was live on their site.

Eager to understand the impact of the new design, the testing team created a test and collected results. The top level results indicated that conversion rates were flat and the redesign appeared to have no impact. Unsure why the results were flat, the team dug deeper into the results.

The testing team focused on how several key segments performed and the results that initially appeared to be flat were actually three sets of segment results that offset each other. In particular, the team looked at their results for desktop, tablet, and smartphone visitors. The new design worked well for mobile users and did, in fact, increase conversions for tablets (+12%) and smartphones (+53%). However, the redesign had a negative impact for desktop users, with a decrease in conversions of (-4%).

On the surface, the 4% decline in conversions for desktop users seems negligible. For this particular customer, however, the desktop still accounts for 88% of site visits. The decline in conversions quickly wiped out any gains received from mobile device traffic and would soon trend negatively due to the high volume of traffic. The test team needed to react fast.

After some brainstorming, the testing team devised a plan to revert the desktop search functionality to the initial design while continuing to serve up the new design to mobile users. Had the customer discounted the early results, then the overall negative impact would have been much worse. But they trusted their instincts and discovered the reason behind their results.

Test, Analyze and Act Quickly

Here are a few key takeaways from this real world example that can help you minimize risk:

1. Make it a goal to always test new or redesigned features prior to launch. What appears to be, on the surface, a relatively minor change may have a significant impact on your conversions.
   
2. Dig deep when reviewing test results. In this case, the team analyzed the segments targeted to discover an unexpected result.
   
3. React quickly and confidently based on your data. The testing team at this customer recognized the problem, devised a plan to fix it, and took steps to re-implement the feature design based on whether traffic originated from a desktop or mobile device.

Continual testing is critical to creating a successful multichannel, online presence that continues to thrive – converting browsers to long term customers. If your development and launch schedule doesn’t build in time to test features prior to launch the actual results post-launch may fall well short of the business goals your CEO has set. The customer in this scenario quickly learned that any gains they might reap from the mobile channel would be quickly wiped out by the decrease in website conversions. Testing allowed this customer to switch gears and let the data speak for itself, circumventing a loss in customers, conversions, and ultimately revenues.

(103)