We tested AI shopping experiences with 15M+ consumers, here's what we found

‍The Current State of Conversational AI in E-commerce

When OpenAI rolled out ChatGPT to the public in 2022, online retailers jumped at the opportunity to launch these Conversational AI experiences for their businesses. These experiences took shape in different forms creating a rush to see who could incorporate an “AI something” first.

When we started building alby in 2023, our goal was to understand what was actually working for AI in e-commerce. We conducted extensive A/B testing to explore how shoppers engage with different types of Conversational AI experiences. Our goal was two-fold: 

• Discover what AI-powered experience drives the most value for shoppers 
• Build a solution that empowers retailers to use AI in a way that drives real business results.

Below, we examine the results of these tests conducted with over 15 million consumers.

The Chatbot: What’s Working and What’s Not

AI chatbots are now a common feature on nearly every e-commerce site. They typically take form in a pop-up icon in the bottom right corner of the page and may prompt shoppers with a question like “how can I help?”. 

Many chatbots are designed as extensions of customer support teams, answering questions about order status and providing technical support. Recently, some chatbots have rebranded themselves as "shopping assistants," claiming to help customers make shopping decisions and drive sales. However, their effectiveness varies widely.

The Good

• Effective Post-Order Support - Chatbots provide instant responses to common post-order queries, such as “What is my order status?”, ensuring customers receive timely information.
• Decreased Support Burden - We saw chatbots drive significant support ticket deflection rate of 60%, reducing the workload on customer service teams.

The Bad

• Low Engagement - Very few people choose to interact with chatbots—only 0.3% of website traffic engages with chatbots. 
• No Revenue Lift - Chatbots do not contribute to a noticeable increase in revenue. Their limited ability to drive conversions and upsell products means they fall short in boosting the bottom line.
• Poor Customer Experience - Many customers perceive chatbots as impersonal and frustrating. We observed a Customer Satisfaction Score (CSAT) of 55%.

“Click-to-Prompt” Buttons: An Attempt to Drive Pre-Order Engagement 

alby’s next experiment was with “Click-to-Prompt” buttons. This experience featured clickable buttons that would prompt customers to ask product-specific questions, then initiate a chatbot. The hypothesis was that if customers were able to ask specific product questions, then they would feel more confident making a purchase. 

We tested 2 placements of these "Click-to-Prompt" buttons: 

‍Top of Search Page Performance

• Doubled engagement rate - engagement rate increased to 0.7%
• Small revenue lift - revenue per session saw a small lift

On Product Performance 

• Small dip in engagement rate - engagement rate dropped to 0.2%
• Huge lift in conversions - drove a 30% increase in conversion rate

Through this set of experiments, alby was able to identify which experiences drove better engagement as well as conversion rates. We also discovered the types of questions customers were asking before placing orders. Equipped with this data, we defined the customer problem to scope a better solution.

Defining the Problem: An Information Issue  

By analyzing the qualitative data from experimentation, we saw a wide range of questions that shoppers wanted answered about products. We concluded that there was a significant issue in the shopping funnel: customers need a lot of information when shopping and there is no easy way to find it. 

Customer Information Issues

• Crucial information often resides deep within articles and product pages, making it hard to access.
• Most customer questions are not addressed in community FAQs.
• Questions vary significantly by product and customer profile.
• Different questions have varying impacts on conversion rates.

The Winning Experience: Generative Q&A

Equipped with data and a defined problem, alby starting building a solution that would help provide customers find the information they were looking for in a form that they would engage with. 

alby’s Generative Q&A Engine

alby is an AI-powered shopping experience that predicts and answers customer questions directly on product pages (we call it ✨Generative Q&A✨). With alby, retailers can personalize the shopping journey, understanding the specific needs of customers to tailor responses and recommendations. 

An Embedded Product Experience 

Understanding that an amazing chatbot is useless if no one engages with it, alby developed a more intuitive experience for shoppers. alby’s Generative Q&A is embedded directly on product pages so that it is visible and customers know it is there to answer pre-order questions. 

An AI Shopping Assistant that Actually Sells

alby’s Generative Q&A is a powerful tool for driving sales and enhancing the shopping experience. alby creates real value both for retailers and shoppers: 

• Engagement Rate - 14x increase compared to traditional chatbots.
• Conversion Rate - 2x versus traditional chatbots.
• Revenue Lift per Session - 2.15% increase.
• CSAT Score - 86%, significantly higher than the 55% with traditional chatbots.

Key Takeaways

AI Chatbots are Often Ignored:

Standard chatbots frequently fail to effectively engage customers. While they can be useful for post-order support, they rarely contribute to driving more sales on your website.

Effective Shopping Assistants Drive Pre-Order Engagement:

To be truly valuable, shopping assistants need to proactively address a wide variety of pre-order questions, helping customers make informed purchasing decisions and guiding them through their shopping journey. 

alby's Generative Q&A Boosts Engagement and Sales

alby’s Generative Q&A predicts and answers customer questions that drive conversions. We've determined that embedding alby directly on product pages creates an intuitive experience that maximizes customer engagement and satisfaction.‍‍