Testing Uncertainty—and a Chatbot Named Ginger
Uncertainty has always been a key challenge for testers. But testing a chatbot adds a completely new level of uncertainty. There are a lot of platforms and tools available for chatbot development, but what we lack is a standardized chatbot testing strategy. The way testing is performed on chatbots differs a lot from "traditional" testing (like for an app or web portal) due to the apparent randomness of a conversation with a chatbot. From testing numerous clients' chatbots and her company's own, named Ginger, Rajni Singh has experienced that it is impossible to anticipate all the situations that can happen during a conversation. As they introduced learning components to Ginger such as machine learning and intent training, the chatbot evolved and changed its behavior compared to previous test runs. This increased the need for regression tests and complicated them at the same time. Rajni will talk about the challenges faced during chatbot testing and how to mitigate them with different strategies. She'll share her experience using their own advanced automation framework as well as commercial and open source tools.