STARCANADA 2018 Concurrent Session : Improve Testing of AI Systems with "Grey-Box" Testing Technique


Wednesday, October 17, 2018 - 3:00pm to 4:00pm

Improve Testing of AI Systems with "Grey-Box" Testing Technique

Add to calendar

There are two main challenges to testing systems that incorporate elements of artificial intelligence. First, the same input can trigger different responses as an AI system learns and adapts to new conditions, and second, it is difficult to understand what the correct response really should be. Such behavior violates one of the main principles of traditional testing: the repeatability of test case execution. It's like shooting a moving target and not knowing whether you missed. Testers lose confidence in the outcome of their testing when traditional approaches no longer apply. Yury Makedonov will share his experience in testing AI systems and explain how to improve the process by having direct access to the system’s state. Using example models of machine learning systems, Yury will visualize how the system state can be identified and used to make effective test AI systems while dealing with complex test data. You will learn how to model and apply “gray-box” testing techniques to a wide range of AI technologies, from simple machine learning systems to complex neural networks.

Yury Makedonov

Yury Makedonov was trained as a researcher and worked in a research and development institution dealing with composite materials. He has a Ph.D. degree in physics and math, though he is not a rocket scientist anymore; now he is using his skills and knowledge to improve software quality. Yury has more than twenty years of testing experience, from small startups to large companies and government organizations, and recently has been working as a QA manager, test manager, and consultant.