Top Practical Challenges in Running Online Controlled Experiments at Scale in Industry

Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale and encourage further academic and industrial exploration. To understand the top practical challenges in running OCEs at scale, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018. In the talk I will summarize the top challenges faced across the industry for running OCEs at scale and some common solutions.
Who is this presentation for?
Data Scientists, managers and leaders looking to build a culture to make trustworthy data driven ship decisions
Prerequisite knowledge:

What you'll learn?

Somit Gupta is a Senior Data Scientist for Microsoft’s Analysis and Experimentation team. He helps MSN, edge browser, Office and Windows to innovate faster with trustworthy experimentation. He helps the team run experiments at scale: defining the OEC for the product, design & monitoring of experiments, and interpretation of results to make a ship decision. He co-lectured a tutorial on A/B Testing at KDD 2017. Previously he was a product manager for Windows and Skype. Somit received a master’s degree in Computer Algebra from University of Waterloo in 2011. Prior to that he received a bachelor’s degree in Computer Engineering from National Institute of Technology, Surathkal, India.