Evaluation of Algorithms Using Online and Offline Controlled Experiments.

The online controlled experiments or A/B tests have become an essential scientific tool for decision making for the large web-based companies. However, conducting and evaluating A/B tests require the experimenters to have a deep understanding with respect to the biases present in the data, statistical nature of the metrics and the tests, and different aspects of the results. However, online tests are not always feasible. In those cases, an experimenter is required to evaluate the algorithms offline by conducting simulation studies using historical data. Online and offline evaluations require similar statistical understanding and are very related. In this talk, I will discuss these two paradigms of statistical evaluations of algorithms and provide some guidelines based on my experiences.

Anjan currently heads the development of a question and answering engine for Salesforce. He was formerly the Head of Search Science and Engineering at WalmartLabs and head of data science at Elance (now Upwork). He holds a Ph.D. degree from University of California, Davis and his research area is machine learning. He also runs a consulting company and has in the past advised e-commerce firms for improving their search and recommendation functionalities, and data science strategies and helped them improve their customer experience and revenue.