Speaker "Joan Xiao" Details Back



Transfer Learning for NLP


Transfer learning enables using pretrained deep neural networks trained on various large datasets and adapt them for various tasks. Fine-tuning such pre-trained models in computer vision has been a far more common practice than training from scratch. In NLP, however, due to the lack of models pretrained on large corpus, the most common transfer learning technique had been fine-tuning pretrained word embeddings. These embeddings are used as the first layer of the model on the new dataset, and still require training from scratch with large amounts of labeled data to obtain good performance. 
Finally, over the past year, several pretrained language models (ULMFiT, OpenAI GPT, BERT, etc) emerged.These models are trained on very large corpus, and enable robust transfer learning for fine-tuning many NLP tasks with little labeled data. 
In this talk we'll learn the architecture of these pretrained language models. In particular, we'll share how different transfer learning techniques have been used with BERT to solve various downstream tasks in the NLP community. 


Joan Xiao is a Lead Machine Learning Scientist at Figure Eight and focuses on machine learning applications for text analytics and Natural Language Processing. Previously she led data science teams building big data analytics solutions at companies of various sizes. Joan holds a Ph.D. in Mathematics and M.S.E in Computer Science from University of Pennsylvania.