Distributed Deep Learning At Scale on Apache Spark with BigDL

BigDL, an open source distributed deep learning framework for Apache Spark, brings native support for deep learning functionalities to Spark, providing an orders-of-magnitude speedup over out-of-the-box open source DL frameworks, such as Caffe, Torch, or TensorFlow, with regard to single node Xeon performance, and efficiently scales out deep learning workloads based on the Spark architecture; in addition, it allows data scientists to perform distributed deep learning analysis on big data using familiar tools, including Python. In this talk, we will give an introduction to BigDL, show how Big Data users and data scientist can leverage BigDL for their deep learning (such as image recognition, object detection, NLP, etc.) analysis on large amounts of data in a distributed fashion, which allows them to use their Big Data (e.g., Apache Hadoop and Spark) cluster as the unified data analytics platform for data storage, data processing and mining, feature engineering, traditional (non-deep) machine learning, and deep learning workloads.

Jiao (Jennie) Wang is a software engineer on the Big Data Technology team at Intel working in the area of big data analytics. She is engaged in developing and optimizing distributed deep learning framework on Apache Spark.