Flyte: Cloud native machine learning & data processing platform

Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous etc. At its core it is a Kubernetes native workflow engine that executes 10M+ containers per month as part of thousands of workflows. Flyte abstracts complex infrastructure management from its users and provides a declarative fabric to connect disparate compute technologies. This increases productivity and thus product velocity by enabling them to focus on business logic. Flyte has made it possible to build higher-level platforms at Lyft, further reducing the barriers to entry for non-infrastructure engineers. The talk will focus on: Motivation and tenets for building Flyte Architecture of Flyte and its specification language to orchestrate compute and manage data flow across disparate systems like Spark, Flink, Tensorflow, Hive etc. Deploying highly scalable and fault tolerant Kubernetes Operators Use-cases where Flyte can be leveraged Extensibility of the Flyte and the burgeoning ecosystem. The talk will conclude with a demo of a machine learning pipeline built using the open source version of Flyte.
Who is this presentation for?
Software engineers, Data engineers, Data scientists, Data platform architects, Machine Learning platform architects
Prerequisite knowledge:
Kubernetes, Containers
What you'll learn?
Challenges faced in creating highly scalable and production ready portable machine learning and ETL pipelines.

Ketan Umare is a senior staff software engineer at Lyft and founder of the Flyte project. Before Flyte he worked on ETA, routing and mapping infrastructure at Lyft. He is also the founder of Flink Kubernetes operator and contributor to Spark on K8s. Prior to Lyft he was a founding member of Oracle Baremetal Cloud and lead teams building Elastic Block Storage. Prior to that, he started and lead multiple teams in Maps and Transportation optimization infrastructure at Amazon. He received his Masters in Computer Science from Georgia Tech specializing in High-performance computing and his Bachelors in Engineering in Computer Science from VJTI Mumbai.