Back

Speaker "Maxime Beauchemin" Details Back

 

Topic

Airflow - An open source workflow engine


Abstract

Airflow - An open source workflow engine Company DAGs (directed acyclic graphs) of jobs are evolving rapidly, and orchestrating and monitoring jobs inside and outside of Hadoop has been a growing challenge for all data driven companies. Airflow is a new platform developed at Airbnb that allows data workers to programmatically author data pipelines. It offers an extensible set of operators that can be used to orchestrate workflows across many data systems as well as a rich user interface to monitor and troubleshoot complex DAGs. Airflow is the future of complex batch processing for companies with complex analysis needs.

Profile

Maxime Beauchemin recently joined Airbnb as a Data Engineer developing tools to help streamline and automate ETL processes. He mastered his data warehousing fundamentals while at Ubisoft and was an early adopter of Hadoop/PIG while at Yahoo! in 2007. More recently at Facebook he developed "analytics as a service" frameworks around engagement and growth metrics computation, anomaly detection, and cohort analysis. In his free time, he`s a father of 3 and a digital artist. More about his projects on his blog http://mistercrunch.blogspot.com/