Back

Speaker "Stephen O'sullivan" Details Back

 

Topic

How I Learned to Stop Worrying and Love the Data Scientists

Abstract

Brief Abstract: So how much data engineering should a Data Scientist know? For a Data Scientist to get to the fun part of their job, they normally have to do a bit of data engineering. Like on boarding data. Do a little bit of “wrangling”. Before they get to the fun part part - The Data Science! In most cases this is 50%-80% of the time.

Detailed Abstract: Then comes the handing it over to the Data Engineering team to put it into production (of course via dev, test, and QA). This is when a “little bit” of contention happen. As in most cases the Data Engineering team will have to do “some” modification/re-write/Head shaking/Hand wringing to get the code to be production ready and meet the SLA’s defined by the business. As there is a disconnect in how Data Scientists and Data Engineers develop code / models (I get a front row seat to this all the time). In this talk I’ll take the Data Scientist on a journey. From on-boarding data, and how different data/object stores can help; Understanding and choosing the right data format for the data assets; Explore some different query engines, and some basic query tuning for each; Explain how a distributed streaming platform works, and how you can take advantage of it; Lastly cover some good coding practices. This will give the Data Scientist new skills to help them be more productive, so that can get to the fun part faster! Plus reduce the contention with the Data Engineering team, and make them say - “How I Learned to Stop Worrying and Love the Data Scientists”!

Profile

Stephen O’Sullivan ● What is their background? Stephen O’Sullivan is the owner of Data Whisperers. He is an expert in data architecture, infrastructure, and technical operations. Mr. O’Sullivan has deep experience in Hadoop usage and architecture and cutting-edge open source solutions for Big Data. He brings more than 20 years of experience creating enterprise applications and data management solutions for high availability and scale to his current position. Prior to Data Whisperers, Mr. O’Sullivan was VP, Engineering at Silicon Valley Data Science. Where is led the data engineering team to help SVDS clients become data driven, and obtain their business goals utilizing data. Prior to SVDS, he created and led the next generation data platform team at Walmart Labs as a senior director. He and his team architected and designed the data platform that will be used by all of Walmart's e-commerce business units. At Walmart Labs he spent time evaluating big data / database / datastore / data management vendors, from big name companies to stealth startups, as to how they would be used within Walmart’s eCommerce and store infrastructure. Mr. O’Sullivan evaluated, made recommendations and built solutions to address Walmart’s needs in security, high availability, scalability and performance. Prior to Walmart Labs, his tenure covers 3.5 years as director of system engineering at LiveOps running the company’s data, reporting and API teams; three years in data management for Yahoo’s search group; and eleven years as a senior engineer at Sun Microsystems.