Big Data Ingestion and monitoring

Big Data is everywhere and the need of the hour is to harness the data to make key business decisions. At Flipkart, the vision is to make the company completely data-centric and Flipkart Data Platform was born out of that vision. The Data Platform team handle ingestion, processing and consumption of data for all teams in Flipkart. In this presentation, I would like to talk about how we build our ingestion system which has seen a scale of upto billion ingestions per day and support ingestion of both real time and historical data which could be present in local file system or hadoop. Will also talk about monitoring build around this service.