Speaker "Premal Shah" Details Back



Monitoring and Troubleshooting Real-time Data Pipeline


Enterprise​s​ are rapidly adopting big data frameworks to process and understand large-scale data in real time. Technologies like Apache Kafka, Apache Spark, Apache Storm and Elastic​​search are increasingly pushing the envelope on what is possible. However, while these technologies are quickly maturing, their extremely distributed and complex​ runtime​ characteristics make it challenging for DevOps to gain visibility into how these technologies are dependent on each other. Moreover there are some common concerns like overall pipeline latency, error rates, lag, throughput, back pressure etc. that are critical metrics that need to be assessed and analyzed across the entire stack. In this session, Alan Ngai, VP Engineering at OpsClarity, and Premal Shah, Co-founder at 6sense, will discuss best practices and strategies on how to monitor ​these ​highly distributed real-time data processing frameworks and how DevOps can gain control and visibility over these data pipelines.


At 6sense Premal geeks out over tons of data, dreams about how to process data faster and extract the best signals from it, and obsesses over a blazing fast UI. Premal was one of the co-founders of GrepData, the Y Combinator backed startup that simplified Hadoop to make Big Data more accessible for everyone. Prior to his tenure with GrepData, Premal designed databases and optimized data access for Livingly Media.