Monitoring and Troubleshooting Real-time Data Pipelines

Enterprise​s​ are rapidly adopting big data frameworks to process and understand large-scale data in real time. Technologies like Apache Kafka, Apache Spark, Apache Storm and Elastic​​search are increasingly pushing the envelope on what is possible. However, while these technologies are quickly maturing, their extremely distributed and complex​ runtime​ characteristics make it challenging for DevOps to gain visibility into how these technologies are dependent on each other. Moreover there are some common concerns like overall pipeline latency, error rates, lag, throughput, back pressure etc. that are critical metrics that need to be assessed and analyzed across the entire stack. In this session, Alan Ngai, VP Engineering at OpsClarity, and Premal Shah, Co-founder at 6sense, will discuss best practices and strategies on how to monitor ​these ​highly distributed real-time data processing frameworks and how DevOps can gain control and visibility over these data pipelines.