Speaker "Alan Ngai" Details Back



Monitoring and Troubleshooting Real-time Data Pipelines


Enterprise​s​ are rapidly adopting big data frameworks to process and understand large-scale data in real time. Technologies like Apache Kafka, Apache Spark, Apache Storm and Elastic​​search are increasingly pushing the envelope on what is possible. However, while these technologies are quickly maturing, their extremely distributed and complex​ runtime​ characteristics make it challenging for DevOps to gain visibility into how these technologies are dependent on each other. Moreover there are some common concerns like overall pipeline latency, error rates, lag, throughput, back pressure etc. that are critical metrics that need to be assessed and analyzed across the entire stack. In this session, Alan Ngai, VP Engineering at OpsClarity, and Premal Shah, Co-founder at 6sense, will discuss best practices and strategies on how to monitor ​these ​highly distributed real-time data processing frameworks and how DevOps can gain control and visibility over these data pipelines.


As a co-founder and the VP of Engineering at OpsClarity, Alan brings over 15 years of experience building systems and engineering teams from the ground up. Prior to OpsClarity, he lead teams in solving large scale, complex problems at companies such as eBay, Yahoo and Telenav. Over the years, Alan has worked on building cloud platforms, search, GIS services and navigation, SE automation platforms, and more. He has a Bachelor of Science degree in Computer Science & Engineering from UC Davis.