Enabling Real-Time Insight over a Data Lake at scale

A data lake provides a large repository of raw storage and enables a production platform for data pipelines and applications that require data processing or computations at a large scale. However, it often struggles to deliver real-time analytical responses without significant investment in computing power (such as large clusters, having co-located compute and storage). Emerging Internet of Things (IoT) applications demand systems that support both operational and batch workloads, requiring massive computing power and storage. Designers are challenged to scale compute and storage independently to serve their changing needs. This session will explore how combining stream and batch processing in a data lake can ingest and analyze real-time streaming data, as well as persist and correlate with historical data, thus allowing intuitive searches, queries and visualizations for actionable insight. Discussed will be how using a disaggregation of compute, flash and object storage can enable the delivery of rapid response times and scaling requirements for IoT use cases, without compromising on data persistence, data quality, durability and cost.

Sanhita Sarkar is a Global Director, Software Development, Data Center Architectures & Ecosystems, at Western Digital, where she focuses on software design and development of analytics features and solutions spanning edge, data center, data lake, and cloud. She has expertise in key vertical markets such as the Industrial Internet of Things (IIoT), Defense and Intelligence, Financial Services, Genomics, and Healthcare. Sanhita previously worked at Teradata, SGI, Oracle, and a few startups. She was responsible for overseeing design, development, and delivery of optimized software and solutions involving large memory, scale-up, and scale-out systems. Sanhita has authored multiple patents, published several papers, and has spoken at several conferences and meetups. She received her Ph.D. in Electrical Engineering and Computer Science from the University of Minnesota, Minneapolis.