The Hidden Engine: Unlocking AI Performance via System-Level Optimization

In the rapidly evolving landscape of AI, the spotlight often shines on model architectures and hardware accelerators. However, the true catalyst for scalable and efficient AI systems frequently remains in the shadows: the operating system. This talk introduces “The Hidden Engine”—the OS layer that orchestrates compute, memory, I/O, power, and thermal constraints, serving as the unsung hero behind AI performance. We will delve into how OS-level innovations—such as power-aware scheduling, topology-sensitive task placement, control-plane offloads, fine-grained telemetry, and dynamic resource adaptation—can significantly enhance throughput, reduce latency tails, and lower energy costs. Furthermore, we will explore the emergence of a new class of software systems that function akin to an operating system for AI inference: managing distributed workloads, memory hierarchies, request routing, and GPU scheduling seamlessly across large clusters. This hidden layer, often overlooked, dictates cost, latency, throughput, and efficiency at scale. Beyond the technical mechanics, we will examine trade-offs that are crucial for leaders: development cost, hardware portability, reliability, and operational complexity. Real-world case studies will illustrate how engineering investments at the OS layer yield disproportionate returns, and how organizations can balance risk versus reward.
Who is this presentation for?
Technical leaders, System architects and engineers, Platform engineers
Prerequisite knowledge:

What you'll learn?
Attendees will leave with a new perspective on infrastructure optimization—viewing the OS not merely as a support layer but as a strategic lever for AI performance and operational excellence.

Sujit Reddy Thumma is a Senior System Software Engineer at NVIDIA, where he leads the development of the foundational system software that dictates the performance, cost, and efficiency of next-generation AI. He operates at the heart of the GPUs, architecting the OS-level components, resource management, multimedia, and I/O pipelines for advanced GPU architectures like Blackwell and Hopper. Drawing on over a decade of experience in hardware-software co-design, Sujit possesses a unique, holistic view that connects low-level silicon architecture to large-scale operational trade-offs. His expertise in kernel development and hardware architecture makes him a compelling voice on why leaders should view the OS not as a mere support layer, but as a strategic lever for achieving a competitive advantage in AI. Sujit holds a Master's from Virginia Tech and a Bachelor's from the Indian Institute of Technology (IIT), Guwahati.