Enterprise AI Infrastructure: Strategies for efficient inference at scale

Enterprises are moving beyond experimentation and proofs of concept toward production-scale AI systems. However, scaling AI brings new challenges in availability, resilience, and security—especially with the emergence of agentic AI systems that autonomously interact with enterprise data, applications, and workflows. This talk dives deep into the engineering strategies needed to design and operate enterprise AI infrastructure. It will highlight reference architectures for enterprise-scale inference, including hybrid multi-cloud deployment models, GPU/accelerator utilization strategies, and platform services for model lifecycle management. Key focus areas include: * Engineering AI inference platforms that scale efficiently under enterprise workloads. * Embedding resilience through failure-aware architectures, redundancy models, and performance observability. * Balancing agility and control: how to standardize platforms while enabling team-level innovation. * Real-world lessons learned from deploying enterprise AI at scale, with an emphasis on practical business and engineering trade-offs. The session blends business value articulation with technical depth, drawing on experience with enterprise infrastructure platforms and the latest AI engineering innovations.
Who is this presentation for?
Enterprise and cloud architects designing AI platforms ● Engineering leaders and technical managers scaling AI solutions ● AI/ML engineers responsible for deploying agentic or LLM-based systems in production ● Business and product leaders seeking to understand infrastructure trade-offs for AI at scale
Prerequisite knowledge:
Fundamental concepts of AI
What you'll learn?
Learning Objectives or Key Takeaways * Understand how to design resilient architectures for AI inference and agentic AI at enterprise scale. * Gain insight into scalable operations, including observability, recovery, and resource efficiency. * Learn practical lessons in aligning AI infrastructure with business requirements, compliance, and organizational priorities. * Leave with actionable approaches to accelerate enterprise AI adoption effectively.

Sandeep Kaipu is an Engineering Manager at Broadcom. With over 20 years of experience in enterprise AI infrastructure and engineering leadership, he has driven large-scale platform innovations, holds multiple patents, and authored the book “AI Engineering Leadership”. He advises academic programs, speaks at leading technology conferences, and focuses on aligning AI engineering strategies with enterprise business value. His work spans enterprise platforms, AI inference at scale, and strategies for securing the next generation of Agentic AI systems.