Speaker "Maosong Fu" Details Back



Twitter Heron: Stream processing at scale


Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real- time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage – all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This talk will present the design and implementation of the new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and share our experiences from running Heron in production.


Maosong Fu is the technical lead for ​Heron and ​Real Time Analytics at Twitter. ​​He ​worked on various components of Heron including Heron instance, metrics manager, scheduler etc. He ​is the author of ​few publications in distributed area​ and has a master's degree from Carnegie Mellon University and bachelor's from Huazhong University of Science and Technology.