Streamlio, an open-core streaming data fabric for the cloud era Posted on : Mar 13 - 2019

Apache Kafka replacement and beyond. This is open-core Streamlio's claim to fame, and today's announcement of a managed cloud service brings it one step closer to reality.

This Tricky aphorism of a song came to mind once more a couple of years back, when Streamlio came out of stealth. Streamlio is an offering for real-time data processing based on a number of Apache open source projects, and it directly competes with Confluent and Apache Kafka, which is at the core of Confluent's offering. What's the point in doing that?

In 2017, Apache Kafka was generally considered an early adopter thing: Present in many whiteboard architecture diagrams, but not necessarily widely adopted in production in enterprises. Since then, Kafka has laid a claim to enterprise adoption, and Confluent has acquired open-core unicorn status after its latest funding. This does not make things easier for the competition, obviously.

The question remains then: Why would anybody do this, and how could it work? Streamlio's answer to the why part seems to be that, despite being new for some, Kafka is retro. As to the how: Any offering seeking to position itself as a Kafka alternative would have to be substantially faster/more reliable, while also being compatible with Kafka and offering the options that Kafka offers.

Now, Streamlio is announcing a managed cloud service, bringing it closer to its vision. ZDNet discussed with Karthik Ramasamy and Jon Bock, Streamlio's CEO and founder and VP of marketing, respectively, about the vision and its execution.

REAL TIME ANALYTICS

Ramasamy's bio includes over two decades of experience in real-time data processing, parallel databases, big data infrastructure, and networking. He was engineering manager and technical lead for real-time analytics at Twitter, where he co-created the Apache Heron real-time engine.

Ramasamy's co-founders are Matteo Merli, ex-Yahoo, architect, and lead developer for Apache Pulsar and a PMC member of Apache BookKeeper, and Sanjeev Kulkarni, also former Twitter technical lead for real-time analytics and Twitter Heron co-creator.

The team certainly does not lack enterprise experience, and this is part of Streamlio's message. That also explains why Streamlio managed to secure Round A Funding of $7,5 million with Lightspeed, which as Ramasamy noted has also been involved in other open-core companies.

Ramasamy noted that Streamlio's headcount is below 100 people at this point. He also pointed out, however, that Apache Pulsar, which is at the core of Streamlio, has over 100 contributors and 3.000 stars on Github. The other two Apache projects on which Streamlio is based are Heron and BookKeeper.

Pulsar is the upper layer for Streamlio, and offers an API which is Kafka-compatible -- although there are nuances to this. There are architectural differences with Kafka, which as per the Streamlio team can be boiled down to the fact that Streamlio has a decoupled layer architecture. What we see as being at the core of this, especially when talking about running Streamlio in the cloud, is BookKeeper. View More