Are We Nearing The End Of Hadoop And Big Data? Posted on : Nov 16 - 2018

A few weeks ago, two giants of the big data Hadoop era, Cloudera and Hortonworks, announced they would be merging. The announcement claimed it would be a “merger of equals.” It is fascinating to see these two groundbreaking pioneers coming together.

I remember several years ago when they burst onto the technology scene. They promised to help leaders re-architect their data centers and information frameworks — and substantially lower the cost of storing and processing data, from the many thousands of dollars per terabyte that used to be common. They recognized that the data being generated by diverse sources such as internet-of-things devices, video and non-text media, transaction logs, social graphs and social interaction logs, among many others, was so huge, that traditional data infrastructures were simply not suited to the task. They enabled businesses to do projects that couldn’t be done before. They were at the heart of the shift from solving "IT problems" to solving "business problems," and business leaders quickly understood the potential of these new technologies to deliver new services, powered by data, to their customers. They were also at the core of businesses moving to making decisions informed by data, rather than their executives’ intuition. This is because they made it possible for businesses to analyze all the data they were collecting and use that to make decisions.

Of course today, these two companies face tremendous competition from corporations such as Amazon and Microsoft. Businesses today have a range of options for handling and analyzing large amounts of data, particularly on public cloud platforms. We’re also seeing businesses wanting to move to serverless architectures, where an application can use backend-as-a-service and function-as-a-service functionalities provided by third parties. And here there is the added advantage: that businesses just pay for the compute power and storage that they use.

So, it’s no longer just Hadoop. Cloudera Chief Executive, Tom Reilly, admitted as much, in his comments after the merger: “Hadoop has evolved so drastically that we don’t even mention it anymore.” This analysis provides an overview of the different options available to enterprises instead of using Hadoop. And you have to wonder, if this trend continues, what the future will be for the technology. As the author writes, “The center of gravity has moved elsewhere.”

What this development represents is how big data is now becoming just data. Every organization, large and small, now has access to an unparalleled quantity and quality (and more current/real-time) data than at any time in history. They have more technological options to build services using this data — and this is important because different use cases (using different types of data) mean it’s possible to choose the right technology for what you need. For example, there are numerous open-source options, as well as proprietary machine learning platforms. Many of these make the 10-year-old Hadoop technology look dated. View More