A Decade into Big Data Posted on : Dec 12 - 2017

2016 marked the 10-year anniversary of Hadoop, a name closely associated with “Big Data.” Prior to the advent of Big Data, companies invested in solutions that were not forward-looking; they could only address the immediate needs of businesses. These traditional solutions were way too expensive, especially considering their very limited capabilities.

The data landscape then was quite different from what it is today. Significant upfront investments were required to handle just a few dozens terabytes. Scaling was an issue, as most solutions incorporated specialised hardware and were built with a scale-up rather than a scale-out approach. Things started changing with the emergence of multi-core processors, distributed storage and the rise of social media. Organisations which were driven purely by use cases, now started looking at things from the other end, “the Data.”

Big Data Era – 2008

The first major step towards this data oriented approach came in the form of Hadoop, the Data-Hungry Big Yellow Elephant by Doug Cutting. Born out of the research paper of Google, Hadoop introduced a new way of looking at data. Though the initiative started around 2006, it was not until 2008 that it was officially launched as an open-source Apache project. Yahoo, which hired Cutting, rolled out its two main components – Hadoop Distributed File System (HDFS), a cheaper alternative for distributed-storage, and Map-Reduce (MR), a very efficient way to parallelise computations.

So now we had an affordable tool that could solve some of the common technology problems like Search, Join, Merge, etc. Many organisations were struggling with these problems, and they could see immediate value out of Hadoop.

Banks and financial institutions which ran daily and monthly batch jobs, experienced major improvements as they were able reduce execution times to an hourly basis. Also, with social networking and digital marketing gaining traction, companies wanted to use these social data for more effective online campaigns, targeting and even recruitment, forming a strong business case for data sharing. View More