Microsoft Now Developing Its Own Hadoop Posted on : Aug 29 - 2020

Hadoop might be dead, but that’s not stopping public cloud providers from using it. The latest to make a move is Microsoft Azure, which in July announced that it would begin developing its own distribution under its HDInsight brand.

Microsoft, of course, has been providing Hadoop software on its Azure cloud for many years. It was an early partner of Hortonworks, and basically had an OEM version of the Hortonworks Data Platform (HDP) for the cloud that it called HDInsight.

But Hortonworks merged with Cloudera in early 2019, and the HDP product line is no longer being developed, although it is still being supported by Cloudera, along with its legacy Hadoop distribution, Cloudera Distribution including Hadoop (CDH), until at least 2022.

Rather than form a partnership with Cloudera to use its converged Hadoop distribution, dubbed Cloudera Data Platform (CDP), Microsoft decided to head out on its own and continue developing HDInsight by itself.

Microsoft made the announcement on July 21, during its Inspire conference. “Microsoft’s supported distribution of Apache Hadoop, which will be generally available July 2020, is fully open source and compatible with the latest version of Hadoop,” the company said in its Inspire book of news.

“With the distribution, users can provision a new HDInsight cluster based on Apache code that is built and wholly supported by Microsoft,” the company continued. “Customers will be automatically migrated to the supported distribution.”

Clearly there is still room for the Hadoop family of products in the enterprise, at least on the cloud (Cloudera executives prefer not to mention the “H” word at all anymore, although its software still features the same motley gang of zoo animals that it always has). View More