2018: A Big Data Year in Review Posted on : Dec 13 - 2018

As 2018 comes to a close, it’s worth taking some time to look back on the major events that occurred this year in the big data, data science, and AI space. There’s a lot that occurred – and a lot that we chronicled in the virtual pages of Datanami. We hope you enjoy this retrospective as you prepare for 2019.

Data security continued to be a major topic in 2018, particularly as the rash of big data breaches continued. The IT world had a rude awakening following the 2018 New Year’s celebration when the Spectre and Meltdown vulnerabilities were discovered in practically every computer processor on the planet. Apparently, chipmakers took some shortcuts in developing the speculative execution methods that boost multi-threaded performance in modern CPUs. Failure to apply the patches (which robbed the chips of performance) put vulnerable data at risk.

The sophistication of machine learning automation tools increased a good deal during 2018, which is not surprising. Data scientists who are looking to boost their own productivity – or data analysts and power users who wanted to swing above their weight – had a smorgasbord of ML automation tools available from a raft of vendors like Cloudera, DataRobot, H2O.ai, Anaconda, Domino Data Labs, IBM, SAS, and Alteryx – not to mention cloud offerings from AWS, Azure, and GCP or open source kits, including those for Python, R, Java, and Scala.

Data governance isn’t a new concept, but it sometimes seemed that way this year, particularly with the General Data Protection Regulation (GDPR) threatening big sanctions on companies that were careless with data. The growing concerns around data security – along with the difficulty data science teams were facing in just corralling and making sense of data – combined to put data governance on the map in a big way.

The rapid growth of the cloud as a platform for storing big data and analyzing it was a huge story for 2018. At the end of the year, AWS had a $27 billion yearly run rate, and was growing at 46% annually. Azure and GCP were growing even faster, although they weren’t even close to matching AWS in the revenue department. Nobody was surprised when AWS unveiled a slew of new ML functionality at its annual re:Invent show at the end of the year

Let’s face it: Big data, as a defining name and concept, is on its last legs. The new paradigm that’s emerging is being defined with an old-ish phrase, artificial intelligence – but one that’s being infused with new meaning. As the big data bubble shrinks, AI’s just keeps getting bigger. The smart money folks on Sand Hill Road poured money into AI startups at a rapid clip in 2018, and there’s no sign that it’s abating. View More