Using DataOps to create business value from big data Posted on : Apr 21 - 2020

Now is the time to take all that data your organization has amassed through the years and, with DataOps, use it to make actionable business decisions.

Data is not only "big," it's also unruly. It populates every pocket of the enterprise. Every information system, every cloud, is dripping with it. And not unlike Jed Clampett's "bubbling crude (oil that is)," it takes a lot of machinery, a lot of refining, to make it useful.

It took the 2010s to build out the infrastructure of big data, the constellation of platforms and applications that store, tag, govern, manage and deliver it. In that regard, I see the past decade as a test lab with the focus on developing, implementing and integrating a large swath of heterogenous solutions to streamline turning data into real business intelligence.

That period of testing has paid off. In 2020, we've not only turned the page on a new decade, we've turned the corner on making data the true currency of value creation. And, the good news for every enterprise -- whether large or small, brimming with IT staff or manned (or womanned) by a hearty few -- is that turning data into insights at speed and scale is available now for everyone. (And it doesn't cost a lot or take years to implement.)

The rise of DataOps

Two complementary developments have delivered this transformation. One is the evolution of an Agile mindset, a framework for approaching, implementing, and demanding more from data management solutions, called DataOps.

The other development is a series of technological breakthroughs, perhaps not obvious amid the sheer volume of data management solutions in the market, that make end-to-end data management not only possible, but safer, faster and more useful than ever before.

First up is DataOps, which Gartner calls "a collaborative data management practice focused on improving the communication, integration, and automation of data flows between data managers and data consumers across an organization."

I can say from firsthand experience that the rise of DataOps, modeled after the success of DevOps (an Agile engineering framework for enterprise IT that streamlined application development, deployment and continuous improvement), is the result of reorienting data management around value creation. It's a get-to-value-first, get-to-value-fast philosophy that enables the enterprise to either fail or succeed quickly and rapidly build on what works.

A lot is riding on this shift. In Getting DataOps Right, O'Reilly's authors summarized:

"The necessity of DataOps has emerged as individuals in large traditional enterprises realize that they should be using all the data generated in their company as a strategic asset to make better decisions every day." They concluded, "Just like the internet companies needed DevOps to provide a high-quality, consistent framework for feature development, enterprises need a high-quality, consistent framework for rapid data engineering and analytic development." View More