85% of big data projects fail, but your developers can help yours succeed Posted on : Nov 11 - 2017

Big data has been a big disappointment, but there's a way to ensure yours won't be.

A year ago, Gartner estimated that 60% of big data projects fail. As bad as that sounds, the reality is actually worse. According to Gartner analyst Nick Heudecker‏ this week, Gartner was "too conservative" with its 60% estimate. The real failure rate? "[C]loser to 85 percent." In other words, abandon hope all ye who enter here, especially because "[T]he problem isn't technology," Heudecker said. It's you.

Old dogs, new tricks

Well, not you exactly, but rather the difficulty of grafting modern big data practices onto existing infrastructure and into company cultures that are ill-prepared to embrace big data. After working through scads of botched big data projects, Heudecker told me that the primary causes of failure are the difficulty inherent in integrating with existing business processes and applications, management resistance, internal politics, lack of skills, and security and governance challenges.

It's interesting that management would be a key blocker of big data projects, given that they've been the biggest cheerleaders for big data, touting its power to transform their businesses on earnings calls and to the press. Ask them how those projects have turned out, in fact, and 80% will tell you that they've been super successful and have delivered significant value, according to a NewVantage Partners survey.

Get into a postmortem with the teams actually responsible for running these projects, however, and you get that 85% failure rate that Gartner has uncovered. Ironically, this may partly derive from a tendency for executives to trust their gut rather than data, as a Fortune Knowledge Group survey revealed. Probe the NewVantage respondents a bit more and they, too, admit to a cultural mismatch that is nettlesome to overcome:

If there is any sobering trend in these results, it lies in the apparent difficulty of organizational and cultural change around Big Data. More than 85 percent of respondents report that their firms have started programs to create data-driven cultures, but only 37 percent report success thus far. Big Data technology is not the problem; management understanding, organizational alignment, and general organizational resistance are the culprits. If only people were as malleable as data. View More