An In-Depth Look at Big Data Trends and Challenges Posted on : Oct 19 - 2018

Our friends over at Qubole, the data activation company, announced the results of its 2018 Big Data Trends and Challenges report — an annual, global survey of IT and data professionals on the trends, challenges, and solutions of big data and machine learning initiatives across the enterprise. The survey found that the value achieved from big data in on-premises environments has significantly lagged behind expectations, and as a result, companies are rapidly shifting to the cloud for applications centered around machine learning (ML), and analytics.

According to the survey, 73 percent of businesses are now performing their big data processing in the cloud, up from 58 percent in 2017. The shift towards cloud is necessitated in part due to the ever-growing volume and diversity of data that companies are dealing with, as 44 percent of organizations now report working with massive data lakes over 100 terabytes in size.

Facilitated by this shift to the cloud, machine learning programs are expected to expand across a wide range of use cases in the next year. A majority of respondents cited improving data security and threat protection as the top priority for their machine learning initiatives while optimizing customer experience (49 percent) and predictive maintenance (43 percent) were also high on the list of ML priorities.

Apache Spark and Presto have also shown impressive gains among big data frameworks in the last year. 31 percent of respondents now report using Spark as their framework, with a 29 percent growth from 2017. Presto, while in use by a smaller 13 percent of companies, saw its user base grow 63 percent in the last year. The survey results also indicated a shift by organizations away from homegrown approaches in favor of open source technologies.

The size, diversity, and applications of big data are accelerating at a near-exponential rate, and businesses are quickly discovering that traditional data management systems and strategies are no longer capable of supporting their demands,” said Ashish Thusoo, co-founder and CEO, Qubole. “Instead, a new generation of cloud-native, self-service platforms have become essential to the success of data programs, especially as companies look to expand their operations with new AI, machine learning and analytics initiatives.”

Additional findings from the report include:

Data teams continue to face a number of challenges when implementing both big data and machine learning projects:

• Respondents cited a lack of experience slowing project progress (44 percent), struggling to keep up with new data sources (42 percent) and issues with constantly evolving use cases (41 percent) as their top challenges.
• On the machine learning side, analyzing extremely large data sets (40 percent), ensuring adequate staffing and resources (38 percent) and integrating new data into existing pipelines (38 percent) were cited as the primary obstacles to machine learning projects. View More