Big Data Is Hard-We’re over a decade into the big data era that emerged from the tectonic collision of mobile, Web 2.0, and cloud forces. Bolstered by progress in machine learning, we stand at the cusp of a new AI era that promises even greater automation of rudimentary tasks. But despite the progress in AI, big data remains a major challenge for many enterprises.
There are lots of reasons why people may feel that big data is a thing of the past. The biggest piece of evidence that big data’s time has passed may be the downfall of Hadoop, which Cloudera once called the “operating system for big data.”
After acquiring Hortonworks, Cloudera and MapR Technologies became the two primary backers of Hadoop distributions. The companies had actually been working to distance themselves from Hadoop’s baggage for some time, but they apparently didn’t move fast enough for customers and investors, who have hurt two companies by holding out on (Hadoop) upgrades and investments.
In Hadoop’s place, we have the public cloud vendors and their loose collection of data storage and processing options. Companies can do everything on the Amazon, Microsoft, and Google clouds that they sought to do with Hadoop, at the same petaybyte scale. In fact, the clouds have even more processing options, and none of the requirements to actually stand up and manage physical clusters, which is fueling huge growth in clouds.
But companies that were hoping the cloud would solve their data management challenges will be disappointed to find that things aren’t any easier on the cloud than they are on-premise, says Buno Pati, the CEO of Infoworks, a provider of data orchestration tools.
“The cloud doesn’t solve that problem,” Pati told Datanami recently. “There’s not a cloud vendor today that can give you a highly automated, integrated, and abstracted system on which you can manage the entirety of your data and analytics activity.”