Data Predictions- We’re witnessing the shift from big data collection to real-time data consumption, and 2019 will mark the tipping point as intelligent applications — not people — become the primary consumers of data. With that in mind, here’s a look at the trends in the data trenches that are facilitating this shift for companies worldwide and the predicted implications across multiple aspects of technology.
Onerous Mega Projects Become Iterative Micro Adjustments
I’ve seen firsthand how companies of all stripes are shifting from big data moonshots — such as all-encompassing data lakes or broad internet of things (IoT) platforms — to embrace the benefits of incremental improvements by leveraging “in the moment” data insights, and I believe this trend will accelerate in 2019. In part, this is a recognition of failure: These mega projects cost too much and took too long to implement (if they were even completed at all). Many were never delivered as promised, and those that were ended up being outdated by the time they were ready for production. It’s also a recognition of the larger shift from slow planning cycles to fast iteration. When getting data is slow and hard, lengthy planning is necessary and complexity is expected. When getting data is fast and easy, fast iteration and adjustment wins every time.
Streams Proliferate As Lakes Dry Up
A corollary to this is that 2019 will mark the decline of the data lake as organizations look to leverage real-time data streaming rather than dumping data into one big but slow and murky reservoir in hopes of deriving eventual value. Data lakes reflect the mega projects reminiscent of earlier big data initiatives, which generated a lot of revenue for consultants but few results for organizations. While there will certainly still be a need to store and archive data for long term needs, I think companies will focus more on extracting value and insight as data streams into the organization rather than on just collecting data for eventual refinement and use.
Data Becomes Part Of The Corporate Fabric
With the shift to real-time processing of fast data will come an emerging IT focus on providing a data fabric or backbone that applications and users tap for data as needed rather than hoarding their own copies. A 2017 survey of corporate executives by NewVantage noted over 85% of firms have initiated programs to create a data-driven culture. New technologies — in particular, next-generation publish/subscribe and related messaging solutions — are making this possible, bringing multiple benefits.
One of these is simplicity, as companies move away from a patchwork of data systems and data stores to embrace a more integrated organization-wide data fabric. Secondly, we’re seeing the elimination of silos: When data becomes distributed and ubiquitous rather than trapped in isolated, discrete systems, silos disappear, and that data simultaneously becomes more accessible and delivers more value.