Five years ago—in February 2012—an article in the New York Times’ Sunday Review heralded the arrival of a new epoch in human affairs: “The Age of Big Data.” Society was embarking on a revolution, the article informed us, one in which the collection and analysis of enormous quantities of data would transform almost every facet of life. No longer would data analysis be confined to spreadsheets and regressions: The advent of supercomputing, combined with the proliferation of internet-connected sensors that could record data constantly and send it to the cloud, meant that the sort of advanced statistical analysis described in Michael Lewis’ 2003 baseball book Moneyball could be applied to fields ranging from business to academia to medicine to romance. Not only that, but sophisticated data analysis software could help identify utterly unexpected correlations, such as a relationship between a loan recipient’s use of all caps and his likelihood of defaulting. This would surely yield novel insights that would change how we think about, well, just about everything.
The Times was not the first to arrive at this conclusion: Its story drew on a seminal McKinsey report from 2011and was buttressed by an official report from the 2012 World Economic Forum in Davos, Switzerland, titled “Big Data, Big Impact.” But the pronouncement by the paper of record seems as apt a milestone as any to mark the era’s onset. The following month, Barack Obama’s White House launched a $200 million national big data initiative, and the frenzy commenced: Academia, nonprofits, governments, and companies raced to figure out just what “big data” was and how they could capitalize on it.
The frenzy, as it turned out, was short-lived. Five years later, data plays a vastly expanded role in our lives, yet the term big data has gone out of fashion—and acquired something of an unsavory reputation. It’s worth looking back at what, exactly, happened to the revolution we were promised, and where data, analytics, and algorithms are headed now.