The industrial data revolution: What founders got wrong

In February 2010, The Economist published a report called “Data, data everywhere.” Little did we know then just how simple the data landscape actually was. That is, comparatively speaking, when you consider the data realities we’re facing as we look to 2022.

In that Economist report, I spoke about society entering an “Industrial Revolution of Data,” which kicked off with the excitement around Big Data and continues into our current era of data-driven AI. Many in the field expected this revolution to bring standardization, with more signal and less noise. Instead, we have more noise, but a more powerful signal. That is to say, we have harder data problems with bigger potential business outcomes.

And, we’ve also seen big advances in artificial intelligence. What does that mean for our data world now? Let’s take a look back at where we were.

At the time of that Economist article, I was on leave from UC Berkeley to run a lab for Intel Research in collaboration with the campus. We were focused all the way back then on what we now call the Internet of Things (IoT).

At that time, we were talking about networks of tiny interconnected sensors being embedded in everything — buildings, nature, the paint in the walls. The vision was that we could measure the physical world and capture its reality as data, and we were exploring theories and building devices and systems toward that vision.

We were looking forward. But at that time, most of the popular excitement about data revolved around the rise of the web and search engines. Everybody was talking about the accessibility of masses of digital information in the form of “documents” — human-generated content intended for human consumption.

What we saw over the horizon was an even bigger wave of machine-generated data. That’s one aspect of what I meant by the “industrialization of data” — since data would be stamped out by machines, the volume would go up enormously. And that certainly happened.

The second aspect of the “Industrial Revolution of Data” that I expected was the emergence of standardization. Simply put, if machines are generating things, they’ll generate things in the same form every time, so we should have a much easier time understanding and combining data from myriad sources.

The precedents for standardization were in the classical Industrial Revolution, where there was an incentive for all parties to standardize on shared resources like transportation and shipping as well as on product specifications. It seemed like that should hold for the new Industrial Revolution of Data as well, and economics and other forces would drive standardization of data.

That did not happen at all.

In fact, the opposite happened. We got an enormous increase in “data exhaust” — byproducts of exponentially growing computation in the form of log files — but only a modest increase in standardized data.

And so, instead of having uniform, machine-oriented data, we got a massive increase in the variety of data and data types and a decrease in data governance.

In addition to data exhaust and machine-generated data, we started to have adversarial uses of data. This occurred because the people involved with data had many different incentives for its use.

Consider social media data and the recent conversations around “fake news.” The early 21st century has been a giant experiment in what makes digital information viral, not only for individuals but for brands or political interests looking to reach the masses.

Today, much of that content is in fact machine-generated, but it’s machine-generated for human consumption and human behavioral patterns. This is in contrast to the wide-eyed “by people, for people” web of years ago.

In short, today’s data production industry is incredibly high volume, but it is not tuned for standard data representations, not in the sense I expected at the time of those predictions over a decade ago.

The state of innovation: AI versus human input

One thing that has clearly advanced substantially in the past decade or so is artificial intelligence. This sheer volume of data we are able to access, process and feed into models has changed AI from science fiction into reality in a few short years.

But AI is not as helpful in the business data processing domain as we might expect — at least not yet. There is still a surprising disconnect between AI technology like natural language processing and structured data. Even though we’ve had some progress, for the most part, you can’t talk to your data and expect much back. There are some situations where you can Google for a quantitative question and get back a little table or chart, but that’s only if you ask just the right questions.

For the most part, AI advances are still pretty divorced from stuff like spreadsheets and log files and all these other more quantitative, structured data — including IoT data. It turns out the traditional kinds of data, the kinds of data we’ve always put in databases, has been much harder to crack with AI than consumer applications like image search or simple natural language question answering.

Case in point: I encourage you to try asking Alexa or Siri to clean your data! It’s funny, but not very helpful.

Popular applications of AI haven’t projected back yet to the traditional data industry, but it’s not for lack of trying. Lots of smart people at both universities and companies haven’t been able to crack the nut of traditional record-oriented data integration problems.

Yet, full automation evades the industry. Part of that is because it’s hard for humans to specify what they want out of data upfront. If you could actually say, “Here’s precisely what I’d like you to do with these 700 tables,” and follow up with clear goals, maybe an algorithm could do the task for you. But that’s not actually what happens. Instead, people see 700 tables, wonder what’s in there and start poking around. Only after a lot of poking do they have any clue what they might want to happen to those tables.

The poking around remains creative work because the space of ways to use the data is just so big and the metrics of what success looks like are so varied. You can’t just give the data to optimization algorithms to find the best choice of outcome.

Rather than waiting for full automation from AI, humans should get as much help as they can from AI, but actually retain some agency and identify what is or isn’t useful, then steer the next steps in a certain direction. That requires visualization and a bunch of feedback from the AI.

Understanding the impact of data and controlling data spread

One place AI has really shined, though, is in content recommendation. It turns out that computers are frighteningly effective at targeting and disseminating content. And oh boy, did we underestimate the incentives and impacts around that aspect of data and AI.

Back then, the ethical concerns we had around data and its uses in AI were mostly around privacy. I remember big debates about whether the public library should have digital records of the books you reserve. Similarly, there were controversies over grocery loyalty card programs. Shoppers didn’t want grocery chains to keep track of what food they bought when and target them for accompanying items.

That mentality has largely changed. Today, teenagers share more radically more personal information on social media than the brand of food they purchase.

While I wouldn’t say that digital privacy is in a good state, it is arguably not the worst of our data problems today. There are issues such as state-funded actors trying to introduce mayhem into our social discourse — using data. Twenty years ago, very few people saw this stuff coming our way. I don’t think there was a great sense of the ethical questions of what could go wrong.

This leads to what’s next, and even currently in process, in the evolution of our uses of data. What becomes the role of governments and of well-meaning legislation? Without predicting all the ways tools will be used, it’s hard to know how to govern and restrict them intelligently. Today, we are in a state where it seems like we need to figure out the controls or incentives around data and the way it is promulgated, but the tech is shifting faster than society is able to figure out risks and protections. It’s unsettling, to say the least.

So, were the predictions spot-on?

As a professor, I’d award it a passing grade, but not an A. There is substantially more data available to us with more uses than we probably ever could have imagined. That’s led to incredible advances in AI and machine learning along with analytics, but on many tasks, we’re still just scratching the surface, while on others we’re reaping the whirlwind. I am fascinated to see what the next 10 to 20 years will bring and look back on these issues again.