ARTIFICIAL INTELLIGENCE SYSTEMS are going to crash some of our cars, and sometimes they’re going to recommend longer sentences for black Americans than for whites. We know this because they’ve already gone wrong in these ways. But this doesn’t mean that we should insist—as many, including the European Commission’s General Data Protection Regulation, do—that artificial intelligence should be able to explain how it came up with its conclusions in every non-trivial case.
Demanding explicability sounds fine, but achieving it may require making artificial intelligence artificially stupid. And given the promise of the type of AI called machine learning, a dumbing-down of this technology could mean failing to diagnose diseases, overlooking significant causes of climate change, or making our educational system excessively one-size-fits all. Fully tapping the power of machine learning may well mean relying on results that are literally impossible to explain to the human mind.
Machine learning, especially the sort called deep learning, can analyze data into thousands of variables, arrange them into immensely complex and sensitive arrays of weighted relationships, and then run those arrays repeatedly through computer-based neural networks. To understand the outcome—why, say, the system thinks there’s a 73 percent chance you’ll develop diabetes or there’s a 84 percent chance that a chess move will eventually lead to victory—could require comprehending the relationships among those thousands of variables computed by multiple runs through vast neural networks. Our brains simply can’t hold that much information.
There’s lots of exciting work being done to make machine learning results understandable to humans. For example, sometimes an inspection can disclose which variables had the most weight. Sometimes visualizations of the steps in the process can show how the system came up with its conclusions. But not always. So we can either stop always insisting on explanations, or we can resign ourselves to maybe not always getting the most accurate results possible from these machines. That might not matter if machine learning is generating a list of movie recommendations, but could literally be a matter of life and death in medical and automotive cases, among others.
Explanations are tools: We use them to accomplish some goal. With machine learning, explanations can help developers debug a system that’s gone wrong. But explanations can also be used to to judge whether an outcome was based on factors that should not count (gender, race, etc., depending on the context) and to assess liability. There are, however, other ways we can achieve the desired result without inhibiting the ability of machine learning systems to help us.