January 5, 2022

User-centered Artificial Intelligence - PART 1
‍

Artificial Intelligence has a problem: A recent study by Venturebeat implied that 87% of AI projects don’t make it to production*.
‍
There are multiple root causes we can identify that result in this high number of failures, but before we go into them, I think that at its core, it has to do with a misunderstanding between the (business) people using the data product, and the (technical) people creating it. End users of data products often have little to no technical knowledge of the products they work with.
‍
Have you ever experienced that the AI solution you worked on:

• wasn’t valued or adopted by the business?
• went to production, but still wasn’t really used in practice?
  
• wasn’t delivering the expected / desired business value?
  

Sometimes to the point where management starts to complain about the return on investment of the entire AI (or data science) team / department?
‍
I think most AI departments have at times experienced that an AI solution wasn’t valued, adopted, or used. Solutions don’t make it to production, aren’t used in practice, aren’t wholeheartedly welcomed by the business or other user bases, and therefore don’t deliver the expected results.

The challenge with data products

Studies investigating the problems with AI projects mention reasons such as: lack of support from leadership, siloed data sources, or lack of collaboration.
"But, wait a minute...", you might think, "This is true for other types of software development projects as well. So, what makes projects focusing on AI, data platforms, data science or machine learning so different?"
‍
Let me start off with an example coming from Harvard Business Review (Wettersten & Malgren (2018)**).

Wettersten, J., & Malmgren, D. (2018). What Happens When Data Scientists and Designers Work Together. Harvard Business Review.

This example centers around a data platform, called Rise, which was built especially for college and professional athletes to track their sleep. Research shows that sleep plays an important role in athletic performance, and by adjusting your sleep behavior you can make sure that at the time you need to perform, you will be hitting your peak performance.
‍
The data scientists at Rise developed the solution shown on the left-hand side: it showed all the relevant data in charts and graphs. Rise developers expected athletes to look at these charts and graphs to determine what decisions to make next. But for the players, the experience was challenging. They struggled to find the insights they needed to determine what actions they should take to reach that peak performance they were after.
‍
Rise, determined to improve their app, was convinced that they just needed easier-to-read charts and graphs.But then a design company, IDEO, stepped in. As the combined team of data scientists and designers spent time with players and coaches, they discovered that Rise didn’t have a data visualization problem. Instead, they had a user experience problem. Charts and graphs were far less important for the players than knowing when to go to bed each night and when to wake up the next morning. Within a few weeks, the charts and graphs moved into the background of their app, making place for an alarm clock and a chat tool center stage.

What can we learn from this example?

Data Science as a Product. Going from MVP to production. | by Tad Slaff from Picnic

Now, let’s get back to the initial question: What makes AI projects so likely to fail?
‍
There seem to be four important reasons***, and although there might be more, I think these four touch upon an interesting insight: It's all a matter of perspective.

Technical professionals, such as data scientists and machine learning / AI / data engineers, spend most of their time in a data environment. Whereas the (business) people using their products spend most of their time in a business environment. This creates some interesting differences in the way they perceive the world, and especially the business, organization, client or customer they work for.

• The first difference is that machine learning and artificial intelligence live in the world of probabilities and uncertainties. A typical output of a machine learning-based fraud model will be something like 'The probability of this activity being fraudulent is 73% +/- 5% with a 95% confidence interval’. But people on the business, customer, compliance, or management side mostly live in a deterministic world, and are likely to say something like ‘We want to block all fraudulent activities'.
  
• Secondly, AI projects introduce levels of unpredictability that (usually) don’t occur in ‘traditional’ software development. Oftentimes, within an AI project, it is not known in advance how well a model will perform before building it. It can take a week, three months, or it may not even be possible to get an acceptable level of performance. This makes it very difficult to put together a neat project plan with timelines and deliverables that the business often likes to see.
• Third, model trust is essential when releasing a model to production. By developing a model with the aim of productizing it for the business, the data team is entering a domain in which the business professionals are the experts. In many cases, the intention is to automate a manual process or replace a set of carefully crafted business rules. While these rules aren’t perfect, they were built by people within the business departments, who often possess a deep understanding of the business and its operations. Handing off a black-box machine learning algorithm and telling those people that it is going to replace their current way of working is a challenging task. In the end, it’s the business who takes the responsibility for the profit or loss from whatever process the model is looking to automate, and data scientists need to convince them to put their livelihood in the hands of their models.
• And lastly, being an enabling technology by nature, data science and artificial intelligence can be applied in almost any department within an organization. And so, the container term “business” refers to a diverse mixture of expertise. Each department harnesses its own deep knowledge of their respective disciplines. Moreover, each department has its own culture, way of working, and vocabulary. Many of these departments (oftentimes) possess little knowledge of data technology and/or IT. The data departments on the other hand, often consist of their own disciplines with their own culture, work processes, and vocabulary. Strong data teams will possess deep knowledge of data technology, but relatively little knowledge of the application domain, business operations, or of the exact way in which the rest of the business does their work.

Acknowledging that the day-to-day environment, expertise, and experiences of the end users of your data product substantially differs from the environment, expertise, and experiences in your data team or department is the first step towards designing, developing and evaluating AI in a human-centered way.
‍
My next blog will dive deeper into ways to overcome these challenges, and how to increase the success rates of your data products, so that they are valued, adopted and used by your target users.