January 14, 2022

user-centered AI - part 2

My recent blog was about the acknowledging the difference in perspectives between the intended end users of your data product and the data team. Today I want to discuss how to take this realization and use it to improve the quality, trustworthiness, impact, and adoption of your data products.
‍
For wouldn’t it be amazing if users and stakeholders felt that the data products you develop are game changers? Yes, yes it would. Now let’s make sure they do.

User adoption

The thing about use and adoptions is*: you can present all the ROC curves and test set performance you like, but if the first couple of predictions a model makes happen to be incorrect, there is a risk that users will not give it another chance to prove itself and recover. As mentioned in part 1: trust is essential for adoption. If you want to persuade your users to drop their trusted rule-based solution, you need to convince them and earn their trust. The basic business rules that were previously in place may not have been great, but the business knew in which cases it worked well. And when it didn’t, they were able to intervene accordingly. If all is well, your models will have an operational impact. And if the business doesn’t trust those models, they won’t be used. Simple as that.

There are many scientific studies and models that describe technology acceptance, and even though scholars aren’t fully agreed on the details, the research shows that there are a few important factors that determine whether a product is adopted by the user:

• The perceived value of the product: will it solve the problem, or will it create additional value/opportunities?
• The perceived ease of use: how difficult will it be to learn to use it?
• The extent to which the user trusts that the product will do what it promises
• And the perceived ease of adoption: how easily can the product be integrated in ongoing tasks and processes?

On top of that, with each experience the user has with your product, the user's understanding of perceived value, perceived ease of use, trust, and perceived ease of adoption changes. For better or worse.
‍
So, to create value and impact with data products, you need to understand your user, their context, their operations, their objectives, their concerns.
‍
It may seem obvious and straightforward to ensure alignment on the goal of the AI product: the business wants optimized planning, more accurate forecasts, higher conversion rates. Don't confuse understanding the objective for understanding your user.
‍
User understanding requires more than just great model performance. It also requires good collaboration and understanding between business and data teams.
‍
For example: Your model performs well on a majority of the articles / segments / regions currently in stock, but then a new product is launching this quarter. Now you are presented with a cold start problem, as your model is making predictions with no historical data. Your model was working fine, but if you are working within a business that changes all the time, you will need to take that into account in the design of your system for it to be a solid solution for the business department.
‍
A lot of attention often goes to enhancing business teams so they reach a sufficient understanding of how systems work: what are the inherent strengths and weaknesses of ML models, how to handle edge cases, what features are used?
‍
But there is another side to this coin: data teams should sufficiently understand how the data product will be used. Because achieving that understanding, allows them to make good design decisions. Good design decisions that help their user understand the product, so they can actually work with the product to achieve their objectives. So you should ask your users things like:

• What is the expected output of the AI product?
• How will the product be used?
• What is the business objective you want to achieve?
• What type of person are you? What is your expertise, what are your competencies, your way of working, your vocabulary, your decision making process?
• And what do you need to trust the product?

Mental models

Let’s take a step back and ask ourselves: "How does the user perceive our data product?"

Following Don Norman***, let's look at a user trying to accomplish some goal in the outside world, by performing actions. This is what Don Norman, in his book, the design of everyday things, refers to as the Gulf of Execution. The user is trying to figure out how to use the product, how to interact with it, and what the user can do with it. After trying out some action with the product, the user will then observe the effect. Don Norman refers to this as the Gulf of Evaluation. The user asks themselves: what happened after my input? Do I obtain the expected or desired result? And from there, the user may plan a different action, try out something else, or decide that your product is in fact not helping them at all and they refrain from using it.

When providing the user with some tool to use throughout their activity, the user will try to establish a mental model of how the tool works or behaves so it can effectively be used to accomplish their goal.
‍
A mental model, simply put, consists of causal rules such as: “If I perform action P, the effect will be Q”. Based on this mental model, the user tries to navigate the system, to use it in accomplishing their goal.
‍
The biggest challenge in design is to build a product that can be understood by its users in the context of their task, so they can effectively use it. If we understand how the intended user thinks, in what state of mind they are when they use the AI product, what they are trying to achieve, and what their prior knowledge or skill set is, we can create a design that is intuitive for them to use. We can create the design so that it looks familiar to them, like you see in this example, where the icons resemble real world objects. This replication of familiar things ensures that users recognize the things they need or want to achieve.
‍
The thing is that, if you don’t know your user all that well, you may have certain assumptions about their knowledge or interpretation or mindset. If you do not check such assumptions with your users, your assumptions may be wrong. And if you are wrong, your users may not be able to build the required mental models to navigate and use your product effectively. Not only could this lead to various kinds of user errors, and all the damage caused by said errors. It may also cause your user group to refrain from adopting your product. Because, if the system is not producing value for your user, because, for example, it is not easy to use, or to implement, or cannot be trusted, your users will not adopt it.

Designing for mental models

So, how do we design products that are easy to use and understand? Well we can do either of the following.
1) We can make the system conform to users' mental models (assuming most mental models of your user base are similar).

• An example in the real world is the introduction of the car, where the car initially resembled a carriage but without the horse.
• In a graphical user interface, you might notice that (the majority of) people look for something in the wrong place. A solution then would be to move it to the place where they look for it.
• Or, when you are writing code, or creating a database, and the majority of your users talk about yield instead of revenue, then call that variable or column yield, so they’ll easily find what they are looking for.

2) We help our users create mental models that accurately reflect our design. You can do this by, for example, explaining things better, or letting the design signal to the user how things are to be used. You can add a tutorial, documentation, and offer workshops. Of course, any solution you think of should be evaluated for their effectiveness in actually helping your user build the correct mental model.

• An example of how a design communicates the intended use to the user is the way a handle of a car is designed. When you grab it with your hand, you will naturally close your hand around it and then all you can do is pull.
  
• The other example shows a solution to crappy door handle design. So many times I have pulled a door that should be pushed. You know why? Because the door handle told me to grab it and pull. How much easier would it be when all doors that ought to be pushed simply had a plate instead of a handle? Then all you cán do is push. (This fun video explains it.)
• Now, if we move this to our own product design. We might, for example, allow users to see a preview of results before actually processing the results. This allows them to have a better understanding of what behavior they’re about to trigger, and in what changes that action results. This is especially important when the consequences of the action are wide in scope and difficult to verify.
• And yet another example in our own field would be automated checks, debugging reports and logs, test set-ups, error handling, model monitoring, and so on. These are all built-in features that aid our users to understand what is happening in the code, how the code operates, and why it produces the outcomes it does.
  

Having an accurate mental model of a system often implies a certain level of knowledge and skill. But knowledge and skill aren’t things we were born with - we acquire them. And with new knowledge and skill our perspective changes.
‍
For instance, car mechanics will look at a car differently than a person who has no clue about the internal workings of a car. Knowing what to look for, or listen for, changes our attention and perspective.
‍
And so, if we design a product for a user group, and we make that product really complex or unfamiliar for our target users, even when it is clear that our user group does not possess the knowledge and skill set to understand that complexity, then we must either make it simpler, or we must help them develop the skills and knowledge needed to use our product.
‍
If users don’t understand our product, then we're to blame. We have either failed to explain it well enough, or we have failed to know our user well enough.

Today's blog was about the basics of human-centered design. My next, and final blog in this series, will focus on the design process: How can we talk to and interact with our users to find out what they know, what they need, and what their limitations are?