Managing AI-Powered Products — Important Principles
There has been a lot of talk in the recent years about the use of AI capabilities or machine-learned solutions to improve digital products or services and user experience. If we can cut through the hype and have the necessary building blocks in place as covered in Four Hurdles To Creating Value From Data, there is legitimate value to be had from the use of data and AI to make better products in many verticals. For tech companies that are founded on strong engineering practices and find themselves knee deep in data such as Uber, Twitter and Tesla, the use of ML to extract value is already part of their culture. As for other companies facing a different set of circumstances and yet, still see the need to adopt data and AI to remain competitive, these companies may not know where to start or underestimate the mindset shift needed.
The previous CEO of GE, Jeff Immelt shared from his experience during the digitisation of the industrial world, that their transformation journey involved more than just software and technology. He emphasised that success was predicated on the various roles along the value chain such as product managers and sales people to be thinking and operating differently. While the context was different back then, many companies now find themselves in a rather similar situation trying to leverage big data and AI — a situation where the return on the investment depends on more then just hiring a few data scientists and buying and rolling out the first “AI technology” that they come across from vendors.
In this short article, we turn our attention to product teams and their embrace of data and AI to make better products. We discuss the key differences in what makes up an AI-powered product and how teams should approach the building and managing of their software solutions. We begin by understanding how (or where) AI fits in to the broader make-up of software products. We highlight the single biggest difference in the make-up of an AI-powered product when compared to more typical applications. This distinction is what unlocks the value from data as discussed in Investing in Data and AI — When and Why. We conclude with a discussion on three principles that product managers and their cross-functional teams have to keep front of mind when thinking and planning. These principles exist to help reinforce the virtuous data cycle that fuels AI capabilities behind the products.
Difference with AI-powered products, structurally
The pyramid diagram above forms the basis of our quick discussion. The actual number of layers, the terminology and the definition aside, it is rather common to visualise and discuss the make-up of an application or software product in this logical fashion. If you have a background in designing and building software, a multi-tier diagram such as this should be all too familiar. The purpose or composition of each layer is described in the diagram. At a high level, the UI layer is what the users of your product see and interact with. The actual ability to execute commands, evaluates the input and produces the output resides in the logic layer. The base data and core services layer is what everything else sits on. If the UI is the sensors and the looks, then the bottom two layers are the brain.
The second diagram above shows where the AI capabilities tend to fit in to an application. The layout above is based on my experience of leading teams and being intimately involved in building and incorporating prediction, enrichment, and search and recommendation capabilities into software products for both consumers and enterprises. As you must have noticed by now, the single biggest difference between the two diagrams above lies in the logic layer.
The first part of the difference involves the gradual and thoughtful replacement of aspects of the existing logic and workflow that can be automated, simplified or optimised with models learned from data. For instance, assume that you currently rely on heuristics to suggest other items to your customers after they have purchased something (e.g., if a cheap item in category X was purchased, try to cross-sell a randomly selected cheap item in category Y). This logic is a good candidate to be machine learned and has the potential to improve with more data. In another example, imagine there is a step involved in the content posting flow whereby humans review the content to ensure certain fields conform to a standard. This step can be (semi-)automated to reduce the wait time before items can appear in the marketplace.
The second part of the difference is about introducing new capabilities to make the application behave smarter in ways that were not previously possible or feasible due to the intractable nature of the solution using human defined rules or other reasons. For example, a seller can be given real-time suggestion about the category and the price for the item they will be posting to maximise the chances of attracting buyers. Prediction services can also be developed to infer shoppers’ interests and preferences which can in turn be used in recommendation engines to customise the items the users see.
Principles for managing AI-powered products
In addition to the distinction in the architectural sense of AI-powered products, how product managers and their teams think about building and moving their products forward will need to adapt as well. In this last diagram below using the same pyramid structure, I highlighted three important principles that product teams need to consider and practice. These principles are here to help unlock further potential of the underlying AI capabilities of the product.
Over the years, I have come to realise that while they are rather intuitive for product managers with a strong data science and engineering background, these principles tend to be unfamiliar to the many others. This can create misalignment between product managers and the data scientists they work with and hampers their effort to solve user problems with AI and data. This was discussed at length in Are You Able To Match The Right Data Science Solutions To Problems? In my opinion, the adherence to the principles to be discussed here is what sets apart product teams that are prepared to leverage AI from the rest.
The first principle as spelled out in the diagram above is, if there is a hunch that an existing AI capability can be improved with additional data points collected from the users, we should consider asking for them. As an example, ML models can get better with feedback from users. Imagine that there is a search product that currently allows the users to flag certain results as not suitable for them. This signal is already being used to tailor the items that the users see. If the data scientist working on it has a strong hunch (and potentially some evidence) that the ML models can improve by getting finer-grained negative feedback from the users, then the team should seriously consider it. This way of thinking and the necessary conversations between the data scientist, the product manager and other team members do not happen often enough. The reasons can be many. The common ones include (1) the data scientist can be reluctant to present the idea because they do not know how or fearful about the response they may get, and (2) the product manager is simply unaware of the opportunities or what data gaps are there to be filled.
The second principle is, for every interaction that we permit the users to carry out with the software, we should think about how the data can be used to improve the experience, not only for themselves but also for others. This covers both the data that the users explicitly provide and the exhaust data from their use of the software, with their consent of course which is typically part of the terms of use. With more traditional software products, the value of data is limited to (1) supporting the proper functioning of the human-defined rules in the logic layer and any associated business processes that sit outside of the software, and (2) product analytics, reporting and perhaps A/B testing. Hence, it is not surprising that this principle can be overlooked by product managers or even engineers. If the data scientists are not involved early enough during the design phase, it is unlikely that the team will practice this principle. Moreover, the lag time between figuring out value creation ideas from the data to actually seeing the ideas coming to life can be long. The issue compounds when the product managers do not maintain longer term views.
The third and last point, which can be more of a reminder than a principle, is to always be cognisant of any dependencies on the data and infrastructure layer. This calls for tight collaboration between engineers, architects and data scientists throughout the designing and development of AI capabilities. For instance, if an algorithm requires access to a graph database to find the nearest neighbours given an item, is that infrastructure readily available? In another example, imagine that you have an ML model that predicts the category for an item, is it more appropriate to use the model to categorise new items posted by sellers on-the-fly or in bulk. While the first two principles may call for close partnership between product managers and data scientists, this last one calls for strong engineering practices.
