Real-life AI: Your guide to building a useable product
What is the difference between academic and commercial Data Science? Academic Data Science is like competing in a Formula 1 race. Your driving skills has to be top notch and your car must be flawless. But also you do not have to think about where your destination should be, which route should you choose or pedestrians jaywalking.
So, welcome to commercial Data Science: the everyday driving in the crowded city.
It was at an incredible startup where I kick-started my industrial career in Data Science 6 years ago. That was my first industrial job after 4 years in academia and research. When I decided to switch career paths, I simply thought that it will only be a domain shift. After all both are data science, right? Boy, was I naive!
After reflecting on the “5 things I wish I have known about real-life AI”, one of the most important point that I had to learn was that “doing the right thing != doing something right”. Given the feedback and comments that I have received, here is a dedicated article to to provide you with a practical guide of “doing the right things” in commercial AI projects.
The leaking kettle
Here is a question for you: If you would be creating a brand new kettle, what are the features that you will include?
Before moving to Düsseldorf, I decided to sell the furniture to the next tenant and start fresh in the new city. When I moved in to a new flat with basically nothing but my clothes and an air mattress, I knew that furnishing the place will take sometime. But who cares, fresh starts have their own beauty. After a good first night sleep, I went to the kitchen to prepare my coffee. I searched in the boxes for the kettle. Unfortunately, the next tenant was enjoying her morning coffee using it. I left the kettle back in Bonn. So I went to one of those hypermarkets, grabbed my groceries and a random kettle and went back home.
After using the new kettle for a couple of time, I noticed that the countertop is wet. A couple of days later, I realised that the new kettle leaks each time the water inside exceeds a certain limit. But living in a completely empty flat and having to use my phone as my light source did not exactly prioritise the kettle to the top of my todos.
So, as a temporary plan B, I just have to be careful how much water I put inside. Well, no, I also have to be careful not to place the kettle on its electric base after using it. Well, maybe be on the safe side and unplug the kettle when not in use. Oh, did I unplug the kettle today before I leave or would I to back to find my place burned down?
Now back to the original question: If you would be creating a brand new kettle, what are the features that you will include? If you are like most Data Scientists, speed and energy efficiency of the kettle would have been your list. And that is it. The problem here is, even though the speed and efficiency are indeed very important, your customer would much rather choose a mediocre kettle that doesn’t leak, rather than a very efficient kettle that leaks.
Commercial AI: the 3 things that you need to get right
Here is a fact that we all know, a high percentage of commercial Data Science projects fail. And most of the time the reason behind the failure is not that the model’s accuracy is too low. Despite that the accuracy is crucially important for the success of your product, the accuracy is not the only factor that determines your projects success.
So what are the properties that you need to create to have a successful AI project?
1. The core features: Your purpose and final destination
2. The common-sense features: The unspoken, no-brainer and trust-breakers
3. The relatable features: From good to great
1. The core features: your purpose and final destination
What should a kettle do? It should heat the water up.
What are the core features?
Those are the main goal of your project. Out of the 3 sets of features, the core features are the most obvious, and the easiest one to define.
How to define your core features?
A key to a thorough definition of the core features is asking your customer detailed questions about what they need from the system. Break down their requirements to a set of detailed list. For example, your customer has requested you to build an Email classifier system. Understand how would the actions offer from one class to another. Ask them about if there is a hierarchy to the classification. Enquire about the different departments handling the various classes. Clarify if they expect that one email could belong to multiple classes. Understand the importance and sensitivity of some labels over the others.
Why to granularly define your core features?
Each of those questions would change how you translate the business problem to a machine learning model. For example: Would it a single or a multi-class problem? Should you implement a custom loss function to balance the different accuracy vs precision expectations for the different classes? How should the threshold be set, below which you would not assign any predictions?
You see, the more you understand the details of the system, the better you would be able to design your model. So make sure you ask as many detailed questions as you can.
2. The common-sense features: the unspoken, no-brainer, trust-breakers
If someone would ask you: how would your dream kettle look like? Would you ever answer with: “A kettle that doesn’t leak.”? On the other hand, would you use a kettle that leaks?
What are the common-sense features?
Those are the features that ensure that your system is working in a controlled logical manner.
For example: you are building a dynamic pricing system for an e-commerce website. Your goal is to build a system that assigns the optimal prices for the items as to maximise the final revenue. Would your system assign a negative price for an item? Could the total price of 3 items be less than a single one? Are the assigned prices stable enough, or would they change by a factor of 3 when the user hits refresh? Do you see how those features are not directly related to the accuracy of your model?
Why should you include those features?
Your customers assess what they don’t understand based on what they do understand. If the basic logic of your system is flawed, why should they trust that your complex model isn’t flawed as well? As a customer yourself, would you every buy an oven from the same brand that created that leaky kettle?
How to define those features?
As intuitive and common-sense those features are, as challenging as they are to define. Most of the time, your customer will perceive those features as too obvious to be discussed or explicitly mention. And let’s face it: it is boring to properly define and implement your ‘anti-leakage’ features. Indeed, if there is an Emmy’s prize for the most boring task that could be ever be done, it will go to the anti leakage features. But remember, if there is a Grammy award for the trust-breakers, it will also go to those features. So, here are two practical tips that will help you in defining the common-sense features :
1. Shadow your user: If possible, grab a chair and observe your user in their day-to-day job for some time. Focus on what they do in their “auto-pilot” mode; the things they even won’t explain because it is too obvious. Write those down and aim for the most extensive, comprehensive and boring list you can come up. It will pay off at the end.
2. Use your system as your user: Use the system as your customer would do, before your use does. If their auto-pilot is indeed a no-brainer, you will also see if the basic logic is broken.
Spoiler-alert: if your customer opts-in for a very primitive solution ratherthan your cutting-edge AI system, check if your system needs some basic “anti-leakage” functionality. And do not underestimate the effort it would take to build back the trust.
_______
3. The relatable features: from good to great
How does my dream kettle look like? It is a kettle that can listen to my alarm clock and knows my non-optimal snoozing habit, so it will start heating the water early on, for me to wake up to find the water boiling and ready for my coffee.
Objectively speaking: this feature will save me 1 minute a day. And given my non-optimal snoozing habit, this feature will also reduce the energy efficiency of the kettle. So — objectively speaking — my dream kettle is not better, if not worse, than a regular kettle. But subjectively speaking: I am not after saving 1 minute of my time, but rather a feeling of connection with the smart appliance I own.
What are the relatable features?
The relatable features are the non-core features that extend your main goal to create a more relatable and likeable system.
Taking the example of the dynamic pricing for an e-commerce: An attachment feature could be an ad hoc analysis that shows which products that have the highest price changes from the manual process to the AI one. Maybe you would find out that there is a supplier providing forgeries rather than original brands. Or maybe you would see items that were super trendy a couple of years ago, converse shoes for example, that are still priced in the same range as it was, despite the lower demand. Maybe the black dresses are assigned a higher price, as those are more popular with the end-users more than the pricing experts expected? Do you see how those questions would create that feeling of connection between your customer and your model?
Why should you include relatable features?
Presenting such results to your customer will transform the mere Euros that your system is assigns to the items, to a relatable, understandable and likeable system that your customer trusts. And when your customer likes and trusts your system, they will much more likely use it and integrate it into their daily business.
How to define those features?
Similar to the common-sense features, the relatable features are usually not discussed. And most of the times neither you nor your customer would be aware what those would look like. You need to be creative. And here are 2 guidelines that might inspire you:
1. Think mirrors: Your customers have assumptions how their business and processes work. And often times those assumptions are different from the data. So, use the data at hand as a mirror: validate and extend those assumptions. Do their expectations truly meet the reality? Extend and correct their knowledge. Not only would such analysis deepen your customer’s understanding of their own business, but they will also build trust in your black-box models. Remember: customers assess what they don’t understand based on what they do understand.
2. Think emotions: Working with data is a job that is filled with emotions and surprises. How many times did finding out a hidden pattern excite you? We all do know the frustrations with the data quality. That anticipation that we feel when the system goes live? Or the happiness when the results exceed the acceptance criteria? Don’t keep those emotions to yourself, but carry them over to your customer. As analytical people, we usually miss out on utilising those emotions. But emotions are the root of connection. So share those emotions with your customer, it will deepen their connection to your AI system, a level deeper the pragmatism and intellect.
To conclude:
It is completely natural for us as data scientist to focus on the model’s accuracy. But remember, even though the accuracy is crucial, it is not the only factor determining the success of your AI project. You need to design a system that accurately addresses the core task, behaves in a logicallycontrolled manner, and a product that your customer could relate to.
And now it is your turn: which one of the previous sets of features do you feel that they are the most underrated? Leave your thoughts in the comments below
