Product-driven approach to Machine Learning
In many of these tasks it is often difficult to estimate deadlines and hard to predict quality. For example, you are creating a Music App for listening to music. The task itself - to open an audio file, make it audible, and show a list of files, is easy to develop and predict if you have experience. But if you make music recommendations, then everything becomes more complicated. If we're just making an app, we have layouts, so we know very well how everything is going to look. And when we develop a product, we want to understand what will happen and learn how to run everything with some kind of guaranteed quality.
All technologies have a period when they first appear - there are small expectations from them. Then interest increases very sharply, expectation grows, and then a lot of projects start to use this technology. In the next step, everyone starts to be pessimistic about technology. And only after that does a cool moment start when everyone is using technology productively. Everything related to Machine Learning on Gartner’s evaluation has been in the period of growing expectations for 6 years and it will take more time to be fully accepted by society. Everyone is still conservative and afraid to use ML. But in order to be productive we want to reach the plateau of productivity. And to get there not in a few years, but preferably right now.
Some principles should be used to be as efficient as possible in launching all product features related to Machine Learning.
1. Learn how to formulate a problem properly
2. Keep the product in mind at all stages of the solution development
3. Invest time in evaluating the results of the impact on the product
1. Learn how to formulate a problem properly
It is important to formulate clearly what kind of user problem we are solving. If you are dealing with a product, you have to start with a very simple question - do we really need machine learning here? Machine learning often requires a lot of investment, resources, team, and time. And you have to remember that many problems are solved well without machine learning, even if it sounds less exciting.
Pass Discovery Stage for free
If you have an e-commerce project, it’s comfortable for people to sort goods by price/brands/discount/some product features. If it’s some social network with friends - a person can sort people in alphabetical order, messages - by time, etc. All this can be done without ML.
But what if we have a more complicated task - to sort and recommend a user the most interesting posts in the feed? It seems that we have already formulated the task. But in fact, behind this problem, there are many more options. What task is actually to be solved?
Option number one - we make a news feed, formed from the posts of the groups that a user is subscribed to. And that means the task is to rank a few thousand entries from the groups.
Option two - there can be a much more difficult task that may lie behind the same word “a recommendation”. We can search among the several millions of posts that are created on the platform, basically. And what's the most interesting for the user? Solving the second option will take, for example, three times more time and five times more resources. So you will do it not in the next quarter, but at the end of the year. Therefore, if you fail to formulate the problem, you can begin to solve something fundamentally more difficult.
2. Keep the product in mind at all stages of the solution development
Everything we do should fit our specific product. The same technology or solution can be used completely differently depending on the main features and goals of the product.
For example, the classical task of Machine Learning is face recognition in photos. Seems very simple - to find the faces and to understand who is in the photo. But there can also be many options. The user uploads a photo - it should be analyzed and ML should find the friends of the user and automatically mention them. Or there is a more complicated option - to find people among all the millions of accounts, not only among friends and suggest tagging them on the photo. If we solve a complex problem, we will automatically solve a simple one. But we will spend much more effort and get some great code, and cool technology that we don't really need in the product, and everything could have been done faster.
According to the Pareto principle, 20% of our efforts give us 80% of the result. And our hardest task is to understand, whether 80% of the result is enough for us, or do we need 100%. In the case where we're just tagging friends, those 80% results that you can get quickly are enough - that's what our product needs. If it's authorization, or something related to payments and finances, then we need 100%.
3. Invest time in evaluating the results of the impact on the product
It's always necessary to measure user metrics. If we're talking about facial recognition, it's interesting to look at graphs of how often people agree with our friends' suggestions, and how often people click "No, that's the wrong person." If we just know that this model is 80 % accurate, that information means almost nothing to the product. But if we see a user cancel our suggestions every 10th time, then we're probably wrong too often.
How do you know when it's time to change a metric? An important signal is when one metric is going up, some other product metrics start going down. That means that something is out of sync, and it's time to think about it. Or when we see that the metrics are growing, but it's subjectively uncomfortable for us to use them. Or when we see calls to support and users complain about something, and there is a metric that keeps growing. So maybe it's time to change. In any situation where you see some kind of mismatch between metrics and reality, you need to check if everything is good.
There are three levels of algorithm launch into production.
Testing on the % from all the users.
Setting up criteria of:
- launching an experiment
- switching the algorithm off
- feature failure
- expanding on all the users
Additionally to first option:
- choosing the minimal test group
- setting criteria for expanding the test group
- creating experimental management infrastructure
- maximizing the number of hypotheses tested per unit time
- launching few algorithms on different test groups
- choosing the optimal algorithm
Users usually don't know which algorithm is responsible for what action. For them, it is all hidden and quite convenient. In this case, you can always include a small percentage of users for the start. If we understand that users’ experience became worse, turn it off. It is very important to prepare criteria according to which we will understand that our hypothesis did not work at all, and it is no longer necessary to move in this direction and it is not worth investing more time in it. And if everything is great, you can turn on the algorithm for all users.
2. ALGORITHM + UI
The second level is when we can no longer experiment unnoticed by users, and the user sees that something is going on. We need to figure out what is the minimum test group to include. It could be a separate city, a separate country, a separate region, or a separate age. And what's important here is the infrastructure to manage this tool and to turn it off quickly, if anything goes wrong, because it can attract attention.
3. POINT OF NO RETURN
And the most complicated and fascinating option is when we launch something publicly, and then it has to work right away. It is useful to prepare several variants of the algorithm at once because if you make one algorithm, run it and see from the charts that it works poorly, then you can only get frustrated and lose users. If you had three algorithms run in the test at once, you can quickly figure out which one is the best at the moment and include it in most of the audience, and leave the other two already in a small experiment.