Product-driven approach to Machine Learning

Many types of Machine Learning can be used in your product. It can be a smart newsfeed, recommendations section, computer vision or the whole solution based on AI. When you decide to add Machine Learning to your business, you should know about some fascinating specifics.

In many of these tasks it is often difficult to estimate deadlines and hard to predict quality. For example, you are creating a Music App for listening to music. The task itself - to open an audio file, make it audible, and show a list of files, is easy to develop and predict if you have experience. But if you make music recommendations, then everything becomes more complicated. If we're just making an app, we have layouts, so we know very well how everything is going to look. And when we develop a product, we want to understand what will happen and learn how to run everything with some kind of guaranteed quality.
startup robot
So let’s discuss how to make Machine Learning features more predictable and implement them in your product in the most effective way.
Hype Cycle of technologies

All technologies have a period when they first appear - there are small expectations from them. Then interest increases very sharply, expectation grows, and then a lot of projects start to use this technology. In the next step, everyone starts to be pessimistic about technology. And only after that does a cool moment start when everyone is using technology productively. Everything related to Machine Learning on Gartner’s evaluation has been in the period of growing expectations for 6 years and it will take more time to be fully accepted by society. Everyone is still conservative and afraid to use ML. But in order to be productive we want to reach the plateau of productivity. And to get there not in a few years, but preferably right now.

startup robot

Some principles should be used to be as efficient as possible in launching all product features related to Machine Learning.

1. Learn how to formulate a problem properly

2. Keep the product in mind at all stages of the solution development

3. Invest time in evaluating the results of the impact on the product

1. Learn how to formulate a problem properly

It is important to formulate clearly what kind of user problem we are solving. If you are dealing with a product, you have to start with a very simple question - do we really need machine learning here? Machine learning often requires a lot of investment, resources, team, and time. And you have to remember that many problems are solved well without machine learning, even if it sounds less exciting.

Wanna know if your project needs ML?
Pass Discovery Stage for free
Let’s see the example.

If you have an e-commerce project, it’s comfortable for people to sort goods by price/brands/discount/some product features. If it’s some social network with friends - a person can sort people in alphabetical order, messages - by time, etc. All this can be done without ML.

But what if we have a more complicated task - to sort and recommend a user the most interesting posts in the feed? It seems that we have already formulated the task. But in fact, behind this problem, there are many more options. What task is actually to be solved?

startup robot

Option number one - we make a news feed, formed from the posts of the groups that a user is subscribed to. And that means the task is to rank a few thousand entries from the groups.

Option two - there can be a much more difficult task that may lie behind the same word “a recommendation”. We can search among the several millions of posts that are created on the platform, basically. And what's the most interesting for the user? Solving the second option will take, for example, three times more time and five times more resources. So you will do it not in the next quarter, but at the end of the year. Therefore, if you fail to formulate the problem, you can begin to solve something fundamentally more difficult.

2. Keep the product in mind at all stages of the solution development

Everything we do should fit our specific product. The same technology or solution can be used completely differently depending on the main features and goals of the product.

For example, the classical task of Machine Learning is face recognition in photos. Seems very simple - to find the faces and to understand who is in the photo. But there can also be many options. The user uploads a photo - it should be analyzed and ML should find the friends of the user and automatically mention them. Or there is a more complicated option - to find people among all the millions of accounts, not only among friends and suggest tagging them on the photo. If we solve a complex problem, we will automatically solve a simple one. But we will spend much more effort and get some great code, and cool technology that we don't really need in the product, and everything could have been done faster.

In addition to explaining what kind of problem we are solving, it is also important to formulate what our quality requirements are. In classical tasks that are not related to machine learning, the quality requirements are often clear. If we're sending messages, the message has to be delivered in 100% of cases. If we're talking about facial recognition, and text recognition, we don't have algorithms that will work 100% of the time. And the challenge is to look at the product and decide what's the right quality for us. In the facial recognition example, it can be two very different tasks. The first case is when a user uploads a picture and it's great if he can tag his friends automatically after a few seconds. And the other case is when the user picks up his iPhone to authorize through face recognition. In case we make a mistake when we're tagging people on the photo, it can be unpleasant, but nothing critical will happen. If we are talking about authorisation into the phone, any mistake is super critical because it's personal data that we gave access to.
According to the Pareto principle, 20% of our efforts give us 80% of the result. And our hardest task is to understand, whether 80% of the result is enough for us, or do we need 100%. In the case where we're just tagging friends, those 80% results that you can get quickly are enough - that's what our product needs. If it's authorization, or something related to payments and finances, then we need 100%.
startup robot

3. Invest time in evaluating the results of the impact on the product

It's always necessary to measure user metrics. If we're talking about facial recognition, it's interesting to look at graphs of how often people agree with our friends' suggestions, and how often people click "No, that's the wrong person." If we just know that this model is 80 % accurate, that information means almost nothing to the product. But if we see a user cancel our suggestions every 10th time, then we're probably wrong too often.

It is very difficult to choose the perfect metric on the first try. For example, the social networking service "Recommendations of friends" seems quite simple. The metric is - the more friend requests we send, the better. But if we send requests and no one responds to them or users accept requests but don't communicate? Metrics will become more and more complicated in order to reflect the product better. It is better to tune into interactivity right away because initially many things cannot be predicted in advance.
How do you know when it's time to change a metric? An important signal is when one metric is going up, some other product metrics start going down. That means that something is out of sync, and it's time to think about it. Or when we see that the metrics are growing, but it's subjectively uncomfortable for us to use them. Or when we see calls to support and users complain about something, and there is a metric that keeps growing. So maybe it's time to change. In any situation where you see some kind of mismatch between metrics and reality, you need to check if everything is good.

startup robot
Launching ML solutions in production
Our goal is actually to get our experiment into production as quickly as possible, to the real users.

There are three levels of algorithm launch into production.

Testing on the % from all the users.

Setting up criteria of:

  • launching an experiment
  • switching the algorithm off
  • feature failure
  • expanding on all the users
Algorithm + UI

Additionally to first option:

  • choosing the minimal test group
  • setting criteria for expanding the test group
  • creating experimental management infrastructure

Point of no return
  • maximizing the number of hypotheses tested per unit time
  • launching few algorithms on different test groups
  • choosing the optimal algorithm
Users usually don't know which algorithm is responsible for what action. For them, it is all hidden and quite convenient. In this case, you can always include a small percentage of users for the start. If we understand that users’ experience became worse, turn it off. It is very important to prepare criteria according to which we will understand that our hypothesis did not work at all, and it is no longer necessary to move in this direction and it is not worth investing more time in it. And if everything is great, you can turn on the algorithm for all users.

The second level is when we can no longer experiment unnoticed by users, and the user sees that something is going on. We need to figure out what is the minimum test group to include. It could be a separate city, a separate country, a separate region, or a separate age. And what's important here is the infrastructure to manage this tool and to turn it off quickly, if anything goes wrong, because it can attract attention.

And the most complicated and fascinating option is when we launch something publicly, and then it has to work right away. It is useful to prepare several variants of the algorithm at once because if you make one algorithm, run it and see from the charts that it works poorly, then you can only get frustrated and lose users. If you had three algorithms run in the test at once, you can quickly figure out which one is the best at the moment and include it in most of the audience, and leave the other two already in a small experiment.
The charts, unfortunately, don't tell us everything, although we really wish they did. Let's say we launched the algorithm and most of the audience got noticeably better. They started spending 1,5 times more time inside the product. But 20% of the users got significantly worse on most charts. But at the same time, these 20% of users who have become very uncomfortable with a new algorithm, they may simply leave you in a month, and the number of unique users will fall. So it's not enough to look at one or two charts. Support can help you in this case. Users like to tell you that something went wrong. The main thing is to learn how to work with this large unstructured flow of information. If you create a community around your product, active users will always help you tell your team to understand in which direction to move.
Now to sum up everything. It is important to formulate the task correctly. It is important to set the right quality requirements so that it does not accidentally turn out that we are solving the wrong task for the users. Remember about metrics and remember that all the stages of solving a task are important, even quite routine and spend a lot of time to understand how it all works on real users, not only guided by theory, but by some of our preliminary estimates. It's important to remember that in order to make cool things, you have to experiment, you have to invest time into it. So let's all make cool products together.
Wanna use Machine Learning in your business project? We can help.