Successful AI-NLP Project - What You Need to Know • Robert Wcisło, Łukasz Matug • Devoxx Poland 2022

Hello, everyone. Welcome to our presentation today. Robert and I would like to discuss the artificial intelligence project and what you need to know to make it successful. To provide some background, Robert and I are part of the AI and Data Science team at UBS. We have been working together for the past two years, focusing on implementing artificial intelligence solutions within a large investment bank. In our roles, we have been involved in numerous projects, primarily in the NLP space, but not limited to that area. Through these experiences, we have gained insights into the critical success factors necessary to take these products into production.

Today, we would like to provide you with some clues and information about which areas to consider before starting such a project to increase your chances of success. Let's begin. Many companies start from a place of inspiration. Someone might say, "Okay, we want to do something with AI," or perhaps they've heard that AI is a fantastic tool that can solve all the problems in their business domain. This is where our mission begins. Everyone is excited, and the developers often express enthusiasm, saying, "Oh great, we have the opportunity to test this approach!"

Everything looks smooth. In the beginning, for most of the projects, everything seems fine. However, at some point during the mission, we encounter a problem. This problem indicates that we are not heading in the desired direction for the project.

Based on our experience during delivery, we have observed that machine learning projects often struggle to reach production. According to data from 2018, only 10 percent of machine learning models successfully reach production status. This means that in nine out of ten cases, the projects that individuals work on do not achieve this state of bringing value to the company. While it is likely that the current numbers are slightly better due to advancements since 2018, many challenges still lead to the failure of machine learning products. In this presentation, we will discuss key considerations for ensuring the success of our mission. We will also outline a simplified process for deploying machine learning models into production.

And, of course, this is a simplified process. This is not something that is straightforward; the real product is a bit more complex. However, for the purposes of the presentation, we can break this down into three phases: the Prepare Phase, which occurs before you start working on any code; the Build Phase, where someone is working with the data to prepare for the project's launch; and the Maintain Phase, where you strive to keep the solution robust and fully operational in production without any issues or harm.

Below that presentation, you have these two approaches, which SDRC is probably common for most of the view. If you are a developer, you have the normal Software Development Lifecycle (SDLC), plus now we have the younger brother of the SDLC, which is Amelops. Regarding the lifecycle of the models, you should consider both of these approaches during the process. On top of that process, you have the user experience, which means that everything you are working on and producing as a solution should ultimately bring business value and be beneficial for your customers and users.

This is not equal, but in the end, you should achieve good results. I focus on the preparation phase, while Robert focuses more on building and maintaining the project, as well as providing summaries. I believe the most important clue for achieving a successful model or solution regarding an artificial intelligence product in your domain is understanding the fundamental requirements. In the beginning, someone in the company might say, “Oh, we should use machine learning and make magic happen.” However, in many cases, the people who are not familiar with machine learning attend conferences or explore different business areas, especially if they come from a business domain or tech management background. They may think, “This is something we should try.” When considering any kind of machine learning project, it's crucial to think about the real output of the project and what you should ultimately achieve. Focus on what the application should do, rather than getting caught up in the technical aspects of the project.

In most cases, especially if you're working in a corporate or enterprise environment, machine learning is often narrowed down to labeling. For example, in a fraud detection system, the system ultimately determines whether a transaction is fraudulent or not, relying on labeled data at the end. It's important to understand the customer who is coming to you, the product they bring, and the business need behind it. Consider how the model or solution should be presented at the end. By clearly defining these aspects and simplifying the initial stages, you can address the problems effectively. Many people often say, "I have a problem," but it's crucial to dig deeper into understanding the needs and requirements.

I don't really know what I'm expecting, but I hear that learning can solve that problem without the definition of that one. Try to make it more explicit what you like to do with this kind of application. If you understand what you enjoy doing and recognize that machine learning is mandatory for your project, it's important to note that machine learning is beneficial for some areas of problem-solving but not for all. Consider this as a clue of what you would like to do; you should reflect on how we can help you navigate whether this is a potential machine learning product or if we should conclude that machine learning is not a suitable solution here. Some applications of machine learning include recommendations, language understanding, anomaly detection, and personalization. In cases where you are working with a large dataset and the heuristics are too complex to create and maintain, machine learning may indeed be a viable solution. Machine learning works very well when you are aiming to assist users in achieving their goals within the application or helping customers support those users, but it should not completely replace humans.

I will speak more about this in one of the next slides, but this is a potentially good area for machine learning. Additionally, thinking about the scale, if someone says, "We have one of our problems and we would like to trigger this one time," it may not be worth the complexity of putting a machine learning model into production for that kind of product. In such cases, it might be better to simply use analytics, consult a data scientist, and prepare a report for management or the customer; that could be sufficient. It's important to consider the potential benefits as well as the yellow or red flags when thinking about a machine learning project. It's also good to ask people if they understand machine learning, particularly that it does not always guarantee 100% accuracy.

The models always have some errors and mistakes. If you want to minimize these mistakes, creating that kind of model becomes more expensive because it requires more data, more tests, and convincing your business stakeholders that the model may not always provide a perfect match. If transparency is essential for the logic behind the model, this can raise potential yellow flags, especially since some business areas are highly regulated and expect everyone to understand what is happening in the application. When using models, there is often a need to clearly explain why the model behaves in a certain way. While you can incorporate some level of expandability, it may not produce a perfect representation of business logic. For example, using conventional programming paradigms allows for coding the logic, but you must also consider the data you have available.

Because without the data, you cannot have machine learning. And no ML means no data, no ML. Of course, you can provide workarounds in that kind of situation. You can buy data on the market, or you can utilize some transfer learning.

And, for example, consider taking a model from the market. You have a pre-trained model. Everything can be managed here, but when you think about its significance, you have enough data to work on that kind of project. If you say that machine learning is something that can potentially assist you in evaluating whether you are measuring the right outcomes in that kind of application, I refer to this as a universal performance metric. This metric is not solely for measuring model performance; it's about understanding your application and the business goals you aim to achieve. The business expects you to leverage this machine learning model to determine whether we are doing the right things or not. For instance, this could be the click-through rate if you have an application designed to encourage users to take specific actions.

And this is information. We're addressing the issue of people consuming more material and content from your webpage. You can consider recall and precision, for example, in the context of medical images. If you want to determine whether a medical image indicates the presence of a disease, you might evaluate the recall and precision. In a situation where you identify a potential image with a disease, these metrics can be invaluable. This approach is particularly helpful when you're creating such projects. Firstly, it allows you to differentiate outcomes; if you change your solution—something that happens quite often—you can conduct tests to see if we are yielding positive results or not.

The second issue that often arises in these types of projects is that the different stakeholders do not share a common definition of success. This also applies to the standard software development life cycle, where someone may deploy a model or an application to production, yet nobody utilizes it. Some people may consider this a success while others do not. Having a clear metric for success is beneficial because it ensures that everyone involved in the project understands success in the same way. Additionally, a positive aspect of having a performance metric is that when you are preparing the backlog and determining potential next items for development, you can assess how to align with this universal performance metric.

And if you say, "Okay, this is a potential big gain for the universal performance metrics," not all cases can be classified as our biggest effort for development or for data scientists. Maybe this is a quick win, but if you achieve a significant gain in universal performance metrics, you will see substantial cost efficiency from that kind of product. To establish this, consider the common tooling and infrastructure. When working with machine learning applications, you are bringing two worlds together: the software world, which involves people working with applications, and the data science world.

I think it’s on a slightly different state of maturity, for example, in terms of processes. Now, MLOps is something that is more commonly applied in environments where the full life cycle of the model is considered. However, it is still important to have a common vision of how we would like to work on the project together. Of course, using a Git repository to share code is essential, as is sharing the architecture of the model, which is one of the basics. Access to the datasets is also crucial. All team members working on the project should have access to the necessary data required for this process.

And sometimes it happens in a situation, for example, regarding the use of production data when you're working on a machine learning solution. Very often, you need to work with production data to ensure a good representation of your model. The model should function on production data, so you should have a subset of it. However, for various reasons, some people may encounter regional constraints. For example, you might not be able to use data coming from different countries due to cross-border restrictions. It is beneficial to address these issues at the beginning. Everyone involved with the model should have access to the experimental results. If you are working on the model and preparing different versions, it's important to document the information about your experiments. If you encounter failures, others working on the same problem can refer to this log and know, "Oh, I don't need to test this hypothesis again, as I can see it has failed before." This practice increases the efficiency of the product development process and helps develop a well-functioning model more quickly. Documentation is crucial; you should utilize tools like Confluence or other platforms where you can store information about the product itself.

Sure, here is the revised text with correct capitalization and spelling: --- However, access to previous projects and the data derived from them can be invaluable. If you have any experience in your company with similar projects, it is wise to review that information first, as preparing the computations for the models can be very expensive in most cases. This is one of the most costly aspects of the project, and it also carries a level of uncertainty. When working with a model, you typically cannot determine from the outset whether it will be successful or not. To reduce this uncertainty and increase the chances of success for your project, it may be best to start by examining similar products from the past. By reusing the knowledge and experience you have acquired previously, you aren’t starting from scratch; instead, you build upon the foundation established by earlier industry projects. Additionally, consider your access to infrastructure, particularly if you need GPUs for deep learning tasks. First, evaluate whether you have this type of access concerning the data size you are working with. Secondly, even if you plan to use cloud-based GPU access, ensure that you have the budget for these experiments. As you start exploring different hypotheses, project costs can increase significantly. --- Feel free to let me know if you need further modifications!

Here’s the reformatted text with correct capitalization and spelling: "And in some of the reasons, somebody might say, 'Okay, maybe we would not like to invest so much time in the problem we are solving compared to the cost required to prepare that kind of model.' Moving on, Okay, I think this is batteries. Okay, something is wrong with this. Oh, okay."

Next-Level Work Next Area to Consider: Level of Automation. If you're thinking about a machine learning project, consider what the expectations are and how you would like to replace or assist humans in that project. As a baseline, consider the human effort involved, which means we start with no AI.

We have people who are making some of the actions and decisions, and we are not involving any artificial intelligence in this process. The second level involves using artificial intelligence as assistance; in this case, the AI model provides clues about what the person might select, but a human always triggers the final action. For example, consider a recommendation engine: if you have Netflix and you see all the potential movies to watch, the final decision is made by the human. I triggered the action by selecting the movie I want to watch. Partial automation is a situation where, in some cases, the model decides for the human, while in others, it does not. For example, in a fraud detection system, the model might indicate, "We are almost completely sure this transaction is not fraudulent," and no action would be required from the human in that situation.

We can say that we are not sure; the model is not certain about that. In other words, we are unsure whether this is fraud or not. There are some levels or thresholds of certainty regarding this potential fraud. In that case, you should forward this item for action by a human.

The last part is full automation, where you allow AI to take complete control over your decision-making processes. In this phase, the AI manages everything, including preparing models and scaling up infrastructure. This can lead to a scenario where the model always makes decisions on behalf of the human operator.

And, of course, you need to consider that full automation is not always the best approach, as it may not be worth pursuing in all cases. Instead, select the appropriate level of automation at the beginning of your project. This is a valuable consideration to make early on. It is important to ensure that your applications do not negatively impact anyone using that solution.

We selected this area for fairness. This means that when you are preparing the model, if you exclude certain groups of people from your data, the final model will also exclude those groups. It is important to avoid this kind of bias and data restriction. If you are working with data from different countries, consider these aspects from the beginning. You may initially think you can use that data, but in some cases, you have to consider regional constraints such as GDPR and many other regulatory requirements.

At the end, you might say, "Okay, we have the perfect model," but you cannot use it for certain reasons. For example, the server is located in the wrong country, which prevents you from processing that kind of data. Compliance issues often arise, and this is something that mostly affects large companies. They have their own frameworks for managing the model lifecycle. Similar restrictions apply to data. You may have the model, but if someone from compliance says, "Sorry, you cannot use that because it hasn't been reviewed beforehand," there will be obstacles to moving forward.

It's essential to consider compliance at the outset because this type of regulatory issue can be complex and may increase the time required before the model goes into production. Ethics, while more of a philosophical statement about the solution, is crucial, especially if you are thinking about scaling the application. For instance, if you intend for this application to be used by millions of people, it's vital to consider how not to harm them in any way. Although this concept may seem somewhat vague, addressing these ethical questions early on will be more effective than waiting until the application is already in use. Model authorization is the next factor to consider; it's about how you plan to use the model and how you want to serve it to users. This approach reflects the various ways you can interact with and implement the model.

The binary version of the model means that you have the model available for everyone to download. In this case, you're not considering user experience, scalability, or observability. People can take the model, run it on their own machines, and see the logs. The second potential pattern involves serving the model as a service for other technical teams in your company. This means that you don’t need to focus on the user experience.

You have, for example, the dockerized version, the pack model within the sum of the docker. And, for instance, in Kubernetes, this is a scalable app. You can provide observability and access control for that kind of model. You can create a new solution, which means you embed the model in the entire application.

In that case, you need to think about the user experience of this application, which is a bit more complex. The model is just one part of this solution, and the last pattern is the self-service platform. We have some experience building that kind of platform, where the models are created by the users rather than the owner of the application. In this scenario, the problem becomes a little more complex because we need to consider the user experience—not only in terms of consuming the model but also in building it. Additionally, you need to think about how to provide good data segregation so that each tenant working on the model does not experience data leakage between tenants. Finally, when considering the long-term maintenance of this type of application, you have a team working with machine learning. You must ask whether this team will be available after the product launch. If the data scientist is not available, the model often degrades significantly.

In that case, you need to retrain the model and consider what is happening in that situation. If the resources and people are not available, you may end up with a potentially unusable solution in production. When thinking about scalability, what is your approach to scaling your application? How would you like to measure its performance?

That's common for the older software development lifecycle, software products, and, in the end, thinking about reuse. This means that if you're preparing a good solution or a good approach, you have the dataset and a labeled dataset. You need to consider how you can make this data available for others, for example, within the company, or if you want to make it open source, how you can expose this information to the world. Doing so helps people who are working on the same problem increase efficiency. Now, let's move on to the build phase, which will be more technical.

And Robert, please. Okay, can you hear me? Great. So, in this phase, we will focus a bit on the building part to cover all the things and processes you need to create a machine learning system. Since we don't have enough time here, we'll just cherry-pick some of the concepts, tips, and tricks that may not be that obvious but could make your life easier and save you time and effort.

So, at first, each machine learning journey starts with the data. You have to know that when you create a model, it's not your final model; you will have to retrain it either step by step or in an online manner because the data changes over time. It's important to understand how your data changes—how fast it changes and in what way—so you can address these changes quickly. In the phase of gathering the data, if you are doing classification, you must cover all the classes you have. However, there is a high chance that you will encounter an imbalanced dataset. In this case, sometimes you cannot do anything, but other times, you can work with your client and ask them to provide you with the data you need. This solution is often much easier than creating a more sophisticated model to handle such imbalanced datasets. When working on natural language processing (NLP) projects, it’s also crucial to cover all the templates. If you are dealing with documents, these documents can come in various templates and forms. They might contain slang or have different levels of language purity, and some of them could contain errors.

So, it's good to know what you're facing and to be prepared for that. Here, we can introduce the concept of data provenance and lineage. This involves understanding where your data comes from and monitoring all your data sources. For example, if someone changes the format of your data, it can degrade the performance of your machine learning model. Another important aspect is data lineage, which is the list of all transformations applied to the data before it reaches you. This is crucial, especially when you deploy a machine learning model, as it is not just a single service.

It's probably some kind of pipeline. If your model starts to underperform, the model itself might be fine; the problem could lie somewhere in the pipeline. If you don't monitor it and focus only on the model, it will be much harder to find and debug any issues.

It's also good to make some sanity checks and ask yourself what you might be missing, because you need to ensure your data is as representative as possible. If you're only working with a small subset of data, your job is to make it a good approximation of real-life data. So, look for discrepancies. Sometimes, your data can even be too perfect. For example, if you're training a computer vision model on photos taken by a professional photographer, but then in production it's used by people with their smartphones, the difference in quality could impact the performance of your model. Another important aspect is to maintain some kind of metadata for your data.

For example, using tags can make maintenance and debugging much easier. These tags could be anything.

It could be a date when you receive this data. It could be a batch of the data, perhaps a type of this document, its source, and many other factors. So let's consider a situation where you don't have any tags, and your model in production starts degrading. You notice that your metrics have dropped, let's say by 15% since the previous day, and you actually don't know where to start.

You have to debug it, but you don't know where to start. However, if you have maintained a proper system with metadata, you could see that your model works well for most of the data with most tags. For example, the PDF documents from Germany, which are mostly of low quality and use some kind of slang, are the ones that spoil your performance. You will be able to spot this issue easily if you maintain this kind of system. Data quality is also very important.

When we hear "machine learning," we usually think about big data. The first thing that comes to mind is probably large companies like Google and Microsoft, which train huge models with tremendous amounts of data and significant computational resources. However, the truth is that most machine learning engineers and data scientists do not work with these vast quantities of data. In fact, there is a high chance that your datasets will consist of hundreds or perhaps thousands of data points, not millions. This is why it is important to have good data rather than just big data.

If you have millions of data points, then they will have some noise, but there's a good chance that most of this noise will cancel out. However, if you don't have that quantity, it's really important to identify all the ambiguity, find all the mislabeled data, or data that doesn't accurately represent real-life data, and either fix it or eliminate it. Regarding labeling quality, most datasets are noisy, which means they can be mislabeled or contain incorrect or ambiguous labels. Here, we can see an example of a simple classification task, which is to detect if an image contains a bridge. We have a prompt stating, "Select all images with bridges."

So we have "bridge" here and here, but also this individual here is Jeff Bridges. If we label him, then it's probably an incorrect label, but this also depends on how you, as a developer, interpret this. However, the example in the picture on the right is a bit more ambiguous because we have two pictures of bicycles and one picture where there is no physical bicycle—there's just a drawing of a bicycle. This is a question you have to answer for yourself.

Is this a valid bicycle or not? You will encounter many situations like this, and to address them—helping to train your model instead of reducing your accuracy—it's beneficial to have a labeling strategy. Before we discuss labeling strategies, let's first talk about human-level performance. Humans excel at solving machine problems for unstructured data. For example, tasks such as translating text, summarizing text, engaging in conversation, or identifying objects in images.

These are problems that are really hard for machines, but for humans, they're natural. So, if we're performing a machine learning task on unstructured data, we're usually not aiming to achieve, let's say, 100% of some metric. We are using it to achieve human-level performance. It's really hard to surpass this human-level performance, or in some cases, it may even be impossible. There are two types of human-level performance (HLP).

One scenario is when we know the grand truth, which is how often humans discover the truth. The other scenario is when we don't know the grand truth; it reflects how often two humans agree. Let me give you an example. Let's say we are performing a medical task and we want to determine if someone has cancer. In this case, we can have the ground truth established by doing a biopsy, and the result of the biopsy is that, for example, this person does not have cancer.

Then we perform a medical scan, show this scan to a doctor, and the doctor attempts to predict the result of the biopsy. This demonstrates human-level performance when we have the ground truth. However, if we do not conduct a biopsy but only have the medical scan, we can present it to two different doctors. We can then evaluate how often they agree on the verdict. This represents human-level performance when no ground truth is known. This situation is significant because we usually cannot take much action regarding human-level performance when we do know the ground truth; the difference between 100% accuracy and human-level performance is often an irreducible error due to factors such as noisy or low-quality data. However, if the disagreement arises from two people having differing opinions, we can take steps to improve agreement. To enhance consistency in our labeling framework, we should first assess the current consistency of our labels within our dataset. To do this, we should analyze the dataset for patterns, edge cases, and ambiguities. For example, consider the ambiguity in the drawing of a bike.

And then we have to be clear on how we should behave and what the desired behavior of our laborers is in these kinds of situations. This way, we will increase the human-level performance. We will make our dataset more consistent, which will improve its quality. As a result, we will train a better model. Sometimes, instead of tweaking the model or trying to build a more sophisticated one, it's more effective to focus on these foundational aspects to ensure better outcomes.

Sometimes, instead of tweaking the model or trying to build a more sophisticated, more powerful model, it may be enough to fix the data or improve the quality of the data you have. This can yield results that are equal to, or even sometimes better than, those achieved by training a more sophisticated model. Another aspect to consider is using the right tools. While you might contemplate building an infrastructure for your in-house labelers, there are already ready-to-use tools available, especially for computer vision and natural language processing (NLP). If you don’t have any regional constraints, you could even consider outsourcing this task. For instance, you can use platforms like Amazon Mechanical Turk, where you describe the task, publish the data, and have it annotated or labeled by various individuals. Okay, so we've agreed that we need a labeling strategy, but how do we actually implement it? Let's say we have a situation like this: our task is to mark an area where an iguana is located, and we have three labeling strategies that seem to be quite effective. The first one involves choosing two non-overlapping rectangles where there is an iguana in each of them. The second strategy takes into account that the iguana has a tail that extends right up to a certain point, leading us to consider increasing the size of the blue rectangle.

And the third labeler could say, "But hey, they're close to each other. Let's just draw one rectangle to cover all of them." So the question is, which of these strategies is the best? The answer is that the best strategy is a consistent strategy. It doesn't actually matter there.

Your model—any of these three options makes sense—and it will work just fine if you choose any of them. However, the important thing is to select one and remain consistent with it. Prepare clear examples, as this will assist any new team members who need to label data. For instance, using these kinds of images could help them navigate your dataset and make informed decisions. Additionally, you should establish one single source of truth, which is this labeling strategy. Keep it in one place, making it easy to access and update.

Because, as we've said before, your data is going to evolve and change over time, and your labeling strategy should align with these changes. As more data arrives, you will encounter new edge cases and situations that you didn't initially think you would face. In these situations, it's also important to consider metadata and tagging. Many powerful machine learning models are available for easy use, but they tend not to make mistakes on straightforward examples. If your model doesn't perform well, it is likely struggling with more challenging examples—those are the instances where even a human might be uncertain about the correct label.

In this situation, you should have a list of tags discovered by, for example, browsing your data, and you could tag your data points during the labeling process. This way, if you encounter a problem, there's a high chance that when you look at the tags, you'll discover, for instance, a "red island" of data points that don't perform well. It might turn out that all of them are too bright or all of them use some kind of slang. If you tag these data points during the labeling phase, it will be much easier to debug and maintain the system. So, we have answered the question of which labeling strategy is best. Now, let's try to answer the question: Is 85% accuracy good? Well, it depends.

To answer this question, we need to establish some kind of baseline. This baseline could be a previous machine learning model that we want to exchange or upgrade. It could also be an existing implementation that may not use machine learning and works only on a subset of data using some kind of heuristic. Alternatively, it might just be a simple proof of concept that you complete in, let's say, two days or a week, using the simplest model possible to determine your starting point.

Because it helps you provide context and also allows you to assess your progress on your journey, starting from the baseline and moving towards achieving human-level performance. This is really helpful because if, after two months of using some heavy models, you arrive at 85%, then that's probably acceptable. However, if you're still stuck at 60%, it may indicate that you did something wrong. Or perhaps that is the best you can do, in which case it might be better to use a simpler, lighter version. Now, let's talk a bit about training.

We're not going to cover the entire training process, but we've picked up a few tips that worked well for us. The first tip is to use transfer learning whenever possible. There are companies that train large models and share them with the community. It's easy to leverage the knowledge gathered in these models and, with just a small amount of data, align them to meet our needs. Another important concept is active learning. Active learning helps reduce the time needed for labeling and training by decreasing the size of your data set.

And it's using your model to detect which data points are the most problematic before you even label them. Then you label only these most problematic data points because, as you may remember, I was saying that your model usually makes mistakes on hard examples. This strategy involves labeling only hard examples, as they carry more information and contribute more to your model's performance. Another thing that could speed up your labeling process is pseudo-labeling.

If you already have a model in production that works quite well—this is an important point, as it needs to perform adequately—you can take your new batch of data and pass it through this model. The model will predict labels for your new data, allowing your labelers to accept or correct the model's output instead of labeling all the data from scratch. This approach can save them a significant amount of time if your model is effective. However, if your model performs poorly, it will only create more work for correcting its mistakes. The final point to consider is determinism.

So your models will probably fail someday. The last thing to consider here is determinism. When your models do fail, you need to understand why. It is beneficial to have all the seeds fixed, because if your model fails during training and then does not fail later, it may be really hard to reproduce the results.

But if you have control over all the random things that happen, like weight initialization, by controlling the seed, you can actually reproduce these steps in your controlled environment, making it much easier to find errors. The last point to consider is to start from lightweight models. While transfer learning is often recommended, you don't necessarily have to start with models like BERT, which can have weights of more than a gigabyte. You can begin with something lighter and even use it as a baseline.

You can even use BERT-Tiny, which is 16 megabytes. It is about 100 times smaller than its larger counterpart, but if your problem is not that complex, it may already be sufficient. The last thing we will discuss today is maintenance. Your model doesn’t actually do anything until it is deployed. Deploying machine learning models and maintaining these types of applications is somewhat different from maintaining a standard backend service. It adds an additional layer of complexity, introducing more potential points of failure.

Let's say you have a service that serves photos. If it's just a backend service, you don't really care if a photo is too bright or too dim, or if it's of good quality. However, if you have some machine learning on top, your data could be valid, but you could still produce poor results. There are several steps we could take here. For example, there's A/B testing, where you deploy different versions of your model primarily to verify different hypotheses.

There is also a very useful shadow mode deployment. It's where your process behaves just like before. For example, a human is doing something, but in the background, in parallel, you run the machine learning model just to evaluate how it performs on real-life data. The data changes, so by the time you move from developing your model to when it's actually built and ready, the data can already be different. There could be data drift, and then your model could be rendered useless. There are also other options like canary deployment or blue-green deployments. In canary deployment, you first route just a few percent of your traffic to a new model and monitor what's going on.

So, it's like testing in production, but it's better to fail on two percent of the traffic rather than on 100%. Green deployment involves having a duplicate version of both the new and old models, allowing you to route traffic to the new one. If something goes wrong, it's easy to roll back to the previous version. The last thing for today is monitoring. Monitoring is somewhat different from monitoring a regular backend application. Here, you should consider some additional questions, such as: What are the potential issues with your data? What could go wrong with your model? How can you detect changes in the data or the concept that you are trying to capture?

So, of course, we also need some system metrics, but apart from them, it's also good to monitor the input of the model and its output. For the input, it depends on what you should monitor. Sometimes, for each use case, you should sit down and brainstorm on what could work well. For example, you could monitor the image brightness or the volume of speech for speech recognition applications, as well as any background noise.

You could monitor the average input length and the number of out-of-vocabulary words for NLP applications. For the output metrics, there are actually two types. You could either monitor the exact output of your model or observe what happens after a user receives their response. For example, if you are developing a search function, you could measure how many times a single person submits the same query.

Because you can assume that if users want to find something and they get it quickly, they will only need to type it once or twice. If suddenly a lot of people start typing the query ten times, then something is probably wrong with your model, and they are receiving incorrect results. Okay, let's make changes again.

Okay, the last slide from that presentation is about reusing everything. This is a solution that works well in our environment. It means that if you can reuse anything from your project, let's do it. This approach has two primary benefits: first, it reduces the cost of delivery, and second, it decreases the potential uncertainty associated with machine learning projects.

And the third point is that if you have, for example, a solution available, it may not be worth starting from scratch. This should, in my opinion, be the first and most important starting point of any machine learning project. First, we should look to see if we can reuse something from the past—be it experiences, models, data—and consider how we can make this new data or work reusable for others in the future. By adopting this approach, you are building capabilities within your company and your area, which will make the next project much easier. You will have a greater chance of success, and in many cases, you can achieve significant gains with minimal effort from the team.

And that is reducing the magic around machine learning, where people are expecting great results without significant effort. That is all for today; that is the summary. As for questions, I think we are out of time. However, we will be available at the UBS stand for the next hour.

If somebody would like to know more, we'll be available at the UBS stand for the next hour. If anyone has further questions or would like to know more, we would be happy to meet with you there. Thank you for today.