We’re Using AI Agents at Work (and it’s amazing)

Welcome back. About two years ago, we decided to release an AI assistant to all our colleagues across Prozus. It's an AI assistant-based program.

And what we want to share now is what we learned in the course of the last two years. Both in terms of how we use AI assistants and agent-based AI assistants to amplify our work and change the way we work, but also how we use the same technology to change the applications that we offer to our consumers. Many of you will know Prozuz. For those who don't, here are a few facts about this group: we are a global technology group.

Predominantly in e-commerce, and e-commerce for us encompasses a full spectrum of activities that you might engage in several times a week or even daily, such as ordering food, ordering groceries, or receiving parcels at home. On the other end of the spectrum, there are decisions you make occasionally, like buying a house or changing a car, along with everything in between, including electronics and fashion. We operate platforms in about 100 countries around the world, and this platform serves as an excellent connection point between buyers and sellers, students and teachers, as well as restaurants and diners. Here, you can see some of the brands in our group.

I want to congratulate Swiggy. Swiggy is based in India and had a successful IPO today. So, congratulations to Swiggy! Due to the nature of what we do, we had to start investing in artificial intelligence and machine learning early on.

We won't go through the process, but in any case, you can imagine that we handle billions of transactions every day. Therefore, machine learning is at the core of what we do. It has to be, because it fuels growth. As we start 2018, the ecosystem of AI experts and data scientists across the group consists of about 1,000 people, with several hundred models in production. We also measure hundreds of millions of dollars in impact, in practice, with all these models.

So AI is core to our growth. What I want to share, however, is what we learned in two parts. The first part is how we use AI agents to amplify our work. I will talk about Tocan, our own assistant that we developed in-house, and I'll explain in a second why. The second part is how the same technology, the same core artificial intelligence agents, also power OLX Magic, which is a platform offered to our consumers in e-commerce. Let's start with Tocan.

It's our personal and team assistant from Prozus. What does it do? Well, as you can imagine, I'm sorry for that. It is a very general assistant designed for everyone in the group to experiment with artificial intelligence, generative AI, and large language models (LLMs). It lives in Slack, as many of our engineering teams use Slack or access it through the web.

It's agent-based. It is agnostic compared to LLMs. It uses all the LLMs available. Of course, it has preferences. It connects to our databases.

It's designed for our own use cases, which is the reason why we decided to use it: to develop in-house. There is strong security and privacy. It is designed to produce analytics and insights while respecting privacy, of course, and it's composable in the sense that companies in the group can take pieces and integrate them into their own applications. At this moment, it is offered only to our employees.

Of course, we have some external users, but the focus here is on how to make our companies better. This includes not only the companies that we control but also those in which we have minority investments. The reason we built it is fairly straightforward. It relates to the fact that once we began conducting large-scale field tests of LLMs around 2020, we found that, although GPT-3 was not particularly effective at that time, it was promising. We had the impression that it could offer solutions for a vast number of use cases that we couldn't effectively address before.

People who were using that in our field tests were coming up with solutions and ideas that we didn't really expect. So we quickly concluded that we can't dictate how this will evolve. The best thing we can do is provide the best possible tools to everyone who can test them and figure out for themselves what these tools can do for them. We call this a collective discovery process. It's a large collective discovery exercise where everyone in the group has the opportunity to use this assistant to ensure that they can work better, faster, or deeper—whatever adds value for them. Additionally, they can test use cases that may be relevant to consumers.

So, when we started back in 2023, specifically in the summer of 2022, it was before ChatGPT. In 2023, when we began measuring adoption, we found that there were perhaps 1,000 to 2,000 users. About 50% of all the usage of this tool was engineering-based, with engineers being the first group convinced that this could make their lives easier. This included not only particular developers and software engineers but also product managers, among others. However, this tool had a lot of limitations, the main one being something we discussed in an earlier session.

Every time you interacted with a tool, you had four icons. The first one featured four emoticons: the first is a thumbs up, the second is a thumbs down, and the third is, of course, a heart.

And there's one specific one that we had to add, and that's Pinocchio. It's when you signal that the tool made things up. You're lying to me. You're really lying to me. I can't trust you.

On October 22, when we measured this, it was about 10%. This means one in ten. It's an answer that you cannot really trust, but you don't know which one. So, it was very hard to use the tool at that time. As time went by, you can see how the number of hallucinations decreased from 10% to about 1%, where it currently lingers.

We believe that the 1% will either remain stable, and it's probably something you cannot reduce. In several cases, we can establish sufficient guardrails around that, allowing it to be completely removed for specific use cases. Why did this happen? Because models improved, users became more adept at utilizing the tool and avoided areas where it doesn't function well, and also because we introduced agents. Agents are one of the keys to making these tools work better. Around the beginning of this year, specifically in January 2024, we introduced agents.

And when we talk about agents, this is the stack that we use. The main components are the framework, the orchestration, and the set of tools. You can access this stack through Slack, the web, or APIs. It connects to an agentic framework, which decides how to unpack what you want to do into pieces that you can execute in parallel. It then determines which tools you can use.

It goes to a router that picks the tools. Then, of course, there's a loop of reflection that occurs, but in fact, all these components here are what make this agent stack work. I'm sure many others at this conference today will have similar stacks, or perhaps something else that we can learn from. There is also an important element here, which is that of insights. There is another model that lives inside this stack, and it is designed to create statistics about usage.

It is not designed to focus solely on what occurs, but rather to create a group that the conversation indicates is centered on learning, a specific software language, and so on. This approach helps us build the next version of this tool within the framework of continuous learning. If you examine how the tool operates, you'll see that it serves as a simple data explorer. It's an agent used by people who want to access data in English. Since they may not know how to create queries, they turn to Tocan, which is connected to the data lakes. Users can find answers to their questions simply by entering them in plain English.

Of course, what it needs to do is find out which tables you have to use, which columns to include, create the query, execute the query, and determine if everything works without errors. If there are issues, it should go back and change the plans. It can utilize a variety of tools; in this case, it uses a query creator and executor, but eventually, it will also include the creation of a diagram. All these tools are part of what we can call a normal agent application, and we can envision this becoming the standard for our assistants in the future. Currently, after the introduction of agent-based assistants, we observe that engineering tasks and usage have decreased while non-engineering tasks have increased. This trend corresponds with the growing usability of these tools, leading to a broader range of engineers, including many more non-technical individuals, using them. The ratio of technical to non-technical composition in our teams roughly stands at 4159 to 460.

It is currently used by 24 companies, more than 15,000 mentioned before, and approximately 20,000 users continuously. It is getting very close to about one million requests per month. What have we learned from all this? The first thing I want to underline is that we made an early bet that the costs of running this tool would be acceptable. When we designed the first version, which was around March 23 and even earlier, the cost of tokens in GPT-4 at that time was extremely high.

So the bet was that it was going to go down. This is what happened: it went down 98%. That is fairly common with everything we see in all the other models. However, that's not the whole story.

If you look at the cost of the agent interactions, this is between May this year and August this year. It's a very short amount of time—just four months. You can see that the cost, represented by the yellow line, per token went down by about 50%. At the same time, the number of questions went up; I think it is the blue line. However, the tokens per question increased even further, shown by the red line.

So, it's the red line here. Because these tools require many more calls to the LLMs and many more interactions due to the complexity of the users' questions, the tool becomes more complicated. This means that the trend of costs going down and user engagement going up sort of compensates. Then we can see the green line, which represents the cost for each interaction. At this moment, it lingers around 25 cents each time you actually engage with the assistant and go through the agent process, and so on.

I think this is interesting because this green line represents the event when you actually touch the assistant; you go through an agent process, and so on. I believe this is significant because this green line will eventually drive adoption at the total cost of adoption. At this level, it is good; hopefully, it will decrease, but it could also increase. I am curious to see what you experience as well. The next thing I want to focus on is some of the things we learned about the impact.

Of course, we do this particularly for work because we want to ensure that everybody in the group works better, faster, and more independently, and so on. One thing we can always measure during this time is time saved. Here, you can see the model that analyzes the tasks, which are represented as rows. The bars indicate the amount of time someone spends on requests, reflecting how popular these requests are. The small numbers represent the time saved.

So, this model has been trained to recognize how much time you save every time you ask for something, in comparison to the best option that you have. It also takes into account the fact that sometimes we are wasting your time because you try to get something, but you don't get what you want. Overall, it amounts to about 50 minutes per day or 48 minutes per day, which is 10%. The question is, what do I do with that? That's the first question.

Second question: Is this the main and most important thing? If you ask users, the answer is no. Users indicate that, yes, they can work a little bit faster, but the main thing is that they can work with more independence. They don’t have to tap a colleague on the shoulder to get answers; they can find the answers themselves.

The third benefit is that individuals can work outside of their comfort zone. They may not be proficient in coding in a certain language, but they become skilled in that language because they have the right tools available. The fourth benefit is that it helps them get started. It can eliminate writer's block, allowing them to begin their work.

These factors together make it feel like the entire organization has become a bit more senior, which is probably far more important than the 50 minutes saved per day. Additionally, these 50 minutes are spread across a long array of micro-productivity bursts, making it non-trivial to automate them. However, another important lesson we've learned is that if you truly want to realize the value of all this, you need to change the way you work. It's not just about improving your current methods; you have to fundamentally transform your approach. This may seem obvious, but seeing it in practice with actual numbers is much more impactful.

Here, this iFood, which is the largest food delivery company in our group, is based in Brazil. We work with them to provide everybody with access to data in English. This means that anyone who has a question about data—whether in customer support, operations, marketing, and so on—can ask the data lake in English. Obviously, this has an impact on time savings; it automates about 21% of the requests, meaning that you can handle 21% of the inquiries through the agent or assistant without needing to access the data lake and create queries. This is the automation we refer to. Additionally, on the right, you can see that in some cases, while there is an automation effect, there is also an increase in work requirements.

In this case, the fact that you can solve these data questions with a tool makes it possible to address a vast amount of backlog questions that would otherwise take much longer to resolve. Initially, we thought the first users of this tool would be the data analysts, as they are the experts with the data and know how to manipulate it. While that was indeed the case, the real savings and transformation occur when everyone else is able to answer questions directly.

So we don't really want the data analysts to be the only users of this; in fact, we want the opposite. We want the entire organization to be capable of querying the data in such a way that they don't have to rely on the data analysts for assistance. This is exactly what has happened. Most of the questions are resolved at the source, which removes about 190 to 200 man-days every month from the data analysts' workload. Altogether, this leads to a 75% reduction in the time to insight. While time is certainly important, the key takeaway is that once we recognize this, we need to take action within the organization to ensure that it actually happens. This is what we learned by applying some of the insights we gained.

We published blogs about this. If you are interested, please reach out. We are happy to share. There are many other things that we learn, but they also relate to the way we develop applications. It is the same technology that powers tokens.

And I want to invite to the stage Paul, who's going to share some work on magic. Thank you, Juro. So indeed, you talked about two agents in production that we’re sharing a little bit more about today. The second one is one that we co-developed with one of the companies that's part of the group called OLX. OLX is a very large online marketplace for classifieds, where you can basically buy and sell second-hand goods as well as services. Think of electronics, fashion, real estate, cars, jobs, and so on. They are present in many countries all over the world, and we’ve been working with them to figure out what the next generation of the shopping experience powered by Gen.AI looks like.

So I want to share that with you. We call it OLX Magic. Before I show you that, I'm going to present a demo. This is powered by agents. Just like ToCan, it borrows and utilizes a lot of the technology that we've built into ToCan and incorporates it into this experience.

But I want you to have a look at some of the features that you'll see highlighted throughout the experience. Many of them will be powered by Gen.AI to better help buyers discover and find the products they are looking for. Currently, the app is live—Olex Magic is live, particularly in Poland—and you can access it. A lot of it looks and feels like a traditional e-commerce experience.

You go in, put in a query like an espresso machine, and immediately you'll get a couple of results. Those results, in particular, display features highlighted in blue and tiled colors, which can be refined. It suggests to you whether you want to look for semi-automatic espresso machines or apply other filters and criteria that help the user narrow down what they're looking for. Then, the results get updated according to the choice that the user made, as suggested by Olex Magic. You can also see custom carousels generated on demand for every search query, personalized for the user.

You can also go in and refine your search and say, "Well, actually, I'm looking for machines with a milk foamer." You can create a custom filter based on your description, which gets added to the search query. The results are updated live with that query, powered by generative AI and large language models in the background. One of the features I love most about this experience is the highlighted contextual filters per item that indicate whether it has a milk foamer or not, addressing and highlighting the aspects you care about as a buyer. Another generative AI-powered feature is Smart Compare.

What you can see here are the two products selected by the user, displayed side by side with the criteria identified for these particular items, along with a recommendation for the end user based on the query so far. As you can see, this experience of finding and interacting with items on the OLX website is now enhanced with various Genei features. We've observed from users that this significantly changes the way they interact and find what they're looking for—not just a simple query of results but a much more assistive and advisory experience powered by OLX Magic. Furthermore, in line with the theme of the conference, "Agents in Production," I wanted to highlight some of the tools we've built and integrated into OLX Magic to facilitate this experience. As you can see, it incorporates many elements that traditional agent frameworks and products possess, consistent with what we’ve seen others discuss. It has access to tools and is powered by a set of very powerful large language models.

It has the capability to plan and figure out the actual steps needed to reach the right answer. It has memory, which is particularly important not just for the context of the conversation, but also for the context related to the specific user trying to find something on the OLX website. Of course, the tools that OLX Magic has access to are very specific to its job. It has access to servers, the OLX catalog, both in a visual search way and a text search way. It also has access to the worldwide web for web searches, allowing it to pull in the latest information about what people are saying regarding different products. Together, these features create the ability for OLX Magic to answer and guide the user through their journey.

Now, one of the things that I also want to highlight is what we've learned. Again, as we put agents into production, these new products powered by AI offer new discoveries along the way. The first one's related to actual user experience. I think this was probably precipitated by the ChatGPT moment, where everyone was trying to build a ChatGPT for X—whether it be for OLX, iFood, or any other website or online experience you can imagine. This approach essentially starts with a conversation—a blank canvas where you can begin writing and asking questions about, for example, a marketplace. What I've just shown you includes a couple of product iterations down the road. We learned that this very blank canvas conversational experience is actually far from what users expect in a commercial setting. Therefore, we needed to return to the familiar user patterns when searching for items online. The second point pertains to the agents and the frameworks that we build.

It isn't just about agents; it's about e-commerce agents. What I mean by that is that the systems we develop—the agentic systems—need to cater specifically to the e-commerce journey. This means that the tools, prompts, evaluations, and other components, such as memory, must be tailored for personalization. The guardrails need to be built for an e-commerce journey, creating a unique type of agent and system. In many ways, this is built on the components we discussed in, for example, Token, which offers a more horizontal agent experience but is refined and constrained to the e-commerce role. The third point is that your Genie assistant is only as good as your search. This ties back to the second point about e-commerce agents: the search journey and the pipelines we've established historically did not work for the old magic experience. We needed to build them from the ground up. We should consider how we create embeddings for listings based on titles, images, and descriptions, and then build an entire pipeline capable of retrieving the most relevant items. We can use prompts and large language models (LLMs) to filter out what items are most relevant based on the user's conversation.

And so it required a lot of work to get us to this point. I think this is the moment where we take a couple of questions. I believe Euro and Dimitrios are ready to join us on stage. If you have any questions about this particular set of agents in production that we shared with you today, please feel free to ask. There are a lot of questions coming through.

And the key is that I think one of the most popular questions in the chat is: how long, and how many human hours did this take to put together? Which one? The assistant to Ken or Ox Magic? Well, Magic has been around for a couple of months now, but as I mentioned, it's gone through various iterations. So there are a lot of different things that we have discovered.

And the experience has absolutely changed a lot. The underlying tools, the LLMs, the prompts. So we're on a journey that started about, you know, like I said, months ago. For Token, it's more years, like you described, Euro. So that's something that we started with some of the earliest LLMs out there.

So we've been on this journey for a while now. Yeah. And what about Tokane? Yeah, Tokane has been the one that's been over two years, yes.

Do you have a number of people working on it, like actual human hours? Well, we don't know the human hours, but the team at Tokan consisted of around 15 people. Okay, cool. So, 15 people working for two years, powered by agents as well.

Well, that brings up a great question. Yes, regarding how you are saving time, what does that look like? Can you break that down a little more? Is it just subjective? No, no, no.

We train the model to understand the difference between working with the assistant and working without it. There is a model that resides within the token, and its sole purpose is to analyze the interactions it has been trained on. It's a large language model (LLM) that examines all interactions. For example, if this interaction is about debugging code, we have observed through the training set that using the assistant saves a certain amount of time.

So it actually estimates the time saved by looking at interactions and considering comparison activities. Here’s another great example: now the chat is coming through for real, and there are a lot of questions.

We've seen many discussions about costs, right? They touched on various aspects, but let's ask, why do costs go up while token costs go down? I think we haven't published anything specific about how many tokens we use in the end-to-end journey, from the moment you have a question to the moment you get the answer, including all the loops involved. We probably should address that. I don't think we have done so, right?

We haven't done it yet, but I can see that there are builders watching because this is probably the most exciting thing. You can see how the number of tokens per question changed once we introduced agents and the different types of costs per token as we introduced cheaper, smaller models for specific tasks. There are many interesting applied lessons that we learned, and we should consider publishing a bit more about that. Yes, we just published it because it's useful for everyone, and we should perhaps publish a bit more on that topic.

Yes, we just published it because it's for everybody. Someone mentioned that there might be ways to implement prompt catching to minimize costs. Are you doing any of that? Yes, we are. We've probably tried everything you can imagine in terms of optimizing costs, such as switching models, selecting the right model for the right question, caching, using reserved instances, and leveraging our own GPUs.

We've tried all the different combinations you can imagine. This is the best we can get so far. All right. Well, speaking of money, this is the last question, and there are a ton of inquiries in the chat, so I'll let you guys answer the rest there. How are you evaluating the ROI of OLX Magic? In terms of OLX Magic, it has just started, so we haven't published the metrics yet. However, what we measure are the metrics that make sense for e-commerce, right?

Which are transactions, which are engagement, and so on. We see the upside, but this is something that we have not shared yet because it's still a project in development. It's rapidly evolving, but I can tell it is positive. In general, we look at this as making it work, then making it fast, and then making it cheap.

Ah, making it cheap is what matters in the end. Well, awesome, guys. I really appreciate you doing this.