Emerging Architectures of LLM Applications 2025

We wanted to talk to you today about the emerging architectures of LLM applications in 2025. If this name is familiar to you, it might be because you have already attended one of our webinars about a year and a half ago, when we conducted the same session and knew significantly less about LLM architectures. So, it's time for a refresh. This topic is also very relevant to the a16z article that we'll mention shortly. But first, let me introduce myself. My name is Gad. We are a consulting company primarily focused on the implementation of AI applications in production. We work with some incredible companies like Notion and JFrog, as well as some other companies you may not have heard of yet. Additionally, we have a very special guest with us today.

His name is Jeff. Jeff would like to introduce himself. Hey, everyone. How are you doing? I'm super excited to be here today.

Gad and Gabriel, and all of TensorOps, thank you for having me. How are you doing? I'm super excited to be here today. I work at Google.

I'm a partner engineer, and I work with some partners like TensorOps, who, by the way, are super deep in this space, so I'm sure you'll love their presentation. My focus is data and AI strategy with my partners, so I'm going to talk a little bit later today about some cool things that Google is doing. Thanks a lot! I'm glad to have you here, and I'm really happy to join. I think people are really going to love your demo. We also have Gabriel here. Gabriel? Hey, I'm Gabriel. I'm an AI Solutions Architect at TensorOps, and it's great to be here. That was short, but I think people are going to learn to appreciate you over this presentation if they don't know you yet. So, what are we going to talk about today?

We are going to talk about agentic workflows, and we will essentially be overstriking the word "agents." We want you to release yourself from the concept of agents. You will hear a bit about agents, but we will focus predominantly on the workflows. We will also discuss the underlying architectures that need to support these smart workflows—these agentic workflows. Additionally, we will explore the future of LLM applications.

So, how do we think your applications are going to change in 2025, and what underlying architecture will you need to support these changes? There are some incredible insights that may have slipped through all the Reddit threads and Twitter feeds you’ve seen before. So, let’s just jump into it. Maybe the motivation for this discussion is that, as a consulting company or professional services firm, our goal is to build applications with customers. To be honest, innovation isn’t always easy; not everything succeeds. However, we have noticed that one reason for the success or failure of applications is choosing the wrong architecture. Many people working in this space are experiencing the same challenges. These days, there are a lot of buzzwords floating around.

You have, you know, chat, computer vision, guardrails, RAG, memory, and agents. All of these components are, like, mixing together. When you're trying to build an application, it's really about putting these in the right place. So, if you think about traditional code, you're trying to write an application that does something.

Maybe it's for ordering pizza. If you understand the business well, what you can do is wrap the architectural components into components of the software. You can have, as you see here, a controller layer, a service layer, and a data access layer. Your code and your architecture will be well organized, making the quality of the program or software you're creating much more maintainable and less prone to bugs. On the other hand, if you don't understand what you're building and what components are supposed to serve, you'll end up with spaghetti code.

And so what we did at TensorFlow is we looked at the architecture that a16z presented in 2023, which was really interesting, but we felt like there had been some advancements since then that needed to be refreshed. If you look at the architecture that we developed for ourselves, we tried to map out the components of modern NLM applications. Instead of going into each and every one of them—because we won't be able to cover everything today—what we will do is show you how certain design patterns or products that you might want to build can fit on top of this architecture. So, without further ado, I'm going to jump into the first design pattern and the first architecture that you can apply based on that diagram.

So if you think about what companies or individuals have built up until now, it was mostly chatbots or co-pilots. What makes them unique in the way that they operate is that the user interacts with the AI on a very short and iterative cycle. The user writes a prompt, like, "Today, I want to order a pizza," or "I need to rewrite my homework," and then it’s sent to an application like Gemini or ChatGPT. The AI system responds very quickly, perhaps within one or two seconds. Then the user goes back and edits the response.

And then it’s doing these loops back and forth with the AI. What’s interesting about this is that this system is fault-tolerant to AI errors. So how can I say that? Because obviously, everybody has seen that Gemini and OpenAI, all the AI systems, are making mistakes. But the only exit point out of these systems is the human.

So the only last gateway out of these systems is the human. This is probably one of the, I would say, nine out of ten architectures that I've seen are either copilots or chatbots. This is what people built in 2023 and 2024. Now, why did this kind of design pattern become very, very popular? It’s built around the limitations of LLMs. LLMs are incredibly powerful creations, I would say. So it's software, let's say 100 gigabytes, maybe 200 gigabytes, maybe half a terabyte.

But within these terabytes, you can create software that knows how to program better than most people in the world. And if it's not impressing you because you're already fairly established developers, not only can it program better than most people globally, it also understands accounting in Bangladesh better than many individuals. In fact, some accountants in Bangladesh may not perform as well as ChatGPT—no offense, this applies worldwide. While large language models (LLMs) are very knowledgeable in broad contexts, if you consider LLMs in the context of your specific application or what your boss told you, they can become essentially useless. So, you train them with huge amounts of data.

So, they have huge general knowledge. However, when it comes to what happened a minute ago or what my boss told me, or how my organization handles these kinds of cases, then LLMs are useless. In order to make use of them, you have to provide them with a lot of context. If you look at this kind of system, it allows the user, who has a lot of contextual awareness—such as an engineer working for a company who wants to improve their code—to provide context that the LLM doesn't have. Keeping that closed loop allows the user to continuously feed all of this context into the AI.

Now, the other part that large language models (LLMs) are not so good at is that once you give them very complex and long tasks, they get confused. We refer to these confusing errors as "hallucinations." When you give an LLM a really long document to edit, it tends to forget the original task and may omit some cases or parts of the text. Therefore, when you want to perform a really complicated task, rather than just asking it to proofread an email, keeping the user closely in the loop with the context is very effective.

But, you know, Gabriel, I think you have something to say about these design patterns. Yes, I think I need to preface this: most chatbots are lame, though not all of them. For example, ChatGPT is an amazing chatbot. However, in many cases, people don't go to websites looking for chatbots. When they encounter these chatbots, they often find that they don't work as intended, requiring users to do much of the work for the chatbot.

So, in the best case, they're getting an experience as good as talking to a human. In the worst case, they kind of feel alienated by this chatbot and then have to go to a human anyway. Now we see in the news that new polls are coming out, and people are sharing their thoughts about chatbots, with most consumers reporting that they don't really like them. I would also agree that the idea of chatbots is somewhat limiting. However, inspired by Langchain, it’s important to note that when people discuss AI, they should not view it as a system that needs to be tightly controlled to prevent mistakes.

One of the directions you see in system design is the development of systems that can operate autonomously for longer periods. If you think about large language model (LLM) applications, the best one might be able to function independently for, say, a year without any human intervention. You would simply send it a prompt for what it is supposed to do, and it would manage tasks by itself, such as cleaning your house or driving you to Alabama. One of the architectures that support this capability is known as cognitive architecture. There is a highly informative article on cognitive architecture for LLM agents, specifically for language agents, which I believe you should read. We have included a reference in the archive. Many of the slides you will see here are inspired by their work. The idea is that we aim to enhance the capabilities of LLMs, enabling them to work for longer durations, handle more complex tasks, and maintain awareness of past events and future expectations.

And you see, these cases actually make a huge impact on customer support already. You see this in coding and software maintenance applications. I don't know if you're using GitHub Copilot or Cursor, but you're going to see these kinds of architectures in the underlying structures of such products. And obviously, in what's called AIOps, which is not operations for AI, but AI for operations.

The main components of such an architecture are, obviously, a large language model (LLM), some kind of memory (both short-term and long-term), the ability to plan, tools for retrieval, and the capability to take actions in the environment. Additionally, there are various interfaces, many of which are designed to help these systems become aware of the overall context. If I really want to focus on how these systems should be implemented, I would look at the architecture diagram we created—or the several architecture diagrams we produced—where these systems primarily focus on orchestration. So, let's try to understand why orchestration is so important for this kind of system.

This is an example from a chatbot that we developed, which can be considered a co-pilot, in collaboration with one of our customers. What you can see here are the sessions that terminate with errors after a certain number of iterations that the user has with the system. Let me explain this to you. In this specific setting, represented by the blue line, approximately 40% of the sessions result in users expressing dissatisfaction with the results. However, if you allow users to iterate more with the co-pilot, they generally become increasingly satisfied with the outcomes.

And obviously, we just talked about why it happens and what happens, right? You let the user kind of tweak their knowledge and context into the LLM system. The more the user works with a co-pilot, the better the results become, to some extent. We have never managed to achieve 100% satisfaction. I think it's one of our most interesting insights as a consulting company: you don't plan AI systems to perfection.

You choose other metrics to measure the success of LLM applications. But when you're trying to talk about agentic systems or autonomous systems, what essentially is an LLM? It's a prediction model, and a prediction model has errors. So it might appear very, very accurate to you, but it still has some errors.

The idea of putting it in a workflow means that you're completing each step based on the predictions of the previous steps. These systems are called autoregressive. You have a model that predicts something, and then you have another model that makes its predictions based on the predictions of the previous steps. These systems are referred to as autoregressive.

So, you have some kind of model that predicts something, and then the model would take the prediction that it made and build on top of it another prediction, and then another prediction, and so on. It's not one-to-one like autoregressive models such as ARIMA, but the idea is that the more actions you take, even if your predictions on average seem reasonable, the confidence intervals are really growing. The margin for error is actually getting bigger and bigger because you're compounding these errors on top of each other. So, when you're trying to work on large language model (LLM) systems, the question is: how can you allow them to run for longer periods without compounding this error from one step to the next until it reaches a point where it just doesn’t work anymore? I’ll let Gabriel maybe explain a little bit about how you can actually implement that.

Sure! Here is the text reformatted with correct capitalization and spelling: "Yeah. So this kind of starts with us being able to make the models and the agents plan out their actions so that they can replicate this groundwork. Grounding that we showed in the RAG application to avoid this autoregression and this compound error. A good example of this, and this is a model-level grounding, is the new one architecture. So we're going to run through an example of this one architecture and present the concepts behind it. The concept that it introduces is called inference time scaling."

Let’s consider the example. We have a riddle: a man and a goat are in a boat, standing on the bank of the Hudson River. They want to cross to the other side. How should they do it? First, let's analyze what this means.

This is a riddle that is designed to closely resemble the one involving the goose, the beans, and the fox, but it is intended to be a bit more complex. The purpose of this complexity is to confuse the models and elicit incorrect answers. If we present this riddle to GPT-4o, the conversational model, it may be misled into thinking we are asking about the other scenario with the fox, leading it to provide the most common response. In this case, it would likely suggest that the man needs to take three trips when, in fact, after the first trip, the man takes the goat across the river and is already there.

So he could just stop at the first step. I love this example. I think it really shows how language, like conversational-level models, which is what GPT-4 is, is really about trying to emit an answer that sounds and looks, you know, textually correct. In these cases, you typically have a lot of steps, so it tries to create steps even though they aren't necessary. I love this one.

Yeah, it's a great one. So let's now run this same question through the new—well, not new anymore, as AI goes—but the reasoning model from OpenAI. You see here, we ask the same question. It thinks about it for a minute and 38 seconds, which, in terms of latency, is quite a lot, and then it's a perfect answer. It tells us that it's close to the other riddles, which is exactly our intention.

And then it tells us that in this case, there are no restrictions, so it can be done in a single trip. And that's the perfect answer that could only be achieved by stopping reasoning and then answering. This is the type of stuff that these models are going to be really, really good at. It's often used in math and programming because there are similar challenges. More broadly, this presents the concept of inference time scaling.

So, this is, like I said, a model-level planning algorithm. This technical paper for O1 introduced the concept of a thought period for these models, which is very powerful, as we saw. The idea is that these models will become smarter with training time. Yes, they still take ages to train and a lot of money. However, if we scale up the test time or the inference time, they will also become smarter with the increasing budget that we give them to keep trying to figure it out.

And so, in this technical paper opening, I introduced these two plots, which illustrate my points. On the left, we have the training time scaled with accuracy. As we increase the training time, the accuracy improves. Obviously, this requires more money and time, but we already knew this. On the right, we see that as we increase the testing time, the accuracy also increases significantly, as you can see.

This kind of shows the power of these types of models. It seems that, given enough time, they can answer any question. If we ask about the meaning of life, maybe it will take 7.5 million years to figure it out, but I guess the answer is 42. So, Gabriel, there isn't a lot of knowledge about what OpenAI is doing behind the scenes, right? When we're talking about making models smarter, we mean enhancing their ability to plan effectively.

So, we saw that if we want to come up with a better execution plan, using GPT-4.0 for that kind of task may present challenges. Since it is not a reasoning model and does not have reasoning built in, it might produce some hallucinations, resulting in a plan that is significantly less efficient. So, can you only rely on reasoning-level models to achieve this? Not really. It’s good that you pointed out that GPT-4.0 does not have reasoning built in because the chain of thoughts, which is what O1 is using internally, was already an orchestration algorithm we were using before with non-reasoning models. We still continue to use it with GPT-4.0, GPT-4.0 Mini, and LLaMA models, right? The idea behind inference time scaling is to start spending more tokens to obtain a better answer. Of course, more tokens lead to increased latency and cost, and the final answer will be based on many reasoning tokens that are generated. However, those often don't really matter; they are just a means to an end. This is what OpenAI's O1 accomplishes. These models, although I haven't confirmed it since they are not open source, are reported to have been trained with human-generated chains of thought.

So they train these models to explain everything they are doing and then provide a final answer, obfuscating this chain of thought. As a result, the users of these reasoning models will never see this final chain of thought. As we can see in this graph, the idea is that inference will now take a much larger portion of the total time that the model requires to operate. However, this is not the only approach, as we mentioned. There are ways we can achieve this without fine-tuning a model for reasoning, and we can explore many different methods. Google released a great paper on these various methods and the optimal way to implement them in August of last year — quite a while ago in AI terms. The paper presented three main approaches. The first of these approaches is Best Event. The idea is that instead of generating a single response to a problem, we will generate multiple responses, and then create critiques for each of these responses. We can then use these critiques, along with the responses, to produce a final, improved answer. This process serves two purposes.

It increases the accuracy quite a lot in more complex issues, and it also allows us to customize the number of responses to the budget that we want to allocate for this reasoning model. More responses will lead to better accuracy, but they will also introduce more latency and cost. Another approach, which is a bit more complex, is to implement a "best of" method using a tree search algorithm. The idea is to start with a problem, perform some reasoning, and then proceed through a step of that problem.

And then you use a model to evaluate how good this reasoning was. After that, you expand the better nodes of this tree. As you can see in the figure, it acts and receives a score of 0.6. There's only one.

And then there are two nodes: 1.8 and 1.4. Then, it expands the first one, and continues doing so intuitively. This process will effectively ensure that you are always expanding the best options—1.8 and 1.4—iteratively. As a result, you consistently expand the best thoughts, leading to a strong chain of thought that ultimately guides you to the correct answer.

So if I'm thinking about the first agent, for example, in the length chain, it was the react agent, which kind of introduced the concept of a loop, right? One step is a chain of thought, and then you go and perform actions, observe what you've got, and then reflect back to the agent. After that, you initiate another chain of thought, and you do this kind of loop. So essentially, what you're saying to me is that you can create this loop in a more sophisticated way. Yes, and basically have many thoughts, discard the bad ones, and expand the good ones. I wish I could do that myself. Why would you want to do that?

So, when you have a model like Gemini 2.0, which has reasoning built in, why would you need to do that outside of the model? Because it's another way to increase accuracy. If you have a very good model, it's already going to be, let's say, 80% accurate. By adding this research, you can eliminate the worst answers it provides. Within that 20%, you might be able to improve accuracy by an additional 5%, which could be extremely valuable for your use case.

And a good example is, let's say you're building an AI product. You're not supposed to get to 100%. But if your requirement is 80% and you're at 78, maybe this is the one that saves your profit. So it's always useful to keep that in mind. A simpler way to do this inference time scaling is to implement a method we discussed before.

It's called iterative refinement. Essentially, instead of generating just one answer, we keep refining it until the end, until we are satisfied with the result. The innovation that Google presented in this paper is that we can also perform this process in parallel. The more parallel threads of improvement we have, the more budget we can allocate, which leads to better accuracy. Additionally, there’s an interesting quote in the paper. Google defines what they call "compute-optimal scaling" for this type of inference. They found that easy and medium questions benefit more from this approach, while very hard questions, which the model would struggle to answer, do not gain advantages from it.

The idea is that this allows you to increase accuracy, but it won't make your Llama 7 billion model beat GPT-4.0. It's just another way to slightly improve accuracy or increase it significantly, but it has a plateau. So, when you're talking about orchestrating, if I go back a few slides, the concept of this architecture is that we have something you might call an agent, and you have that loop, which is the reasoning loop. Effectively, in order to keep the whole thing going, you need to come up with a plan for what your next action should be. We talked about having a large language model (LLM) in place to make decisions about the future, but there is the question of whether you still need the reasoning loop. Are you still going to need that external model? Will you still require reasoning, planning, and orchestration that is independent of the model?

Because what you've seen in GPT-4.0, for example, is that it comes up with a flawed plan. When you move to a smarter model, you don't need additional steps like thinking step by step, or perhaps you implemented some kind of reflection on top of GPT-4.0 in the past to ensure that the plan makes sense. Then, another model would inspect it. So, will all of this looping still be required? The answer is probably yes.

Because, as we said, some of the components will definitely remain at the model level. The models are extremely important and intelligent in generalizing various tasks, knowing what to do both inside and outside the context of their training. However, when you have a specific system that you want to purchase or build—perhaps involving a user interface and some Python logic or business logic that you want to implement—there needs to be a way to integrate all the components of the product you are trying to build. This could be anything from a chatbot for automating customer requests to a code assistant. There are two design patterns that are emerging and leading the industry now. I will say there may be a few provocative ideas that some people from LangChain may not be entirely comfortable with, but we'll try to address them in the best way possible. On the left, you’ll see the concept of graphs and LangGraph as one of the best and most emerging tools for agent architectures. It essentially allows you to build a graph, with orchestration achieved by traversing this graph and determining the steps to take. We're going to explore several design patterns that this technology enables. By the way, if you are not familiar with LangGraph, it was inspired by NetworkX. Additionally, there is another design pattern that is more message-based. If this sounds familiar to you, it’s because, when you think about traditional orchestration—setting large language models (LLMs) aside for a moment—and reflect on conventional software, there are two design patterns that have come to dominate: graph-based (like Airflow) and event-driven. For example, in an ETL (Extract, Transform, Load) process, Airflow is often used to orchestrate data processing.

So instead of explicitly defining the graph, you're going to create triggers for events, and effectively, these triggers will be consumed by some other components. If I'm thinking again in terms of traditional software, these represent two design patterns, and we should probably examine them to understand how they were built and what purpose they serve. This understanding will help us implement them now in the domain of AI. Let's start with graphs. As I mentioned, LangGraph is built on the concept of graphs, utilizing an event-driven architecture underneath. Essentially, you can do everything that is event-driven on top of LangGraph, and they have even added some advanced features recently, which Gavin is going to discuss. The idea is that if you know the workflow you're trying to implement, then graph-based orchestration is exactly the type of technology you would want to use. This discussion is inspired by a recent, highly informative blog post titled "Building Effective Agents."

The idea is to express business logic solutions using the concept of graphs, and you may want to introduce large language models (LLMs) as part of these components. In some cases, LLMs can act as processors. Imagine that an LLM is performing some kind of classification on the text, while the flow is predetermined on the graph. In other instances, more complex architectures can be built, such as the orchestrator synthesizer, where the LLM decides which other LLM to call. Then, there is another LLM that synthesizes the outputs from the previous steps.

You can build graphs in which large language models (LLMs) participate as part of the compute aspect of the graph, while also dynamically choosing where to route the graph. This introduces more dynamics into the way you are orchestrating the process. What's interesting about this approach is that graphs are really great because you can think of each graph as an instance of your orchestration. When you are orchestrating a graph, you are essentially orchestrating a process. For instance, when you look at the OpenAI Assistant's API, launching an assistant means you are initiating a job or what they refer to as a "run." Effectively, you are launching a job. You can use this concept of runs and jobs to monitor all of these diverse AI applications that operate as separate instances.

Graphs are a really, really nice way to represent information. I think that LangGraph chose an excellent approach to begin exposing the world to agentic workflows in terms of graphs. It's really easy for developers to think in terms of edges and nodes, and to understand how to connect them, as well as what the first step and the next step should be. However, to be honest, this can be quite limiting when it comes to expressing complex ideas. Let me provide a good example to clarify this, particularly when discussing event-driven architecture, which is another design pattern I'd like to mention. Imagine a system where an employee is being onboarded in an onboarding system. The idea is that this system would emit events and send them to Google Pub/Sub. On top of that, you would have subscribers, such as a system that allocates email accounts to new employees and another system that issues badges.

In this case, the system that emits the event indicating that an employee has been onboarded has no idea what the next steps are supposed to be. It is the responsibility of the consumer to know what their job is once an event reaches them. I believe we will see more of these architectures and designs as we aim to build more complex systems. For example, imagine that you are trying to build a code assistant. This concept is, by the way, taken from the Anthropic blog post.

I forgot to mention, but we'll include it later, perhaps in the slides. So, if you think about it, I want to write a code assistant. All right, my first action, as I'm trying to build a graph with all the components and nodes, is quite simple. I'm going to start writing the code.

Okay. When you're trying to decide what your next action will be, you might ask yourself: Should I run the code, test the code, or write a test for the code and then run the test to see the results? It can be quite difficult to determine the right order to execute these tasks in the context of the project's workflow. It would be beneficial to communicate a more dynamic way of processing this task.

So in these cases, what I believe would prevail is eventually a much more flexible architecture, which is purely event-driven. I will also say that when you're looking at what Lama Index is doing with their workflows, it's also event-driven, and there is a way to build a graph. So if you try to draw all the possible flows, eventually it will create a graph. And Gabriel, maybe LangChain is also doing this, right? Yes. So in its back-end, LangChain is essentially just a pub subsystem.

But they expose kind of two interfaces these days, and I wanted to use those two interfaces from the same back-end to show you the different ways to program agents. So on the left here, we have what's considered the normal LangGraph way to do things. This defines a typical React agent. You can see that we add nodes, which are functions. The nodes include the call to the model and the call to the tools, and then we add the normal edges.

So, from the tool, you go to the model, and then you start to add conditional edges. However, I want to add another agent. Let's say my agent needs to call another agent. The only way to do this elegantly is to add the agent as a tool of my first model or to add more conditional edges. Very quickly, this starts to break down. LangChain is addressing this issue with what they call "handoffs," and LlamaIndex has implemented similar functionality in their workflows from the beginning. Instead of defining the graph beforehand, you will have an agent function that runs your normal agent. At the end, you will return the next node where you're going to go. In this case, in the command being returned, you could specify the next agent and provide an update parameter, which will give it a custom state.

So, it's much easier to do a handoff. That's the pattern we are trying to implement here. Okay, I will say that Lengraph is built on top of an event-driven architecture, but effectively, once you actually need to define, many of these agent frameworks that you're seeing these days do the same. You are basically defining the node, and then at the end of the node, you are kind of forced to decide what your next step is going to be.

Even if this decision is LLM-based, it's still pretty much enforcing. If you think about event-driven architectures, there is one that is closer to graph orchestration, which is called point-to-point. This is a lot of what the architectures we are seeing are implementing. I think that what we will start to see in the upcoming year is a much more publish-subscribe kind of way of communicating. Publishers and subscribers are involved, where the node that releases the message doesn't know who is going to consume it. You would see an agent releasing a message to the message bus, and then other agents consume this message and decide what to do with it without you, as a developer, having to think about what needs to happen or how to code the logic. Obviously, it's a bit presumptuous to say this because the graph is simplifying a lot, and it helps you constrain and ground the system. I don't think that graphs are going to die.

So, if you look at traditional software architectures, graphs and event-driven architectures still survive. I do think both of them are going to endure. However, for more complex architectures, I believe PubSub will be much more adequate. With that, I would like to present a completely different approach. Okay?

We kind of discussed how agents are not really aware of what we think. What you should really consider in this context is smart workflow orchestration. This involves taking complex workflows that represent business problems and allowing AI to solve these issues. It's less about the agent and more about the orchestration. We presented one direction in which the industry is heading, which is toward longer and longer processes. However, perhaps we should return to the drawing we saw, where the user interacts with the AI in a very close manner.

So the user is sending requests to the AI, it’s waiting, and then it’s going back to the user. There may be a completely different direction in which the user is actually working even closer with the AI. Instead of releasing the AI into the world and simply saying, "Come back to me with results in five days," like with traditional methods, the AI is going to work much, much closer to the human. We will see a few cases that Jeff is going to share with us. So yes, agency is one direction.

I would let Jeff kind of build this story. I think he has an incredibly interesting demo to show. Jeff, are you ready with the screen sharing? I am. Okay, so I'm going to stop sharing, and I'm going to let you.

Great. I am okay. So I'm going to stop sharing, and I'm going to let you. Great.

Okay. Yeah, thanks so much. It's really interesting what you're discussing regarding the agent side of things. Here at Google, we're observing developments that are truly industry-wide. There is the concept of agents in the typical chat space, where they essentially augment what you're trying to do, whether in your job or personally. You might be chatting with this agent or sharing a video, and you're essentially asking it to enhance your knowledge about something, generate content, or help you solve a problem. Then there are the workflow agents you mentioned. For example, let's consider a sophisticated chatbot or agent designed for retail or an insurance website. In these cases, a specific workflow must occur for the agent to succeed in its activities. But yes, definitely!

I want to talk a little bit about multimodal and what we're doing at Google right now. I’m sure that a lot of people have already heard about Gemini 2.0. It's been released experimentally, and the great thing is that everyone can try it for free. There are three ways you can do that.

AI Studio, which I'm about to show you, allows you to use our end-to-end LLM Ops platform, Vertex AI. You can also access it directly using the SDK of your choice. I would really urge everyone to try it out; the full multimodal capabilities are available. So, what does that actually mean? Basically, many of us have been interacting with AI over the last few years. Even as recently as last year, you would send text to an LLM and receive generated text back, or you would send text and request audio back. Similarly, in the inputs, you might be uploading something like audio or video and waiting for some processing to occur, either in batch mode or synchronously. The nice thing about Gemini is that it's multimodal by design.

So, what we've done is pulled in all the different modalities you see on the left for the inputs, and Gemini doesn't really care too much about what you're sending it. It's all API-driven and capable. As of now, with the experimental version, you're getting text and audio as the output. When we start discussing what you can actually do with that, when you add some features of Gemini, it also allows for natural interaction. Previously, this used to be much more of a synchronous flow, where you ask a question and get a response.

If you're unhappy with a response, you can ask another question, and that creates a natural flow. The truly great advancement happening with Gemini 2.0, which is enhancing the agentic AI era, is the introduction of an open socket. You'll see this shortly in my demos, but you can now use that open socket, either API-driven or through the interface. Essentially, you can interrupt Gemini at any point to have it update something or express your differing opinion, and it will respond accordingly. Additionally, some of the other fantastic features combine Gemini 2.0 with the Vertex AI platform, which supports Gemini's advancements in responsibility.

This is something that is extremely important. The patterns that we're seeing a lot with many of our customers are, you know, let's say the last two years have been the prototyping phase of using LLMs for companies of any size, really. Let's get something up and running. Let's get an MVP. Great, it's working.

Let's now get that into production, and we'll fix the problems later. As you can imagine, a lot of issues can arise, and there have been many famous examples of this. You might be receiving hallucinations that can be hurtful, or you might encounter hate speech. Even worse, there could be a bad actor trying to generate hate speech or instructions on how to do something that could cause harm. The great thing about Gemini 2.0 on the Vertex platform is that we're taking a lot of that into account for you, so you don't have to build that capability yourself. Additionally, I was just mentioning that you now have live streaming, which I'll show you in a few minutes. To build on Gad and Gabrielle's points from the agent's perspective, Gemini 2.0 is enabling functionality for reasoning, tools, and grounding. Finally, you'll also see a focus on governance.

And we're going to go into that in a little bit as we talk about Vertex AI. So Gemini, through some of these demos that you'll see, has features like built-in code execution, where it can execute code inside of a virtual environment and literally show you what it's doing. It's not just generating textual outputs; it's actually executing something. In addition, you can utilize function calling.

So, this takes things beyond generating responses. Through common API sets, such as OpenAPI, you can execute activities. This allows Gemini 2.0 to interact and perform actions rather than just providing information. Finally, what's really important—and pretty amazing—is the grounding functionality that we recently announced, where you can ground in Google Search. This feature will help you when your agent or LLM needs to access real-time information or information beyond its training date; it can do that through Google Search. There will also be footnotes to show you the source of the information. As expected, we have different RAG (Retrieval-Augmented Generation) options available, and Vertex AI Search will be generally available shortly.

That just takes care of the vector embeddings for you, so you don’t have to build it out yourself. With that being said, let's take this for a spin. I have a couple of demonstrations that I’ve prepared here, so let's go ahead and start. Okay.

Great. So I'm sharing now. Welcome to Google AI Studio. Think of this as your rapid prototyping environment where you can execute a lot of the functionality that I spoke about. You can really start enjoying many of the features of Gemini 2.0 with very lightweight capabilities.

So, one of the... I'm going to jump right into the multimodal streaming. If you haven't seen this before, it's really cool. I hope you enjoy it. Let's start. Hey, Gemini, how are you doing today?

I'm doing well, thanks for asking. How can I help you today? Great. What am I wearing? You're wearing a black-coated sweatshirt with a Google Cloud logo on the right side.

It also has a circular patch on the left side. Yeah, what do you see on the patch? The patch on your sweatshirt has a Google Cloud logo and says "Certified Professional" underneath it. Nice. Based on the swag that I'm wearing, where do you think I work? Based on the Google Cloud logo on your sweatshirt and the "Certified Professional" patch, it's likely you work at Google Cloud.

Okay, good guess. Let's jump to another topic. I've had a really busy day. Unfortunately, I wasn't paying attention when I was driving home. And if you look at my phone, this happened. You can see that it looks like two cars have collided—there's a light blue Tesla and an orange car, both with damage to the front bumpers. Is there a lot of damage to that Tesla? Assume that everything you see is the extent of the damage. Yes, there does appear to be a significant amount of damage to the front of the Tesla. It's pretty bad; I'm probably going to need to get this towed.

Can you recommend a tow truck company in San Francisco? Sure. Here are some towing companies in San Francisco: B&A Towing Service. Awesome. That's great.

I'll call them in a minute. What about repair shops in San Francisco again? Okay, here are a few options for repair shops in San Francisco.

The Garage SF offers complete auto repair and body shop services. Excellent. Lastly, they provide car rental services in San Francisco.

The garage offers complete auto repair and body shop services. Excellent. Last thing, car rental in San Francisco. Is there anything that you can recommend? Sure, here are some car rental companies in San Francisco.

Citi Rent a Car. Awesome. Let's switch gears real quick. I also happen to be a developer, and I was looking at this code today. What are your thoughts on it?

What does this do? That code appears to be using Python to access a Google Cloud Storage image and then send it as a prompt to a multimodal model to generate a blog post. Very cool. Let's change gears one more time. I love science fiction shows, whether they're movies or series, and I also enjoy watching dystopian shows. Based on this list, what do you think I should watch tonight?

Based on your love of dystopian and science fiction shows, I would recommend you watch "Silo." It's a science fiction show that is number one on the top chart of TV shows. Great! Were there any others? Yes, "Shrinking" is also listed on the top chart of TV shows, as well as "Severance." Ooh, "Severance" sounds cool!

The only problem is I don't have a subscription to Apple TV+. Can you recommend any other providers to watch this? "Severance" is exclusively available on Apple TV+, but it's also available to rent or purchase on Fandango at Home, Amazon Video, and FlixFling. Additionally, it can be streamed for free on Hoopla, Plex, and the Plex channel. Sounds great! To purchase, okay. So, um, that was a lot of fun! We explored a lot of different modalities and reasoning. Think about modalities within modalities. I was showing a picture of crashed cars during my video session, and Gemini was really able to understand what a picture of crashed cars looks like through my video session.

And Gemini is able to really understand what it's looking at and to reason, make assessments, and call external tools. I'm not sure if you saw the background while it was doing this, but you could actually see the debug of the external function calls and the code execution it was performing. There is a tremendous amount of capability with Gemini 2 out of the box. The API is available, so I would really urge you to check out the documentation.

We also have a Google-provided example set on GitHub where you can see them all. Go ahead. Sorry, you're still showing the agenda. If you want to go back to the deck... Yeah, I'm about to switch back.

I'm just kind of filling in some details here. Thanks. Yeah, so there are a lot of other options, but I would definitely urge you to check out the GitHub. We can send that out to you as well. So, a couple more slides here, and then I'll turn it back over.

So, I wanted to talk about the end-to-end development with Vertex AI. As Gad showed you before, there's a lot of complexity—rightfully so—for managing LLM operations. This isn't just a pattern for LLM operations; you also have it for MLOps, DevOps, and for working with infrastructure. Whatever it is, there's always going to be a process that you need to follow to ensure consistency, governance, controls, and repeatability. While all of this is definitely possible with common open-source tools, many patterns that we see come from the fact that a lot of companies have been experimenting in the past few years. Many of them have MVPs, and this is the year when the majority of organizations will start having full production systems, whether they're externally facing to customers or internally facing to internal users. There will be full production systems.

It is now more important than ever to make sure you're properly taking care of these systems. As we discuss various topics, let me point out that this is the Vertex AI logo. We will be talking about things like the Agent Runtime, as well as Gemini 2.0, which has been integrated into Vertex AI. Additionally, we'll discuss different reasoning toolsets, including Allini 2.0, which is also part of Vertex AI. All of the toolsets that I showed you on the previous slide are enabled within Vertex AI. When we talk about data, whether it’s a vector or a custom data source, it all routes through Vertex AI with its LM operations, which are crucial. This includes all prompt management, versioning, and logging, which are also handled through Vertex AI. We also discussed the importance of safety in the context of responsible AI, and you'll be able to access that as well.

LLM, so it's the same. Not just Gemini 2.0; we also have a full model garden with well over 100 various sourced models that you can choose from. And of course, fine-tuning, dataset management, and experiment tracking are all aspects of your operations cycle that you can manage inside of Vertex AI. I've been talking to a lot of our customers recently about how they can make the jump to an organized and governed way to run their LLM operations. The good news is that your notebooks, if you've been doing a lot of custom coding, can be brought right into Vertex AI, allowing you to really hit the ground running. From there, you can expand and put together the proper guardrails and processes.

Here are the key takeaways. I hope you've been inspired by what you've seen in the demos and, of course, by what has been presented by GAD and TensorOps. We are definitely well into the multimodal era right now. The announcement of Gemini 2 makes that very clear and delivers the message that Google is your AI platform. Agentic AI is a significant growth driver.

So, don't think of agentic AI as just an internal efficiency driver. Consider various workload processes internally to save time or improve SLAs. This is also something that is externally facing. For example, think about the situation I mentioned with the damaged vehicle; imagine car insurance companies using similar workflows, etc. Finally, Vertex AI can handle the end-to-end platform development.

So, if you're feeling overwhelmed and open-source LLM operations is not your core business objective, and your developers and data analysts are starting to get bogged down, this is something to consider. With that, I'm going to conclude, but thank you all so much. I'll stop sharing now. Alright, I'm going to take over and, in the meantime, I received a question privately. If you have questions, you are more than welcome to ask them. I received a question about what "moremodel" is and why it is different. Jeff, would you like to take that? Yeah, absolutely. What are modes?

Modes are the various channels of interaction. Typically, you would think about the very beginning of some of the models that were out there. You would send a text message to the model, and it would generate a response and return it to you. Alternatively, you could send a text and ask it to create a picture, and you would receive that picture in return. Now, with the same model and at the same time in the same stream, you can inject text, video, audio, and, of course, pictures, as you saw.

And the model in the same session is able to interpret all of that. So what's happening with Gemini 2.0 is that it has brought in the capabilities that would previously have taken multiple models. Of course, on the back end, it's doing that in an agentic fashion, if you noticed some of the coding going across the screen during the demo. Multimodal refers to multiple ways to interact with the input and then multiple ways to receive your response in the output. I love this response.

I think, in terms of architecture, what it means to most of the people who are going to build AI applications themselves is that a lot of the tasks that you had to do before have become simplified. For example, if you designed a co-pilot, you would upload an image, and then that image would go through processing in one model that would turn it into text. The text would then be transformed in a large language model (LLM) to generate an answer. Now, we can send everything to one model that has a unique representation of all the data together, which greatly simplifies the architecture. One of the major takeaways I have is that everybody is saying 2025 is going to be the year of agents. A lot of what you're going to see are agents that you send a message to, and they operate over a long period before returning an answer. We see OpenAI stating that they have a model that can think for about three and a half days before providing an answer. However, I think the more interesting product angle is not just AI that goes off and does tasks—everyone understands that there will be a significant implementation of autonomous AI systems. What might be overlooked is the collaborative, interactive way of working with computers. I saw someone mention that Entropic has an operator that does just that. It’s a nice demo, and I think you just saw Google showcase an excellent demonstration of Gemini 2.0 coding—essentially co-programming with a developer. We'll send that to you afterward; you can also see a similar example in LangChain. This is a great illustration of an interactive way of communicating; it's called Canvas. The idea is that, instead of going back and forth to the model and creating a lot of grounding, planning, and orchestration, you effectively shorten the interaction time with the user to make it really smooth. Okay, so we are just about one hour into this webinar.

I have to be honest with you; we have quite a few more slides to go. This one is going to be very, very, very deep, so brace yourselves. If some people need to leave because it's already late in their time zone, we really appreciate your attendance, but we are going to continue with some more nuances, especially relating to the initial deck we presented, which covered the long-running agents or the long-running agentic workflows we described. Again, thanks a lot, Jeff, for the demo. I believe we're receiving incredible messages here in both the public and private chat, so thank you very much.

So, you know, we've shown you two technologies, and very often when you're presenting technologies, it's quite easy to imagine what they could turn into, right? A few years ago, we discovered that AI or neural networks could press the pedal, and perhaps they could also steer the wheel. So, cars are going to drive themselves automatically in two years. Then, a few years ago, we found that neural networks can identify tumors in radiological imaging, leading some to claim that radiologists will be obsolete in five years.

2024 came, and still, radiologists are driving their cars to work. I don't say that this technology is not going to evolve, but sometimes technology requires a little more effort than it appears. We wanted to show you the impact of the emerging cognitive architecture we've discussed before. This amazing architecture can become more complicated, and we will explore what you can do about it, including some valuable tricks and errors we've seen companies implement and ways to work around them. So, Gabriel, I'll hand it over to you. Yeah, thanks! I wanted to delve again into the Koala Agent framework. We briefly discussed it in the initial slides and used a lot of their diagrams. This paper is an excellent source.

I wanted to dive deep into some of the components that are listed, as I believe it's important to discuss both the pitfalls and the technical challenges associated with each of these components. This paper establishes a great framework for what an agent is and what an agent should be. If you know this framework by heart, you'll likely build some very effective agents in the future. So, I really wanted to focus on this concept for us. The three main components for an agent to function well are memory, action space, and decision-making.

For memory, we're going to focus on two types: long-term and short-term. We will dive deeper into them shortly. For the action space, we have four main categories: reasoning, retrieval, learning, and grounding. The decision-making component will contain most of your language graph code and orchestration code. Let's start with memory and take a closer look at it. For short-term memory, we want it to keep track of the current conversation. Think of it as a list of messages that your agent will monitor during human-AI interactions. It should also allow these conversations to be resumed if they are interrupted, similar to how ChatGPT handles it.

You can just press one of your previous threads. So, this is like, you know, LangChain implemented memory for the chat. Yeah, it's the most basic list of messages you can have. Of course, there will be some complexity down the line. And then you have long-term memory, yeah. That is because LLMs are effectively stateless, right?

So you're inputting some text into the LLM, it makes a prediction, and then it doesn't remember what it did before. Therefore, you have to implement it externally as some kind of database array in the memory. Yes, otherwise the LLM has no idea what's going on in your conversation.

So you always need this short term. So far, you sound to me like 2023. I'm guessing the next one is going to be more innovative. Yeah, that's where we get it. It's interesting.

So, in long-term memory, this paper divides it into three types of long-term memory, inspired by how our brains work. We have episodic memory, which stores events that happen—like today, when I went out and it was cold. Then, there is semantic memory, which involves remembering facts and definitions in general. Lastly, we have procedural memory.

In humans, this usually relates to muscle memory—how you move around and how you perform the most basic tasks that you don't even think about. It's the kind of stuff that becomes automatic. Now, let's discuss short-term memory. We want to keep track of the conversation, as we mentioned, but there are some complexities that arise when we start having very large message threads with language models. For instance, let's say we're talking to ChatGPT for a year, right?

The thread is going to be incredibly long, so we need to do something about it. I mean, I don’t want to be rude to OpenAI, but ChatGPT would probably get filled up much more easily with its context window. Gemini 2.0, with a 2 million token context size, is going to get filled in a year. Just saying.

Let's say you speak to Gemini for 10 years; eventually, it will be filled. It will run out of memory. I've tried to speak to it for 10 years. I'm getting there. Yes, in 10 years, I think it will be a problem.

We'll have a billion-token context window. So, let's see; we have three main options in this case. The first is that we start dropping the first few conversations like a sliding window. The old conversations begin to be completely deleted because they no longer fit. However, this can cause the model to appear a bit unintelligent, as it forgets things from the beginning of our conversation.

So the user might be shocked by the fact that it doesn't know this. This is the naive approach, obviously. There are two other options that require post-processing of your context window or short-term memory. One of them is to use a large language model (LLM) to filter out the relevant messages. Realistically, most of your conversations with ChatGPT are not that interesting.

I know I can probably delete them and keep most of the context of the conversation by just keeping 50 of the messages, let's say. Another approach, of course, is to combine it with filtering to summarize the previous messages and then just keep that summary in your short-term memory instead of all the conversations. I think the more interesting part is that much of the implementation we've seen uses memory. They keep it as some kind of object in the computer's memory, where when the session ends, the memories are released. However, if you implement persistence, you get a much better experience because the user can go back to the conversation and continue from where they left off, which is really powerful. An interesting aspect is that even though your short-term memory persists, it's still considered short-term. It remains in memory, even if it's stored on disk, making it long-term in a sense. However, it is still classified as short-term memory. Now, moving on to long-term memory: this type of memory, in the context of large language models, can be relevant across different conversations for the same user, covering aspects like user information. It can also contain memory relevant across different users. For example, consider retrieval-augmented generation (RAG). Let's look into these types of memory in depth, starting with semantic memory.

Like I said, this is going to be where we state facts. For example, in an agent, it could be a retrieval-augmented generation (RAG) system. This represents semantic memory. Then we will have episodic memory, which will consist of things that I did for a human, but for an agent, it could be different.

Past mistakes it made—that's episodic memory. And I think that, sorry to interrupt again, because I just love interrupting you since I'm from the Middle East. I believe that one of the most frustrating experiences that companies have with these LLM applications, even if it's just a chatbot, is that it keeps making the same mistakes over and over again.

And I think most of the errors are because they're not really implementing a write mechanism. So the LLM is only reading from the long-term memory. It's only reading from what it knows. It only reads from the retrieval system. I'm not going to say retrieval augmentative.

I'm not going to say that word. They are just reading, but they never write. I can understand why companies might be a little concerned about implementing writing mechanisms in LLM applications. However, I believe that many of the reasons why LLM applications are failing to provide users with the experience they desire is due to the lack of implementation of long-term memory writes.

I feel like the writing part is very complex. If only we were going to dive deep into it soon. I guess we can get a lot more context around that. The last type of memory that I wanted to mention is procedural memory. For humans, as I said, this relates to motor skills and muscle memory.

For agents, it will involve elements such as your system prompts. For example, the instructions on how you perform a task to an agent constitute procedural memory. It's included in the prompt, but it technically resides in this architecture's memory. While it is considered long-term memory, it is often injected into your short-term memory. This is a very important distinction.

As I mentioned, this act of remembering requires that at some point we decide to save information. For example, ChatGPT remembers certain details about us, but we are not always aware of what it chooses to remember. It doesn't save every conversation in one place for continuous use. There must be some representation of us as users when we interact with ChatGPT, and you can actually find some of this information. You might be able to locate what it remembers in the configuration settings, and you should probably have the option to download it—though that raises legal questions. In short, we can't save everything. For instance, some users may interact with ChatGPT frequently, perhaps five times an hour, because their jobs require it, such as in programming. It is impractical to save every message and expect it to be accessible every time we send a message.

We need to decide what to save and when to save it. This is a really complex task. There are two main memory creation methods. One is to write during runtime, where you're chatting with ChatGPT, and it decides this is valuable and saves it. The other method allows for saving to occur in the background.

So you're chatting with a chatbot. It doesn't save anything, and then your conversation is sent to a separate chain—a separate language model—that decides what is relevant to save. Each of these approaches has its own challenges. So, let's dive deep again.

So, when updating during runtime, the usual method involves providing a tool to the model, which then decides when it wants to share information about you. For instance, if you add a prompt stating that whenever the user mentions personal information, the model should call this tool and save it to memory. This is exactly how ChatGPT operates. As you chat with it, it decides whether to save information about you for future queries, which is why it gradually becomes better at interacting with you. When you ask it questions about yourself, it can retrieve a lot of personal details—it can feel overwhelming when it seems to know how old you are and where you live. The advantages of this approach are that the memories are immediately accessible, and you can observe how these memories are created. If you examine the traces in the case of ChatGPT, you'll likely find that they are not visible. However, in an agent with a graph structure, you can see when it invokes the "store memory" function.

It's very clear what it's doing. The downside is that, of course, if the model is accessing memory, it's not responding to you. So obviously, this will affect latency and distract the agent from doing its job, right? If it's preoccupied with the question, "Oh, should I save this to memory?"

Should I not save it to memory? Is it relevant? It's not doing its other job. Maybe if its job is solving complex mathematics, it's being distracted, basically. So then we consider the other approach: updating the memory in the background.

So, if we want to update in the background, there is going to be no latency because it's done after the conversation is finished. This approach is beneficial in terms of modularizing the memory logic and the application logic of the agent. However, it introduces a whole new set of challenges. We need to consider when to write updates—should we implement a cron job that checks the user's conversations every hour? If we check too frequently, we might save too many irrelevant items to memory, so it's essential to define a schedule and determine what constitutes relevant memory. We may need to create an entire memory architecture that periodically removes irrelevant information. As you can see, this architecture becomes very complex, and it takes a lot of infrastructure to support simple tasks like remembering who you are. This brings me to the next point. Yes, sorry, I think there was a question about what grounding is.

So we're now going to talk about grounding. We mentioned it in a rather naive way, and perhaps we should have paid more attention to it. Gabriel, could you focus a bit on what grounding is?

Sure! Here’s the text with correct capitalization and spelling: "So, yeah. The action space focuses a bit on what grounding is. The action space is going to be the toolbox of your model or your agent."

So, it has four types of actions, three of which are internal. The first is reasoning, and this is very simple. Reasoning is the act of examining short-term memory and generating information. Essentially, this involves calling the LLM (Large Language Model).

Retrieval involves reading information from long-term memory. For example, retrieval is going to be used frequently, right? So that's all right, I guess. Learning is the act of writing information to long-term memory, and we just saw that this can occur in real-time when you're running the agent, or it can happen in the background with a very complex architecture—it's still learning. Then you have external actions, and this is where grounding comes into play. For the purposes of this paper and this framework, grounding is the act of interacting with the surrounding environment of the agent. So if you're writing code, grounding is the act of running that code.

If you are a robot, grounding is the act of interacting with the physical world. If you are a chatbot, grounding is the act of speaking to your user, assuming that you are trying to communicate with that user, and that user is what grounds you, even though the user might be prompting you with any task. To emphasize a little bit about grounding, when we talked about long-running operations, you would create an LLM (Large Language Model) system that generates a plan and then operates on top of that plan. The operation itself could be an LLM that responds to the plan. In this context, you're operating within a world of predictions—predictions and more predictions. For me, the idea of grounding has several definitions. However, I believe grounding essentially means connecting the AI system with the real world. This involves introducing data points from the actual environment. As Gabriel mentioned, this could involve conducting a search on Google.

So, for example, Gemini can search on Google and then check if the results it provides have references to real websites. If it's reliable, like, you know, I don't know if it's CNN, but at least it has some kind of reference. So, it gives you some sort of connection to the real world. And that's, for me, the idea of grounding.

So, getting on to decision-making, the last part of this framework, the decision-making is where 90% of your agent code is going to be. This is what LangGraph usually handles; this is what LAMAI is going to handle as well, particularly the retrieval aspect, but mostly this. This process is a cyclical process between planning and execution steps. Planning will involve either a reasoning step, which generates more tokens, or retrieval. In the case of RAG, this will involve making queries to vector databases and all that.

And execution is going to be either performing grounding—interacting with your environment—or learning, which involves saving information to your long-term memory. This decision-making process, as I mentioned, is the core of all these toolboxes. I want to share a final thought about this framework that I find personally important. How did you manage to go through this entire webinar about architectures without mentioning Retrieval-Augmented Generation (RAG)? Didn’t you? Oh, you did mention it, but I shouldn’t have assumed. RAG is fine, but I personally don’t really like the definition of "agentic RAG" as it seems to be more of a buzzword than anything else. There is no true "agentic RAG"; it’s simply agents that perform retrieval—nothing more. The retrieval just happens to be a bit more complex in these agents. So, when we refer to agentic RAG, we’re describing an agent that decides what to retrieve, when to retrieve it, and how to use it. I recall that more than a year and a half ago, we had another webinar with the same group discussing emerging architectures in the middle of 2023. During that session, we concluded that RAG isn't quite what it seems; it operates more like a search and retrieval system. While we have seen implementations of RAG in certain contexts, we believe that when discussing true architectures that are viable in production, we should think of it as a retrieval system supporting an energetic workflow system. I also want to emphasize that you don't build your applications around your retrieval system; you build your retrieval system around your agent application. Now, let’s delve into a bit more complex material that still fits within this architecture. We’re going to go into multi-agent systems over the next four or five slides. Before we continue, I know we're slightly off schedule, but hopefully that’s okay. After all, we're having fun, and that’s what matters!

So when we want to add functionality to agents, a lot of times we just add a new tool and include more line graph code and additional conditional edges. Very often, we start kind of overdoing it. I know this happened to me a lot at the beginning. The agent can become confused when it has to manage 10 or 15 tools. Additionally, it may be performing five tests simultaneously.

And this is achieved by having 10 tools or 15 tools, right? It will be performing five tasks at the same time. This is the moment when you begin to think, what if my agent has a tool that is another agent? This marks a significant point in the development of multi-agent systems and helps you understand the challenges that come with them. The advantages of building these multi-agent systems are that we can improve a specific task of our agent without regressing in the performance of the other tasks.

So, because it's two separate agents, we can leave one alone and just improve the other when it needs to be improved. We can attribute blame to a specific part of our system and even have different teams working on different agents. This specialization is also really useful because we can give the agents even more tools and more context, as each one performs a specific task really well instead of attempting to do fifteen tasks not so well. Another significant advantage of this approach is that we get to decide what the agents communicate to each other. Now, I’ll briefly discuss different types of architectures for these multi-agent systems. There are three common patterns that I see.

So, one type of network involves agents that are all at the same level, allowing them to communicate with any other agent they choose. For example, a front-end developer and a back-end developer would be at the same level of hierarchy, enabling them to collaborate and discuss needs, such as "Your API needs this" or "Your front end needs that." On the other hand, there are supervisor or hierarchical architectures, where a top agent instructs subordinate agents to complete specific subtasks. For example, a software engineering manager might tell a front-end developer to work on the front end and instruct a back-end developer to focus on the back end. In this scenario, there isn't necessarily direct communication between the front-end and back-end developers, as the manager oversees both. More realistically, customer solutions often adopt a hybrid approach. You won't develop around a specific hierarchical architecture or a precise network architecture; rather, you'll implement whatever fits your use case. This is what I observe most frequently in real-world scenarios. Additionally, when considering multi-agent systems, it's helpful to recognize the complexity of building a graph for each agent involved.

And probably this is where messages are going to be effective, right? Definitely. Here, we kind of talk about it. I just wanted to leave you with a thought about what each agent should see, and then we're going to go over a very brief use case. So when Agent A calls Agent B, we must think about what that agent needs to see. Because certainly, the answer isn't always everything.

They don't need to have the same context. Let's say the manager is telling the backend developer to develop a backend feature. The backend developer doesn't need to know what motivated the manager to want that feature, right? If the CEO spoke to him and then the user sent a request, it doesn't matter, right? All the developer needs is the context to perform that task, right?

If the CEO spoke to him and then the user sent it, like it doesn't matter, right? All it needs is the context to perform its task. When the backend developer comes back to the manager with the feature in his hands, all the manager wants to see is the feature. He doesn't care how he got there. This is a good analogy for how we should think about this communication.

You need to give it just enough context to perform its task and not one token more. Otherwise, it will confuse the other agents. This is kind of the example that I wanted to show you, so let's build an R&D group. Let's say we have a simple graph of agents. It's a graph, and inside there are agents with half messages.

So the user asks a backend agent to develop a backend, and then let's say that the backend agent only has the option to ask the frontend engine to develop a frontend for this backend. This is a very simple workflow. It's not necessarily correct in terms of software engineering practices, but let's assume it exists. After the backend agent completes this logic, it will need to communicate and will ask the frontend agent to develop a frontend for its REST API, right?

But the front-end agent doesn't need to see the whole back-end code. It just develops its front end based on the API documentation. Additionally, the front-end agent doesn't need to return all its thoughts on why it decided to develop the front end the way it did. So, all we're left with is just enough information for these agents to perform their tasks. In the end, the user only receives the back end; the front end is not even aware that all of this happened.

And I think this is a good example of minimal communication and patients getting good results. I believe that's it for my agent. Before I go, I just want to wrap it up and say thank you to everyone who stayed for the 90 minutes we were here, especially those who joined from India and Israel. I’ve noticed quite a few participants from there.

So, thank you so much. And, Jeff, I think we do have a community. We are running both from there, so thank you so much.

And Jeff, I think we do have a community. We are running both sites on-site and remotely. I recommend that you join our meetup group. If you sign up for this one, you'll receive an email with the details, some references, and of course, the recording of this session.

TensorFlow is a Google Cloud partner. If you are interested in building applications or exploring some of the technologies that Jeff showcased here, we would love to hear from you. We can even assist with some credits to help you test out your initial applications. We are eager to learn about your use case and see how we can support you in getting onboard with Google Cloud.

And eventually, what do you see today? We asked ourselves, are AI systems going to be more autonomous or more continuous? Are there going to be more instances of, "Here’s a message, go do this stuff for me, and come back two years from now with the right answer"? Or will it be more about the continuous collaboration between humans and AI? We are seeing examples of both.

And I think the architectures need to really adjust to what you're trying to build and the way that LLMs communicate with each other, the way that they communicate with the environment, and the way that they organize themselves, whether it's doing planning or orchestration. All of that needs to really support the use case and the product vision that you have. We've seen that context and orchestration really are the key elements. So LLMs are going to get smarter. Google's going to come up with a Gemini 2, something, and maybe a 3. I don't know what the branding of the next model will be.

Maybe Jeff smiles because he knows. But the models are going to get smarter. The whole question is how you're going to connect them to the environment. You've seen it being done through prompt engineering, and you've seen it done through showing engineering.

You think you've seen it done by, um, showing images and displaying the screen of your computer. However, there are many more ways to introduce context into your LLM systems. What you eventually realize is that these elements can work together. If you think about an agent as a unit that performs specific tasks—typically, for example, an agent that answers support tickets—then when considering the next level of AGI, it will operate at an organizational level.

So, you're going to think about AI that collaborates with AI to create organizations. There, the whole topic of AI communicating not just with humans, but with the environment and with each other, and grounding is going to be extremely, extremely crucial. So, the last word is going to be to choose the best-fit architecture for a use case. There is not a single one; there's probably one that fits your case the best. With that, I really want to thank everybody who showed up, and we look forward to seeing you in the next one.