How to Make AI Agents that ACTUALLY WORK

All right, we're just about ready to get started with our next session. Patrick here is going to bring the chill vibes instead of the music. How's it going, Patrick? I'm doing great. How are you, Skylar?

Great, great! I’m having a wonderful day hosting this conference on Track Three. We’ve had a lot of fantastic talks so far, and I’m super excited for the next one!

So I'll just kick it off with a quick introduction. I'm going to take it away, but Patrick is here, a staff engineer from the Vertex Applied AI Incubator at Google. The big thing you should know about Patrick is that he just published a white paper on agents with some collaborators, leveraging his wealth of experience in working on conversational AI for a long time. You should definitely go check that out.

It just dropped on Kaggle, so that's definitely a way to read up more on these ideas. But without further ado, let's jump into it. Let's take it away. Cool.

Thanks, Skylar. And thanks, everyone, for having me today. You know, it's been a really cool conference so far. I've been watching a lot of the talks, and they're all really great.

I got to meet some llamas, and I thought "Dalai Lama" is the best name for a llama ever. It's really great! You know, it's funny—when Demetrius first asked me to do this presentation, I was going to put together something about agents with Gemini and things like that. I started noticing that everyone was kind of doing presentations on how to build an agent.

So I made a slight pivot, and I'd like you to accompany me on this journey today. Really, what I'm going to talk about are the lessons learned over the last couple of years of delivering generative AI agents and being in the trenches. To get started, a little bit about me: my name is Patrick Marlowe, and I'm a Staff Engineer at the Vertex Applied AI Incubator at Google.

It's a mouthful. What does that mean? You know, my team and I work on a lot of cutting-edge stuff when it comes to large language models, function calling, the Gemini SDKs, and things like that. I have been in the conversational AI and NLP space for a little over 12 years. Right now, my focus is on working with multi-agent architectures, memory usage, and understanding how agents use these tools efficiently. As Skyler just mentioned, I dropped a white paper this morning, of which I was a co-author. It was really cool to be a part of that. So, if you're interested in agents—and obviously you are, since you're here—go check out that white paper!

I'm also a large contributor to the open source community. I manage a couple of open source repositories at Google and contribute to LinkedIn as well. I really enjoy that part of large language models in the agent community and related areas. It's really great!

Okay, so let's get started. I'm going to do a super brief history of large language model applications, and then we'll get into the lessons learned. So, in the very early days when we first started this whole generative journey, there were basically just models. As a user, you would send a query into that model, and the model would logically respond with a bunch of amazing tokens. You would look at the answers and say, "Hey, these are really great." However, one of the things that we noticed very quickly was the concept of hallucinations; the model didn't always get things right or was confidently wrong.

And so very quickly, we moved on from the architecture of just models to the concept of retrieval-augmented generation, or RAG. I would say that 2023 was definitely the year of RAG, as that was when everyone started implementing RAG solutions. This also led to the rise of amazing companies in the vector database space. Now we have vector databases that allow us to create embeddings for all different types of data we want to store.

And now, when we send those queries in, the model can actually ground itself with external knowledge from these vector databases. We had fewer hallucinations. However, it was still somewhat like a single-shot architecture. There was a query submitted, followed by a retrieval, then some generation, and we were done—right? The end.

But what we were really lacking was the ability to add another level of orchestration around that. And that's really where we get the idea of agents. So, agents kind of emerged in late 2023 into early 2024, or at least became more mainstream, I should say. We started to see this concept of reasoning with agents and orchestration with agents, allowing for multi-turn inference and similar capabilities happening under the hood. You know, agents have access to various tools and sometimes multiple models, which enhances their functionality.

The concept of agents that we’re going to be discussing today, particularly in the context of delivering in production, revolves around a specific architecture. There are many variations of this architecture, but generally, this is what we refer to when we discuss agents in production. As we move towards production, the first point I would like to emphasize is that production is not just about having a simple model. Over the past couple of years, there has been a significant focus on questions like, "What LLM are you using?" or "Are you using version 01?"

Are you using 3.5 Turbo, or are you using Gemini Pro or Flash? There's a lot of hyper-focus on that model, but the reality of the situation is that your production agents are more than just the model. In fact, there's a lot that goes into it, right?

That's whether you're taking that model and adding grounding to it, or adding tuning, or all the prompt engineering that goes around that. You really start to see that more and more gets layered into this, and you begin to realize that there's an entire system design practice that goes into building out an agent, including orchestration, API integrations, CI/CD, and analytics. There's so much that truly goes into it. So, you realize that an agent is, again, not just a model anymore; you're really starting to think about your software development lifecycle best practices, how you do system design, and how you put everything together. If I'm predicting the future here, I would say that at some point, many of the models we're looking at are all going to become commoditized.

They're all going to be fast. They're all going to be good. They're all going to be cheap. And so really what you're left with is that ecosystem that you're building around your model to make things truly great.

And that's kind of where your secret sauce comes into play for building your models and deploying your agents in production. Today, I'm going to touch on a few of these aspects and summarize them into three main points. I will discuss a concept called meta-prompting, touch a little on safety and guardrails, and then finish up with evaluations. After delivering hundreds of models into production with various developers, customers, and partners, these are the three consistent themes we observed across everyone who was successfully deploying high-quality agents in production.

And so I want to share all that information with you today. First, we'll start with meta prompting. The TL;DR around meta prompting is that, essentially, you're using AI to build AI. Now, a meta prompting system—kind of the whole architecture—starts simply like this: you have some sort of meta prompting system, which we'll discuss in a little bit more detail shortly.

And that meta-prompting system is going to generate some prompts for a secondary system. We will call that the target agent system. This is the agent you will put into production. That agent will then produce some sort of responses, which you can evaluate. You can send those evaluations back into your meta-prompt system. Thus, you can go around the circle as many times as you like to optimize the types of prompts you will be sending or using in your target agent system.

So, to give you a little bit of intuition about how something like that might look in production, here is a handcrafted prompt that I wrote. I'm saying, "Hey, you're a Google-caliber software engineer. You've got exceptional expertise in data structures and algorithms. Here are the things that you can do: you can solve coding problems."

You can debug code, write docstrings, perform reviews, and much more. This represents a typical type of prompt that we might encounter in production or that someone might create from a prompt engineering perspective. The key point is that when we actually feed this to a meta prompting system, we are asking the language model (LLM) to take this prompt and rewrite it with higher fidelity—essentially embellishing and adding more descriptions to enhance clarity. This way, a secondary LLM system will be able to use it with greater accuracy. After processing it through an LLM optimizer using this kind of meta prompting technique, you receive something like this.

Now, semantically, it’s pretty simple—pretty much the same as the previous prompt that you saw. However, the difference lies in how it gets structured and the language that is used. I like to think about it in terms of how humans aren’t always necessarily great at explaining themselves. What you’re doing here is leveraging the LLM's capability to write and elaborate—not necessarily embellish, but rather describe in higher fidelity and more detail the tasks that you want completed. There’s a significant difference between putting something like this into production versus putting the small snippet that I was just showing previously into production.

To illustrate how that architecture could come into play, let's discuss one of the initial concepts for implementing meta prompting, known as seeding. Essentially, this process begins with your meta prompt system and a specific system prompt. This system prompt might look something like this: "Hey, you're an expert at building virtual agent assistants. You're going to help this user write prompts for this virtual agent assistant system."

Then you, the developer, would write what we'll call the seed prompt. The seed prompt is essentially the same type of prompt that you saw me write previously in the handwritten prompt section, but I'm going to provide a bit more information. I'm going to say, "Hey, you know, the end user's company that you're writing this prompt for is Google Cloud. They're a software engineer, and here's the task that we want to have accomplished." What happens next is that the metaprompt system will generate these target agent prompts. You can go around this loop as many times as you want, refining that prompt in any way you desire and modifying it as needed.

But when you're satisfied with the prompts that have been generated, you then drop those into your target agent system. This is a really great way to build an agent for the first time, especially if you're not very skilled at prompt engineering. You might think, "Hey, I'm not a great creative writer, but I need something high fidelity to really kick off my first agent." It's an excellent system to use to start your agent's prompts efficiently. Now, the second way that we can use metaprompting is in the concept of optimization. The way optimization works is by taking this system a little step further.

Once we put our agents into production, our agents will start producing responses. We can take those responses and evaluate them against different types of metrics. Many other speakers today have discussed evaluations and related topics. Imagine evaluating your responses based on coherence, fluency, semantic similarity, and similar criteria. Essentially, what you do is take those evaluations and send them back to your metaprompt system.

And you say, "Hey, given this existing prompt that I have, and here's some set of evaluation data points that we now have, can you optimize my prompt for better coherence? Can you optimize my prompt to reduce the losses in my tool calling?" Right? And then what this meta-prompt system will do is, again, just kind of go around that cycle and write better prompts to optimize for the metrics that you want to improve. Now, you might be looking at the system, and you might be feeling a little bit like this "This Is Fine" meme, where it's just like, you know, we're writing prompts to write prompts to produce prompts to do all kinds of prompts. And like, I totally feel you on that.

The cool thing is that there are a lot of really great systems available for prompt optimization. If you're following along, take a screenshot here or capture this with your phone. This will take you to a YouTube video that goes into more detail on how to perform meta prompting with Gemini, OpenAI, Claude, and other systems. Additionally, if you're familiar with them, there are excellent open-source systems like D-Spy, Adolfo, and Google's Vertex Prompt Optimizer. There's a wealth of fantastic tools available for prompt optimization.

So, it's a little bit about meta prompting. We're going to move on to the next topic. The next topic we'll discuss is safety. I think for a lot of users putting their agents into production, their architecture might look something like this. Especially when deploying an agent for an internal use case, they might think, "Hey, my users are all super friendly. I'm going to provide them with a user interface, and I'll implement some prompt engineering as my 'defense layer' against my agent, so they don’t do anything foolish."

And then I'm basically going to let this cycle repeat, right? But the reality is, if you put those agents out into the wild, like you actually let them go into the public domain, there are a lot of bad actors out there. Those bad actors are going to be looking to break your agent. They're going to be looking to do prompt engineering, and they're going to try to figure out prompt injection.

You're trying to figure out ways to, you know, kind of game the system that you've built. It’s important to think about that ahead of time and start implementing multi-layered defenses against these types of threats. It's not always about prompt injection, right? This conversation really starts with considering input filters. When a user query comes into the system, you can implement measures such as language classification checks, category checks, session limit checks, and similar strategies.

So, you know, for example, there are many prompt injection techniques that unfold over the course of multiple turns. A simple safety measure is to limit the number of conversation turns to, say, 30 or 50. This is important because, if you examine the long tail of conversation turns, many of them are where bad actors tend to operate. Therefore, it is crucial to implement these types of input filters. On the agent side of things, most people already have their APIs secured or are implementing some form of safety filters, which is excellent.

But you also have to think about the return journey. Once the response has been generated from the agent and you're about to send it back to the user, what types of things are you doing to protect that payload? A lot of times, it's important to consider aspects such as whether you need to add error handling or retries. Does your LLM API sometimes return a 500 error? Being able to manage these types of issues, control generation, and handle JSON outputs is crucial for creating a really safe solution.

The other thing that we found that users often overlooked is the concept of caching. I think there's a tendency for developers to always want to build the latest and greatest systems and use the coolest technology. However, if we take a step back, the only thing that truly matters when putting these systems into production is the outcome they achieve. It doesn't matter whether they achieved that outcome with the latest, top-of-the-line model; what matters is that they delivered a high-quality result.

And maybe that high-quality outcome comes from caching responses. You can completely bypass your agentic system by simply knowing that you're going to have the same query over and over again. Cache the response to that query and send it back to the user. You don't even have to involve your LLM (Large Language Model) or your agent system at all. You save money on tokens, and you gain a lot of speed as well. These are just some things to consider as you're building your agentic system and putting it into production. The last piece is really about feeding these signals back into your data analyst teams, your data science teams, and others. This can help you take a closer look at analytics and see what people are talking about. Using those signals can inform how you update your prompts, your input filters, or your output filters, and all of the other aspects involved.

So, a lot revolves around safety and creating this type of layered defense. Think about this as you put your agents into production. The last topic we will touch on today is evaluations, or evals. You might be thinking, "Patrick, what are evaluations?"

Why do I need them? The one thing I would say is that if you're building agentic systems, the number one thing you could do is implement evaluations. If you don't do anything else, just implement evaluations because it really helps you provide measurable insights and serves as a barometer for how well your agents are performing in production. Evaluations are simply a process to measure the performance of your agent and identify losses and areas for improvement. So, imagine a scenario that goes something like this.

We've probably all already gone through this. You build a new agent framework, or a new agent, and you launch that into production. Everyone's high-fiving and patting themselves on the back. We did a really great job!

Prod. Awesome. Right. And then you go to release your next feature that's going to be attached to that agent.

Maybe you're adding a new tool called "Connection" to a database. You change your prompt a little bit, but all of a sudden, all of your users are coming back and saying, "Hey, this thing is garbage. Now it's not responding correctly. It's hallucinating. There are all these issues going on."

And you and your team are sitting there, wondering what's going on, right? You're starting to look at manual processes, basically inspecting the data manually for every single response that’s coming back from the agent. In my base agent, the responses were coming back like this, but in my updated feature agent, my responses are coming back like this. Why is this happening? You might feel a bit like our friend, Mr. Vincent Vega here, just wondering what's going on.

How do I solve for this? And why am I having to do all this stuff manually? Fortunately, there are many amazing frameworks available that assist you in evaluating your agent systems, your RAG systems, your LVC-LEM systems, and everything in between. They all fundamentally begin with a concept called a golden dataset. Some people also refer to these as expectations.

It's all really the same kind of stuff. What this boils down to is essentially defining the ideal scenario for what happens when someone interacts with your agent. So, when a user says this, the agent should respond with that. When a user responds with this, the agent should say that. When a user responds with this, the agent should call a tool with these inputs, and then it should respond accordingly.

These expectations allow you to take them and compare them against your agent responses, specifically the things that are actually happening at runtime and inference time. You can then score these against a variety of metrics, such as semantic similarity, tool calling, coherence, fluency, and more. As you iterate on your agent, your expectations remain mostly static, enabling you to observe changes or variations in performance as you deploy changes over time. For instance, you might notice that your tool calling is suffering, which is causing a decline in semantic similarity and impacting user satisfaction with the system. Evaluation serves as a way to measure what is happening in your agentic system. To illustrate this further, consider a real-world scenario that we frequently encounter with our customers and partners, specifically evaluations related to multi-stage RAD pipelines.

So I'm sure all of us have built something like this before. Essentially, imagine you have a user query that comes into a system, and you have a model that performs a query rewrite on that input query to fix spelling errors or similar issues. Then, that query goes into a retrieval system, such as a vector database, which fetches various pieces of information. At the end, you take that query along with all of the retrieved information and send it to a model again for summarization, resulting in an output. You’re evaluating the output of this pipeline and assessing it, saying, “Hey, cool, we're getting a really great score here.”

And then someone says, "Hey, Patrick, you can actually improve this score if you use a re-ranker in the middle." So then you implement re-ranking in the middle, and you see that your score is going up. You're like, "Hey, everything is going great!" Then let’s say your vendor comes back to you and says, "Hey, we just launched this brand new model. It's the best thing since sliced bread. You've got to put it in everywhere in your systems because we think it's better than everything you’ve built before."

Right? So you say, "Okay, cool. I'll try your new model." You throw that new model into production, and then what happens? All of a sudden, your evaluation pipeline is saying, "Hey, things are suffering."

Your responses are coming back really poorly. The interesting thing is that you don’t really know why that’s happening, especially since you're using that model everywhere. Is it the case that the query rewrite is suffering, or is it the summarization that’s lacking, or perhaps the re-ranking that’s affected? You don't really know what's going on because you're only evaluating the outside of that pipeline.

So, this is where it's also important to think about evaluations not only from an end-to-end path perspective, but also at every stage of the pipeline. This means performing evaluations on the query rewrite itself, as well as on retrieval and summarization. Essentially, this allows you to start to determine what is happening at each stage and identify that, for example, the largest losses we are seeing are occurring during the summarization stage. In that case, we could revert to the previous model for summarization. By having two models in production, we can achieve the highest quality results.

So again, the takeaway here with evaluations is that if you're not doing anything else, you absolutely have to be conducting evaluations on your agents in production. This is essential for understanding what is happening at each stage of the process. If you're looking for a follow-up resource or a takeaway, take a screenshot of this or capture it with your phone. This will direct you to one of our open-source repositories, where we have a wealth of notebooks and code on how to run evaluations using the Vertex SDK and the RapidEval SDK model. You can also do this with many other frameworks; it doesn't really matter to me.

What's important is that you're actually doing evaluations on your production agents and understanding how to use some of these frameworks. Cool. All right, wrapping everything up and then coming back around to Q&A as we have time. Remember, the lessons learned here are essentially that after working with developers like yourself and after collaborating with partners and customers, we have identified the most common practices that people are trying. These are the most impactful findings.

At building high-quality agents in production, we utilize various meta-prompting techniques or prompt optimization techniques, as well as implement different stages of safety, guardrails, and error handling. Additionally, every single engagement we have worked on included some form of evaluation. Without evaluations, there is really no way to determine what is actually going on within your system or how well it is functioning. With that, I will bring Skylar back on, and I guess we will jump over to the Q&A.

We can kind of go from there. I'll leave this up if you want to connect with me as well. Awesome. Thank you so much. There are a few questions in the chat.

We'll just jump right into it. First question: When do you plan to release the Vertex AI Agents API to utilize the most powerful Gemini models for building assistants? Well, I can't give you a specific answer to that because many of these features are still in preview. Unfortunately, I can't provide timelines for when these will be released, but rest assured, they are being worked on. As of today, if you're looking for something that is already available as a general availability product for building agents, we have the Conversational Agents platform.

This was previously known as Dialogflow CX. The Dialogflow CX platform, a conversational agents platform, has a fully functioning API. With this API, you can build all of these agent experiences without having to go to the user interface. One of the open-source libraries that I manage, called Scrappy, is actually a wrapper around this. So, I suppose this is a shameless plug: try out Scrappy to build agents!

It's really, really fast to build agents. You have access to all of the latest models in that framework as well. Awesome! How do you think about managing versions of agents? Yeah, this is a really interesting one.

It comes up all the time. So, you know, for me, I think the easiest way that we have found to manage versions of agents is to break up the agent into all of its individual components and think of it all as code. The way we manage agents internally is by pushing all of the prompts, all of the functions, all of the tools—everything—into Git repositories. Essentially, we treat everything as code, which even includes the prompts themselves.

And so you're looking at what the differences are between these prompts. Do we need to roll back to previous commits and things like that? Honestly, the best way to manage versions of agents is to treat them the same way you would in a traditional software development lifecycle with CI/CD. Awesome. We're out of time.

This is a great presentation. I really loved the key takeaways we had here, and I thought they were simply summarized. So, everyone, Patrick has left his information here, so go ahead and connect with him. With that, we'll move on.

Zeon Ford. Thank you for your time, Patrick. Take care. Cool. Thanks, everyone.

Thanks, Scott.