Building Reliable Agents // Eno Reyes // Agents in Production

Next up, you know, let's see, are you around? Hey, how's it going? Okay, I don't know if you—where are you located? I'm based in San Francisco. San Francisco, okay, so far enough away not to have to swipe on the llama.

One of the points we discussed in our last conversation highlighted some of the challenges of distributed systems, even in more traditional software. This becomes even more complex when you introduce agents and the need to establish a new understanding of reliability, especially in the context of multi-agent systems. It sounds like an increasingly significant and difficult problem to solve, and it appears to be something you have been dedicating your time to. So, I will let you share your screen.

Okay, cool. You know, I'm very excited to see what you have to share with us and how to build AgentEx systems reliably. I'll be in the background, taking in some questions. So, I'll come back in about 20 minutes. If folks have questions until then, please write them in the chat below, and I'll get to them very soon.

Eno, the floor is yours. Amazing. Well, I appreciate you having me and welcome to everybody here. My name is Ina Reyes. I'm the co-founder and CEO of Factory.

And today I'm going to be talking about, as the slide says, building reliable agentic systems. This slideshow is essentially a quick overview of some concepts that the factory team has encountered and some ideas that we have applied to help us raise the bar on reliability for our core product suite. For additional context, Factory's mission is to bring autonomy to software engineering. What that means, concretely, is that we are building a platform that helps software engineers in organizations, both small and large, collaborate with AI systems so they can manage all of the tasks that a software developer might need to handle. This includes coding, but also extends beyond that.

A lot of tools are very focused on, for example, the IDE experience and writing lines of code. As we see it, there is a lot of overhead to being a software developer that touches code, documents, and aspects like review, organizational practices, and onboarding. Across this entire range of tasks, there is often a need for not just a simple one-off AI interaction, but rather a broader, multi-step complex workflow. Therefore, bringing autonomy to software engineering means we need to build systems that can handle the complexity of real-world environments. To kick this off, I like to start with a definition because the term "agent" is really overloaded these days.

There's a lot of discussion about agents. Every kind of AI company claims to build agents. But what really is an agentic system? There are many different traits and features that can be associated with agents. However, what we've found is that, among the various workflows—and some people use the term "cognitive architectures"—there are certain boundaries, but it's actually quite difficult to draw an explicit line around what constitutes an agent.

But there are three characteristics that we think, or we like to think of as being characteristic of an agentic system. These are planning, decision-making, and action execution. Agentic systems make plans that determine one or more future actions of the system. This can be simple, such as outputting a single action, or it can be complex, like a multi-stage, multitasking plan that potentially coordinates multiple actors. Agentic systems also make decisions.

That means they evaluate multiple pieces of data and select their next action, state, or decision based on, typically, an algorithm or selection process. A lot of people will sometimes call this reasoning if the decision-making process is sufficiently complex, or maybe even to us, it may seem semi-unexamined or explainable, and we can't really point to the exact algorithm. The third thing is, and arguably the most important, the environmental grounding. What this means concretely is that agentic systems read and write information to an external environment, which enables them to achieve goals within that environment. Ideally, all of this information that the agents are able to react to and adapt to helps them achieve their core goals.

And so, you know, what we're going to do is actually jump into these three concepts. Basically, I'm going to talk through some high-level problems and some solutions that our team came up with to address these issues. So let's talk about planning first. One of the biggest challenges with building long-term plans with an agent is that they tend to drift away from their core responsibilities. If you've ever interacted with tools like Baby AGI or many basic agents, you might notice that when you ask them to do one thing, they often stray from that task.

And, you know, if that task requires a bunch of different individual substeps, it's very difficult to get it to converge into a single path. Inspired by the Kalman filters used in robotics and control theory, our team has put together a couple of techniques that basically pass intermediate context through the plan to ensure that the plan remains focused and reliable, promoting convergence. Obviously, there are some drawbacks; for instance, if you are off in your initial prediction of what the next step should be, or in an individual step of reasoning, that error can propagate as you pass that through each additional step. The second technique we have encountered is subtask decomposition.

The second technique that we've encountered is subtask decomposition. Basically, this means that if you break down your complex plan into smaller, more manageable tasks, it allows for finer control over the execution of that plan. This is particularly helpful when trying to understand exactly how to solve complex problems. However, the risk is that it introduces a higher potential for including an incorrect or misleading step, and even one misleading step could be problematic. The third technique is what we call modeling.

Predictive Control This concept is another idea taken from robotics. If you've ever worked on or around self-driving cars, you're probably familiar with it. The essence of predictive control is that planning is dynamic. In most complex tasks, you need to constantly reevaluate what you are doing and what the next step should be. Consequently, your initial plan at the beginning of the task will likely look different from your final plan once it has been executed. Adaptive planning involves continuously adjusting your approach based on the evolving circumstances, making it distinct from your initial plan.

Real-time feedback can be a super helpful way to ensure that your agent is able to dynamically adjust to problems or issues that arise. The benefits here are pretty obvious. However, the downsides are a bit harder to track; you really do introduce a significantly higher risk of straying from the initial correct trajectory. This ties back to the first point: if you are tackling complex problems with agents, you need to consider how to keep them consistent while also being able to adapt to new information. Finally, the last thing we found to be quite helpful is clearly defining what your plan criteria looks like.

You know, the success criteria and structure of a plan will ultimately determine how successful and competent your agentic system is. This can be implemented in a variety of ways, whether it's through instruction prompts or few-shot examples. You can also perform static type checking of your plans. One major thing I won't do is be too prescriptive about how you implement these techniques. Since all of you have different systems, it's challenging to provide specific instructions.

But hopefully, if you've built an agent or want to build one, this serves as a helpful reminder that we need detailed instructions and clearly defined success criteria. This way, the agent isn't just starting from scratch each time. The downside, however, is that it can be challenging to build and scale multiple explicit criteria. In a very open-ended problem space, it isn't always easy or obvious what the criteria should be, as you may encounter completely different versions of the same problem. So that covers planning; now let's move on to decision-making.

You know, one of the biggest issues with LLMs is that they are next-token predictors, right? So the output of an LLM, which is likely to be a core part of your agentic system—although people build agentic systems without LLMs—is ultimately conditioned on all of its prior inputs. This means that as you build out a response, every additional layer of your response is influenced by that initial thought process. Therefore, there is a need to sample from multiple different outputs in order to get a better sense of what the correct answer might be.

And so we call these consensus mechanisms. They can encompass a wide variety of approaches. People have experimented with methods like prompt ensembles, where a variety of different prompts that vaguely recreate the same question are used, and then all the responses are aggregated and analyzed. Another method is cluster sampling.

So imagine getting 20 answers from the same prompt and clustering the responses that look similar. Then, for example, you could sample from each of those clusters and compare the results. Finally, there's self-consistency, which is probably one of the easier methods. This approach simply involves outputting responses multiple times and taking the most consistent answer. All of these methods tend to improve the accuracy of decision-making, but they ultimately require greater inference costs, leading to a trade-off. The second kind of decision-making method is explicit reasoning or analogical reasoning.

And this is actually probably one of the best-studied LLM techniques and confidence-boosting strategies for LLM-based systems. You've probably heard of concepts like chain of thought, checklists, and chain of density. People have even gotten really creative with the idea of "tree of thought." By explicitly outlining the reasoning strategy or using analogies to guide decision-making processes, you can significantly reduce the complexity of those processes, making it much simpler for an LLM-based system to accomplish tasks. The advantage here is that with reduced complexity, you're able to achieve significantly more consistent responses.

And consistency, though it doesn't always mean accuracy, does imply that the process followed will tend to look quite similar. The downside here is that it reduces the creativity or natural instincts of the LLM because you are enforcing a reasoning strategy. If your reasoning strategy is not sufficiently general or is simply incorrect for a given problem, it will significantly diminish your agent's ability to solve that problem. There are trade-offs involved in this approach. It felt necessary to mention this, but one obvious way to improve your agent's decision-making ability is to work on the model weights.

So, if you have an open-source model that you know is really good and you have specific tasks in mind, such as decision-making tasks like classification or tool use, you should probably consider fine-tuning or otherwise impacting the actual end model weights. The significant downside is that this process is expensive and can lock in quality. Therefore, if you have a known distribution and want to improve the quality within that distribution, this is a great strategy. However, we have found that fine-tuning is rarely the actual solution. Often, base models are not only good but actually better for many problems because they excel at handling larger distributions of tasks.

But your mileage may vary, and you may be building a system where fine-tuning is the problem to be solved. Probably the most interesting aspect is simulation. This is a pretty advanced decision-making strategy, and if you can operate in a domain where you can simulate decision-making and explore multiple decision paths, it can be extremely valuable. The basic outline of the problem is that you want to sample and then simulate multiple decision paths forward, ultimately determining the optimal choice based on some reward criteria. This doesn't need to be a full...

The RL experiment, language agent tree search, which I mentioned earlier, is a pretty advanced technique, but a very doable one for exploring decision trees. It basically imitates Monte Carlo Tree Search using LLMs. You can imagine that if you're operating in a domain like software development, where you have tests that can be executed, and the success criteria of that test is whether it passes or compiles, you're actually able to model these decision-making processes with a path that leads you at least toward whatever reward you wanted to set up. The benefit here is generally improved quality.

The cost here comes with a couple of downsides. One, this can be slow; two, this can be expensive. However, what we found to be the biggest downside is that this process is really difficult to model and debug. You are incorporating a lot of assumptions into your system when you build these simulations and rely on them for decision-making. Therefore, you must be very cautious about developing these simulations.

And the scope of what you're trying to simulate and where you use this can really impact the amount of time your team is spending, the quality of the outputs, and all of these kinds of factors. I know this is pretty rapid-fire, but hopefully, you're taking notes. If you have any questions or if you're honestly just building the space as a nice point of reference for how other people are thinking about building these things, please feel free to reach out. Now, onto environmental grounding. One of the most critical aspects of building agents and agentic systems is the interfaces that your agentic system uses to access the rest of the world. Therefore, it is essential to build dedicated tools for your agentic systems.

And in particular, if you're building an agentic system that interacts with computers and systems, you should really be thinking about where you want the agent to be able to access things and what abstraction layer you want your agent to be operating on. For example, a calculator is a very simple tool. If you give an agent access to a calculator, it will be able to do math a lot better. If you give access to a custom API wrapper around a couple of key actions in a platform, the agent is going to start to become a lot more fluent in that platform. If you give your agent access to a sandboxed Python script that allows it to execute potentially arbitrary code to achieve its goals, it will obviously be able to do much more.

But with that breadth comes decreases in accuracy. So, as you move up and down that layer of abstraction, you should think about where you want your agent to be operating and where it can be most effective at taking the reins of the tool, and where you actually need to hard-code or define those interfaces. One significant downside here is that it can be difficult to design and maintain these tools if they are agent- or LLM-specific. The second biggest consideration for environmental grounding is explicit feedback processing. Your agent is likely to be operating in a complex environment, which means that the feedback from different interactions and processes will be quite substantial.

One example of this is that if you were working with coding agents, you might frequently encounter logs. This could amount to truly hundreds of thousands of lines of standard output and debug logs. Therefore, when you think about this log data, pre-processing it is critical to building your agentic system. If you consider humans, we ingest huge amounts of information. You've probably heard this before: we consume vast amounts of data. The bandwidth of our eyes, ears, and sensory capabilities is enormous.

But we focus on the details and the information that matters. Agents don't have a complex biological architecture, so you actually have to define their signal processing and feedback processing. If you can parse through the noise and provide your agent with more critical data, it will be significantly better at solving problems. The third point is bounded exploration; you really need to allow your agent to gather more information or context about the problem space.

And so, if your agent charges off on a task without any information gathering, or if it only conducts an initial round of information gathering and then runs autonomously, this could be effective if your agent is tightly scoped. However, as you build increasingly general systems, the need for your agent to retrieve, learn, and otherwise acquire more information becomes critical. This helps the agent understand the problem better. Nonetheless, you also risk overloading the context or straying from the key path, as retrieval can be challenging and obtaining the accurate information may be tough. Finally, human guidance is essential.

So, this is kind of like a final thing to mention, which is that you are probably building an agentic system that will somehow interact with a human. Therefore, you should involve the human's input to improve your agent's reliability. This requires careful UX design, thinking about where it makes sense to intervene or interact with an agent. You should really take the time to think deeply about this. If your opinion is that you'll simply build the agent without further consideration, I encourage you to rethink that approach.

And once the reliability increases, we won't even need it to interact with humans. I would recommend rethinking how you're building this system because, at the end of the day, most of these systems are not going to be reliable enough to perform large tasks independently 100% of the time. So even if you disengage and are unable to accomplish a task, let's say 10% of the time or 20% of the time, at scale, that actually represents a significant issue. Therefore, you need to carefully consider the design of these interactions.

So, that is pretty much it. Hopefully, this was interesting and sparked some ideas. I just wanted to put this out there. We are hiring! If you believe that bringing autonomy to software engineering is a compelling problem, we genuinely think this is the highest leverage problem in the world. Feel free to email me or the team if you're interested in joining us.

You know, thank you very much for this. The folks in the chat absolutely loved the presentation, and we'll be sharing this out as soon as the recordings are ready. I hope this one is one of the recorded talks; I know that some of them are.

I'm very curious about the kind of work that has led you to think so abstractly about so many of these topics. I mean, you have managed to pack so many different ideas into this. I have a bunch of questions, but I doubt we will get through most of them.

Let me see if anyone in the audience has dropped a question. Okay, no questions yet for you. So, let me try to take a stab at this. In one of your earlier slides, you mentioned trying to find ways to ensure that the reasoning converges. Do you remember that?

Any tips on how to go about that? Yeah, absolutely. This is quite challenging because if you have a very simple problem, I can try to ground it with very concrete examples. Let's say you are building an agent for code review, and you just want your system to review code.

The boundaries here are actually pretty tight. When I create a pull request and there’s some logic around what justifies a code review for that pull request, I want a system to examine it. I want it to analyze the other components and then leave a code review comment. There should be some clear and undeniable criteria for this. If that system doesn’t result in a comment in GitHub, then it has failed. It’s relatively straightforward to streamline this process and assert that we should converge on specific reasoning down this path.

For a system that's like a general coding agent that you might want to task with something like, "Hey, I want to build a feature," there are truly a million different ways that your system can do that. If you have very strict boundaries, you can build that into the system. If you don't, you have to use techniques like the ones I mentioned, such as model predictive control, which is similar to common filter-like systems, in order to ensure your system continues down a path aligned with your original intentions. There's another question here from Akert.

If I'm pronouncing it right, what is the most differentiating factor? I imagine that this is how I always suspected the question arose. If you're trying to put together this agent, and there are all these different things that you can do, where should you start? How do you know that doing that specific thing will help you differentiate? Suppose everybody is using the same model.

What is nevertheless going to be the most useful in driving you to the best outcome? I really think that it is what exists in this slide right here: your ability to build a system. In particular, I would go as far as to say that the AI-computer interfaces are the differentiating factor. Most of the model providers are building really capable systems. O1 is particularly great at planning, backtracking, and reasoning.

A lot of this stuff is kind of, you get out of the box with these O1-style models. What O1 does not have out of the box is an integration, a tool, and a definition of that tool that works with some other system that you've built. So, if you are building agents, I highly recommend the interfaces as a place to start. Very valuable advice. Thank you very much.

Next, we have Brooke. To be honest, one of the things I wanted to do was ask you about the simulations you had there. I believe this will be exactly the topic that Brooke is going to present. Thank you very much. If you could please stick around in the chat to answer some of the other questions that folks have had, it would be greatly appreciated.

That would be lovely. Thank you very much, Eno.