Task Utility in LLM-Powered Applications

We're bringing on Julia from Multion. Julia has a background in researching user-based scale satisfaction metrics and methods for the last 15 years, dating back to web search scenarios. She spent six years conducting research on agent-related concepts at Microsoft and now works building agents at Multion, which has the tagline "Building Agents That Complete Tasks from Start to End." I'm very curious to see what Julia has to share with us. With that, I'll let you take it away, and I'll come back in for Q&A at the end. Thank you so much for the introduction.

I'm happy to be here and share some of my work on edge and civilizations. I just want to highlight what you will gain from this talk today. If you're interested in building an AI-empowered application, which is essentially an energetic space for various domains, you will want to understand the potential metrics you need to optimize your application for. Sometimes, it can be quite challenging to determine what needs to be done. For example, a great case study is web search, which has seen a tremendous amount of work and research conducted by a large community to identify the metrics that should be optimized from the user’s perspective. After all, we are building these applications for the users. I was fortunate to be part of Autogen.

So it's likely that if you looked at an agentic space, you've probably heard about it. This open-source programming framework was released in September of last year, and I think it has attracted a lot of attention. I was blown away by the idea of how many people are trying to use the agent-based language model-powered engines to solve all kinds of tasks. Especially within the Discord community, we saw people using it to build tools, teachers, analytics, and other applications. However, the challenging part of machine learning is actually determining your objectives—what you aim to accomplish, what you want to optimize for, and how you want to design your metrics. It's relatively easy when you think about tasks like object recognition, because you can get the ground truth, label the data, and focus on ensuring that your algorithm is performing well. But if you're building something for the end user, you need to consider whether your users are satisfied or dissatisfied with the experience and what your application is actually intended to do.

The motivation behind exploring a scalable approach for evaluating agent performance—or even just understanding the utility it may or may not bring to users—has driven our interest in this area. We have observed many different types of developers around the world building applications for users, and these developers need tools to determine whether the applications or agents they are creating are actually meeting user needs. Typically, evaluations are conducted in machine learning settings where ground truth data is collected and labeled. However, this is not always feasible, especially given the rapid pace of technological advancement. To effectively evaluate these systems, it is essential to first understand the objectives clearly. Moreover, substantial research has demonstrated that large language models (LLMs) can serve as an efficient alternative for gaining insights into what users actually want. They can also assist in evaluating the systems you are building, and this needs to be done swiftly.

And it's a relatively cost-efficient way of doing this. I will not go into detail, as it's not part of the stock. However, if you're interested, you could write many papers where you might highlight this LLM as a judge. Furthermore, this process needs to be automated.

And especially, it's important for tasks where you sometimes don't have any ground truth and want to figure things out by going through your data. Another important aspect of why we really wanted to bring Edge and Deval to the forefront is to empower developers and make them aware of the potential utility that the applications they are designing can bring to end users. We hope they care about this, so that as they improve and change their underlying systems, they know they are not acting in the dark. Instead, they have a tool to determine if their outcomes are getting better or worse. If I may be a bit more structured on this aspect, we've observed a wide range of tasks that users are trying to solve using LLaMA or LLaMA-powered applications. There are tasks where success is not clearly defined. For us as machine learning researchers, success is generally something straightforward, such as a binary outcome: whether the task has been completed or not.

For example, if you see a system writing an email for you, you can then copy this email and make changes here and there. So, what exactly constitutes success in this case? This remains an interesting direction for research. However, there are tasks where success is clearly defined. For example, the goal of your application is to solve the mess problem.

So, one of the benchmarks we all care about is that success is clearly defined. Because we have a ground truth, we know the right answer and can judge whether the solution is correct. Another aspect of these tasks is that when you have a preferred optimal solution, there are cases where multiple solutions lead to the same result, sometimes even incorrect answers. In such cases, you need to figure out which solution is actually better for you. I am bringing this up because what we're going to discuss next is a new way of defining the utility that the LLM (Large Language Model) application can bring to end users. We are trying to be as domain-agnostic as possible, which means we cannot really provide a concrete benchmark. The whole idea of benchmarks at the moment probably needs to be discussed in a separate talk. However, while we are building this approach, we want to show that it makes sense, so we need to start somewhere.

We decided, obviously, that it's going to be much easier to start with tasks where success is clearly defined and see what we can do for these types of tasks in this space. For this particular presentation, I will be using the task of solving mathematical problems, which is clearly defined. We have tested agents on various types of tasks, even those where success was not defined afterwards. You can look this up in the paper or online. First, we need to reconsider the idea that you need to define the utility because you are not building the application just for the application itself.

Help users solve specific tasks. Usually, you have certain criteria in mind. You're building a solution that performs a task faster than another solution, involving less or more effort. However, there may be aspects that you cannot really consider at the moment.

And usually, when you work on new tasks, like how to evaluate teacher performance, you have to define a lot of criteria. This typically requires a significant amount of effort and ongoing user research. However, many developers do not have the necessary support or education to conduct this user research effectively. So, can we create a critical agent that assists developers in defining the criteria that the application needs to be evaluated against? Essentially, these criteria represent some form of utility for the end user. Additionally, the quantifier agents can assess how well a particular solution is performing according to the defined criteria. Our goal is to establish a multi-dimensional task utility framework.

And just to give you something out of scope, I picked the math example for the reasons that I think we are all probably a bit of domain experts in math here, as we have all studied it in school or at some other levels. So basically, here you see the criteria for the math problem, which was defined by the critic agent. This agent was provided with a prompt asking it to explain the task, along with one successful example and one unsuccessful example of the agent solving a particular problem. I selected four criteria, along with their descriptions and accepted values.

So, this output comes entirely from the kind of agent, and none of it was generated by me. At this point, particularly when I designed a new application for a novel domain, I found it valuable. For example, we conducted an experiment in the robotics space, where I am not an expert. The critic agent assisted us in identifying a comprehensive set of criteria, which was later verified by a domain expert. Essentially, this process saved us a lot of time in determining the crucial criteria important for this domain. Again, using a critic agent makes it straightforward to compile this information. It is vital to adhere to the accepted values as suggested by the critic agents; all these zeros, ones, and twos represent how we clarified our data and utilized it for layer histograms. In general, it's better to stick with the established terminology. I could provide longer explanations on why this is necessary, but just trust me – providing the quantifier agent with a clear description is the best approach. Essentially, this entire process comes at no extra cost.

And by the way, you might wonder how important this is. That's what's really significant. For example, one of the first criteria shown for the mass problems was code quality. I found myself asking, "What do we mean by code quality?" Then I realized that Autogen—one of the baselines we use—produces results that help address these issues. That's why we are achieving very high quality; basically, it's about doing the tasks correctly. The question is, if you genuinely want to solve the problem, you need to focus on writing the code. For instance, if you're trying to create something like a teaching agent, is this something you want to pursue or not?

Again, it's up to you. It's your application, so we're just here to help you evaluate and define what the task utility can be. Another thing you might ask me is about the quantifier agent in this case, which sees unseen examples of your agent and tries to quantify them.

So basically, if you define the new solution, we'll provide a new baseline, and you should follow specific criteria for the task utility you want to achieve. The quantifier agents can help you assess unseen examples and determine if you're improving on some of those criteria. We also desperately need the verifier agent because it can inform you which criteria are effective and which are less so. Specifically, the verifier agent will assist you in filtering criteria that are stable and robust. Additionally, these criteria should perform well with adversarial examples. I will show you an example later. You can download the paper here. By the way, the audience is always part of the other chat; now you can just go run it. That's the main part of the talk that you may want to remember. Imagine you are designing your LMPART application. You would use agents that communicate with each other to solve the task.

Produce a lot of logs for you, so you can basically, based on these logs, figure out what kind of criteria are important for your task, and hopefully for your end users as well. You can use quantifier agents to assess the quality of each unseen solution. Then, by using verifier agents, you can filter out the criteria that we think are not really stable. This is possible because there is still a lot to be done to determine how well the quantifier agent works. You can apply this approach to any type of domain, and you don't have to obtain ground truth for this. If you have it, that's great.

But if you don't, then we'll just want to explore. There’s something that you can set up in a couple of minutes to get the first idea of what your users might use it for. I’ll also give you a bit of an example. Let’s come back to the idea that we’re developing something new; this is a new approach to how you can conduct assessments. We call it "assessment" because the term "evaluation" feels a bit overstretched in my opinion. That’s why we start with a task where we can differentiate between successful and unsuccessful tasks. Here, you can see the same criteria; it’s only four, but usually, there can be up to 25 criteria for the majority of tasks. You’ll also notice three baselines: React, Vanilla Solvers, GPT-4, and Autogen.

And you can see that, for example, in terms of clarity, efficiency, and completeness, successful tasks are actually higher on average for each criterion. That's one of the things we were trying to ensure we were following. Since we are doing something new, we need to have some way to verify if we're doing it right or not. However, you can see that there is variation in how the various types of criteria are assessed by different baselines.

And that's kind of up to you to decide if you want to go with one of them. An important part is to keep those assessments, and if you change your underlying agents, you can reiterate the process. First of all, there's the possibility that your new baseline can introduce new criteria because, at times, your solution can bring some of these criteria into play, and you can see if that's something you want to pursue. However, this error analysis does not look very good for us. That's why we think that the quantifier couldn't do the job of properly assessing the error analysis. That's what the verifier is for; the verifier will indicate that error analysis could be a good criterion to consider, but we just cannot assess it in its current state.

And that's definitely more research needs to be done towards figuring out if we can do something about it. Another important aspect to consider is the distinction between task-based criteria and solution-based criteria. Sometimes, if you simply describe to a large language model (LLM) what you want to evaluate based on a task, it may suggest different criteria. However, providing examples of successful and unsuccessful outcomes can enhance the model's understanding. If you only describe the criteria for solving a problem, aspects like code efficiency might not be considered—at least that was the case in our earlier trials. However, since we sometimes use code to solve complex problems now, efficiency has become a significant factor. It's also worth noting that the more iterations we run, the more criteria our critic may suggest, which could raise some suspicions. At some point, it seems that we can consolidate a limited number of criteria, which is a good sign. In my own assessment, I wouldn't worry too much about the accuracy of these criteria. Despite some concerns about how correct they might be, it is only a worst-case scenario with around 25 criteria to consider.

And those people say, "Are they going to be correct?" It's only, at worst, 25 to 30 criteria per domain. As a domain expert, you can easily verify some of those and remove the ones you are not interested in. I would be more concerned about figuring out how complete this set is. Research needs to be conducted in this area as well.

Sure! Here's the text reformatted with correct capitalization and spelling: "But another intake here, please try to, as soon as you propose a new solution, rerun your critique. You might be surprised because the critique may uncover new criteria introduced by your solution. This leads us to another way to view the situation. We have the advantage of examining both successful and unsuccessful cases. By looking at the distribution of quantifier output for successful cases, we can see the dark blue indicated for failed cases. For example, in error analysis, we can observe that the dark blue represents the failed cases."

And, for example, again, in error analysis, you could see that the scores are almost the same on average. That's not what you want to see. However, we learned from the examples how to evaluate success criteria. By using the verifier, you can filter the criteria that are unstable, even if you don't know the outcome. Another interesting aspect was determining if the poorer examples or solutions were indeed worsened for different criteria. This is not an easy task because we only have either successful or unsuccessful solutions, which is why we don't have a gradation of scores. As you can see here, the average values correspond to the categories I showed you in the beginning. What we did was introduce noise into an existing solution, and the hypothesis was that the samples, which would contain noise, would be affected.

So a disturbed sample, as you can see here, should receive a lower score based on the quantifier in comparison to the same examples without noise. We can observe that this is indeed the case. This means that you can generally trust the idea that the quantifier will rank higher the examples that exhibit better quality according to various criteria. Again, this is important because we are trying to establish something entirely new here.

As a conclusion, we introduced a novel framework called AgentEval. It is designed to quickly evaluate LLM-powered agentic applications. Here you have a QR code; you can visit the blog post as part of the Autogen.

Library and just try Agent Evolve for your LMP application. Also, this is actually based on an academic paper that will be presented tomorrow, I believe, at EMNLP as well. We believe that this can be used for any type of agentic applications where you have logs that you can analyze to see how your agents are behaving. It should also be very cost-efficient, so you don't really need to run a lot of LLM calls to get your initial understanding of various types of criteria.

It can provide an interesting outline of what that potential utility can be. For example, for this mathematical application, one of the criteria was that the solution was verbose. You might want verbose solutions, especially if you are designing something for teachers. The beauty of this approach is that, with this framework in place, you can continue optimizing your agents to deliver more robust solutions. You can even outsource this process to users by asking them what kind of solutions they want and what utility they seek. By establishing these criteria and having a reliable method to assess whether your solution meets them, you can effectively optimize your application to align with user needs. Additionally, as I mentioned, agent evolution helps you uncover your capabilities and see how they evolve over time.

Because if you propose a new solution, there's a possibility that it will introduce new criteria for you as well. So, keep an eye on that. We hope this will help developers and the entire developer community figure out how to assess their solutions and align them with the needs or criteria they want to associate with their solutions. There is definitely a lot of potential in associating the solutions with these criteria.

Directions we can take. I'm just super happy I was here today to share this work because I think that it has the potential to uncover ways for developers to actually determine the strengths and weaknesses of agentic applications based on real data and interactions. It also brings awareness to the need to optimize our applications not only for obvious metrics like latency and success but also for the underlying, multidimensional way that users perceive our applications. We have a means to influence that through this. Thank you so much for your attention.

That pretty much concludes my talk today. Awesome! Thank you so much for coming. Unfortunately, we don't have time for questions. So, at this time, everyone can make their way back to Track One for the closing session.

But I really appreciate everyone coming and viewing, and I especially appreciate your time, Julia. Thank you so much! If you have any questions, please feel free to reach out to me either on LinkedIn or via direct message on Twitter, which is also open.

I'm happy to follow up. Are there any questions regarding this work? Again, it's available as part of the AutoGen library, and there is a more extensive explanation in the paper. Yeah, awesome.

Thank you so much. Take care. Thank you.