AutoGen, AG2, Agents, Frameworks, Open-source, and Best Practices

This is an interview with the founders of Autogen, Dr. Ji Wang and Dr. Xing Yun Wu. Dr. Wang is a senior staff researcher at Google DeepMind, and Dr. Wu is an assistant professor at Penn State University. Both of them are responsible for the creation of the original Autogen and now AG2. We will discuss everything from agents to open source and much more. Let’s get into it.

I would love to talk about everything from agents to open source and so much more. So let's get into it. I would like to hear about the origin story of AutoGen. I know, as you've told me, it's a collaboration between Penn State and Microsoft. But how did that come to be? What were you both working on prior to that?

And then how did you meet each other? How did you start that project? Yes, thank you, Matthew, for asking that. Indeed, when AutoGen was created, I was at Penn State University, and she was at Microsoft Research.

Before that, we were both working on another project, an open-source initiative called Fly. It is a library for automated machine learning and hyperparameter tuning. Chi and I have been working on that project for over five or six years now. We started the project in 2019 when I was interning at Microsoft Research alongside Chi. My internship project focused on hyperparameter tuning, and we invented a very efficient hyperparameter validation algorithm. We decided that we should make it open source to benefit others.

And then we released the Flamo library, and since then, we have been working on this open-source project starting from 2019. We have also been building a community around Flamo. Actually, I'm not trying to advertise Flamo here, but it has been adopted by many enterprise users, especially in the fintech industry. Many banks have tabular data and find this automail library very efficient, leading to a lot of adoption. I think we have millions of downloads of Flamo from the library.

So we have been, I think, millions of downloads of the Flamo from the library. We have been working on this Flamo project and then we see the huge potential of generative models. We have been thinking about how to empower people using large language models and those generative and foundation models. Perhaps you can tell a little more about that and how we reached this idea. Yeah, sure. As Chen mentioned, we're both working on Flamo, using AutoML techniques to help data scientists and machine learning engineers choose the best model for their applications. When GPT-3 came out, it represented a very new kind of model with new capabilities.

So, a natural idea for us was to try the AutoML technique to tune the inference parameters of the GPT models and see how much it mattered. It turned out that it mattered a lot. For several problems, such as math problem-solving, coding, and other text generation tasks, we found that choosing the best inference parameters could significantly improve the model's performance while minimizing inference costs. That's when we realized there was a big opportunity to optimize how to best utilize these models and maximize their utility for different applications. It's not simply a matter of running the model once for inference and getting the job done.

There often involve multiple differences and various other steps—not just in using the models, but also in utilizing tools or obtaining human input, among other factors. This creates a vast design space. Based on our experience with AutoML and natural ideas, can we assist developers in navigating that large design space quickly and in providing the optimal solution? However, before we can do that, we first need to establish a solid, fundamental framework similar to what we have in the machine learning area. In machine learning, we have standardized frameworks like Scikit-learn, PyTorch, Hugging Face, and so on. These frameworks provide a simple interface to easily navigate the extensive design space.

And then, on top of that, we can apply some automated machine learning techniques and open-ended techniques to automate the process of optimization. However, at the time we started building AutoGen, there was no good framework available for language models. So, we decided to build one ourselves first. That was the motivation behind starting AutoGen. When did you first see the potential for agents collaborating with each other?

Because when I first saw autogen, it was one of the first times I had even thought that, "Hey, if you allow one agent to communicate with the other, they can correct each other's work. They can iterate on an output before giving you the final output." Um, so it was somewhat new to me. When was the first time that you had that "aha" moment and saw the potential of agents working together? Yeah. Should I answer, or do you want to go first? Yeah.

I want to briefly talk about my "aha" moment. At that time, we were trying to solve a math problem-solving issue. We chose math problems because, first, they are important, and second, they are very challenging. Another reason is that it's easier to evaluate. Lastly, I like math; I'm also an educator.

I'm a professor, and I care about education. I often think about applications related to tutoring, particularly math tutoring. When we try to solve very challenging math problems, we typically need to utilize various tools and sometimes write code for calculations. This process often involves iterative coding, code generation, and debugging, which can include back-and-forth interactions during code execution. It would be extremely useful to have an executor agent that can run the code for us. Additionally, there's a strong case for involving a human in the loop to enhance the learning experience.

For example, we may want to involve the human tutor to assist the student when the model cannot help. By combining these two concepts, we arrive at the idea of a user proxy agent. The original motivation for this is to mimic what a human would do when interacting with, for example, ChatGPT. At that time, ChatGPT could only generate code but could not execute or build code. Therefore, if you interacted with ChatGPT, the human user would need to copy the code, run it, obtain the results, and then continue the conversation.

So it's basically mimicking what an end user would do. This essentially forms two agent chats: one is the assistant agent, which is a large language model, and the other is a user proxy agent trying to mimic what a human would do. That was one "aha" moment for me. This whole process is fully autonomous.

You can see that these two agents are autonomously communicating back and forth, debugging and chatting with each other to solve the task. It's really fascinating. For me, the starting point is really ChatGPT. I believe ChatGPT represents a significant advancement compared to the previous GPT-3 models, which primarily focused on text generation. However, GPT-4 is when it truly made a significant impact on the general public. I think the reason for this is that the chat interface has allowed natural interactions, enabling humans to provide feedback when something goes wrong, allowing for iterative back-and-forth engagement.

And when I think about that, I immediately relate it to some of my personal experiences in college. When I took a seminar in quantum computing and quantum communication, my professor, Ren Yau Duan, gave me a lesson. He told me that conversation or dialogue is a provable way to make progress in creating new knowledge, validating that knowledge, and generating new insights. This process has a deep theoretical foundation. When I consider why ChatGPT has had such huge success, I believe it is a very good demonstration of that theory applied to a very new technology.

Now we have models that are very capable of engaging in conversations, not just with humans. This has huge implications. For instance, if ChatGPT starts as a communication tool between AI and humans, there’s no reason we shouldn't also allow AI models to converse with each other. Of course, humans can still be involved in the conversation, but it doesn’t necessarily have to be just one human versus one AI model.

And later, we see GPT-4 pushing that capability to a higher level. The models demonstrate many more capabilities, including reasoning—very strong reasoning, planning, debugging, and so on. Each of these capabilities can be configured for different agents to play various roles. The whole process includes generating code, performing testing, conducting reviews, and then reverting back.

There's a whole loop of workflow processes. If we think about enlarging the scope of how we're using the language models, we should try to include more steps or processes in the loop and have the models automate as much as we can. Then, the level of complexity of the tasks we can achieve will be much higher. I was thinking about several concrete applications. Some of them involve data generation; however, we usually need to have one agent to address requests from a user, who must provide input. But if we replace the user with an AI agent, we could possibly automate that data generation process, creating a wide variety of training data corresponding to different personalities or backgrounds, and collect a large amount of useful data.

That's just one simple example application. Before I jump into the application, I want to emphasize that if we generalize this pattern, it has significant implications and offers substantial opportunities to model much more complicated processes. As Chui mentioned, we took math problem-solving as an initial case study to develop solutions for more complex processes. After gaining a deep understanding of this, we made the two agents work together. By the way, the two agents consist of one assistant agent simulating ChatGPT, which is backed by GPT models, and another user proxy agent that simulates human behavior, including generating and executing code suggested by the AI agent. The user proxy offers modes that include automatic execution or manual feedback.

After making the two-agent conversation paradigm work really well, we also took feedback from initial users and found that there was a need to generalize this conversation pattern to accommodate more agents, different roles, and varied communications. This would allow us to quickly evolve from a simple two-agent communication to multiple agents. That's where the magic began. I'm going to ask you about some of the best practices you've discovered over time, but I want to save that for a little later. Once you started working on the original auto-generation, how long were you building until you first published publicly? That's a good question. The thinking started much earlier.

Thinking began in late 2022. At that time, we explored using AutoML to tune the parameters for the models. During this process, we started to consider the best approach for accommodating the large design space that includes models, rules, and humans, as well as the various types of entities involved. We wanted to find the most intuitive way for developers to reason within that space. It took several months to determine the right approach. By the time GPT-4 was released in April 2023, we found a solid starting point to use agents as the abstraction for building the agent framework. From then until August, we focused heavily on developing the agent framework, gathering initial feedback from users, and iterating on our design.

I think in August we wrote the first research paper. Jun, do you have anything to add? Yeah, please! That's about the timeline— it took several months to build the initial version of Autogen and to provide some case studies, application studies, and to include that research in the paper.

Sure! Here’s the text with correct capitalization and spelling: "Like, in total, actually it's not that long. It's just several months—three or four months. Yeah. One thing I want to... Sorry. No, please."

In terms of being public, it's actually been public since the beginning. Since April, when we wrote our first line of code, it has been part of the open-source Flamo project. Every single line is written publicly; it's just that we didn't write any reports, so most people didn't find it.

But the Flammow community has some awareness. They have early access to Autogen, our smaller Flammow community. When the more mainstream audience, like myself, first got a glimpse of this, I mean, I know what my reaction was and kind of how the folks I spoke with felt about it, but what was the feeling for both of you? When you put it out there, I assume people were just stunned by it because it was so interesting and so different from anything I had seen before. What was that feeling like in the first couple of weeks after publishing?

I know you have published before, but I really want to get this out there. To be honest, I never imagined this happening. I'm just working on my idea as well as our idea, and I want to make it useful. I didn't anticipate that it could become so popular, and I was very shocked.

And, of course, in a good way. I’m very motivated to make it much better. That’s my reaction. Yeah, I’m sorry, Matthew. I set a goal.

I set a one-year goal for the Autogen project when I spun it off from Flamo. I established a target—a certain number of stars on GitHub for Autogen within one year. I knew you were going to mention a specific number of stars. Yeah. Let me guess, a thousand stars?

Stars. Yeah. Let me guess. A thousand stars.

Yeah, exactly. You're so right. Based on our Flammel experience, we were growing the community steadily but also incrementally. We communicated with every single user of Flammel since the beginning and went through a lot of iterations each time. It has been a very manageable path of community growth, but everything is totally different now.

So just within a few days after the release, I realized I needed to change my goal. That's so cool! One year ago, after several days, that seemed amazing. And I just looked, and it is currently at over 35,000 stars.

So, no big deal. That's amazing! I want to talk to you both a little bit about open source. Obviously, it's no big deal.

That's amazing! I want to talk to both of you a little bit about open source. Obviously, or maybe not so obvious, but in academia—and especially at Microsoft Research—open source is a significant part of it. However, I don't think people, at least on the enterprise side of things, have adopted open source as much as they have in the AI community. I'm really excited about this, and it's something I've been thinking a lot about lately.

I personally, prior to really diving headfirst into artificial intelligence, was not very knowledgeable about open source and the value that it brings. So, talk to me a little bit about your journey to open source. Have you always been into open source? Was it just not a question like, "Obviously, we're going to be publishing, so we need to publish the code as well"? Was it a decision to make AutoGen open source, or was that just the default?

Sure, here’s a reformatted version of your text with correct capitalization and spelling: "Yeah, I can share my journey in the open-source world. It's not a smooth one from the beginning. It's kind of more like a coincidence, actually, for me to get into that world during my initial multiple years of work at Microsoft Research."

I was mainly just working on techniques, writing research papers, and occasionally talking to product teams to ask them to use it, but I never engaged in any open-source work. The first time we did that was for the Flammo project, and I think I was very lucky to be involved in that call. However, it wasn't open-sourcing at the beginning. Initially, we followed our previous playbook; we conducted some theoretical studies, proved our findings with various theories, and then wrote a research paper about it.

At that time, not many people knew about that Flammo work or that AutoML work, until one day, a user reached out and said they found a paper they wanted to try out. They requested us to share the code, but we didn’t have ready-to-share code at that time; it was still a research prototype. We tried to offer a private license to that initial user, although they preferred open source.

So that's the first try. At that time, the code was more like research prototype quality and wasn't ready to be adopted right away. After some time, another user reached out again after reading the paper and asked whether we had an open-source offering. That’s when I thought, “Okay, we can't wait any longer. We should do the open-source release as soon as possible; otherwise, we risk repeating the same pattern.” Since we didn't have much experience at that time, we weren't sure if it was a good idea or how much it would help. Would this really lead to large-scale adoption? We didn’t have the answers, but at least for this one user and possibly the previous users, they had their requirements and needs, and we didn’t have any better alternatives, so we decided to move forward. That opened the door to a new world. Since then, we have learned so much on this journey. Each time we address user concerns, we improve the library significantly in terms of usability.

And it also drives a lot of the follow-up research work in the multiple years that follow. So the library improved continuously, and adoption increased from small to medium-sized companies, and eventually to large-scale companies, including major products from Microsoft and other companies. By the time we started building AutoGen, I believe that lesson was already very clear to us. The decisions regarding open source and the timing for it were both influenced by our previous experiences from building Flamo. You can see that we took a very different approach this time. When building AutoGen, we wrote the initial code with the intention of making everything open source from the very beginning.

So that ensures the fastest speed to get initial user feedback and enables faster iterations. This approach does not slow down adoption from enterprise companies; rather, I would prefer to accelerate it. We have learned a great deal about enterprise use cases and adoption after it gained awareness in the public sphere. This has been a significant learning experience for me, personally. Chinyu, what do you think? I would also love to share my journey.

There are roughly three stages in this process. I used to be a PhD student, and during my PhD, the default mode was to share code and make my research available and accessible to others for several reasons, such as reproducibility and making an impact. This approach became my default mode, and I didn't really think too much about it. It was probably influenced by my PhD advisor, who had high standards.

We want to ensure that our research is of high standard and reproducible, allowing others to audit our work. This approach represents our initial default period. After that, we will focus on the flammable project, where we have observed significant enterprise adoption and much greater usage from users compared to my PhD research. Additionally, we have achieved millions of downloads.

And at that stage, it's more on a personal level. I see that my research is making an impact. I can see the difference it makes; people are using this in their real-world use cases. It feels very fulfilling personally as a researcher.

And that's kind of a personal impact stage. At the autogen stage, we have much wider adoption and visibility. At that stage, I truly appreciate the passion and interest from the users regarding autogen. This allows me to rise to a higher level and consider what it means for contributors and users to have this open-source library. I think open source creates a very healthy ecosystem for contributors. They can make their research and contributions accessible to others and can also build upon the contributions of others, making their work available for the world to use. On the user level, especially concerning enterprise users, it is important to understand these dynamics.

I think one important aspect of open source software is transparency and trust, because all the code is available for everyone to examine. There is a natural transparency involved. Additionally, there is a spirit of sharing; if something good emerges, we should share it with the world to benefit everyone. That's another advantage of open source software.

And that's another benefit. Another kind of benefit to the enterprise is that open source can grow very large and foster a whole community and ecosystem, making it easy for employees to use. For example, if many people are already using the software infrastructure, onboarding new employees becomes easier because they can start building with this software right away, rather than having to learn a tool that is only specific to their internal team. I believe this creates a very healthy ecosystem. Yes.

And I really appreciate, first of all, both of you for contributing to open source. I also think there's something special about the AI industry; maybe because it is so raw right now, or perhaps because many people who are trying to build things also come from research and academia, open source feels like the default, which is really nice. Obviously, you both have been significant contributors to that momentum, and it just feels really cool. It's not something that I'm used to, and I genuinely enjoy being able to go and look at the code, download it, play with it on my own computer, tinker with it, change things that I don't like, or enhance things that I do. I can potentially submit those enhancements for other people to benefit from, so I really appreciate that.

And on that, I want to talk a little bit, obviously, about agents. Let's discuss some practical matters. For the folks listening, many of the people who watch my channel are builders. They are tinkerers, builders, and individuals at companies who are developing their own projects on the side. They want to use agents. What are some of the best practices you’ve seen regarding autogen and agents in general? What should people consider, whether it’s the types of models to use, the number of agents, or how verbose the prompt or system messages should be?

What are some of those very practical, very tangible best practices that two agent experts would give to the audience? Shall I start, or do you want to start first?

Sure! Here’s the text reformatted with correct capitalization and spelling: "Yeah, I can provide some of my insights, which may not be entirely unique, but I really feel that starting with prototyping and having one single instance of the task work is a very good starting point. It's also a rewarding experience and allows you to experiment with different approaches. I think that's generally a very good foundation, at least based on my experience. The best practices definitely evolve over time. In the beginning, when we had the initial version of AutoGen, I had some guidelines in the technical report we wrote. It suggested beginning with a simple setup of maybe two agents: one assistant agent where you can place all the instructions about the tasks you want the AI to perform, and another agent called the user process agent. Assign the right capabilities to this second agent, like code execution, depending on your needs, and start experimenting with that setup."

So when the task is simple enough, the two-agent setup can work quite well already. However, when the task is complex, you will see that a single model-based agent cannot follow all the instructions effectively, especially in terms of completing certain steps in a high-quality manner. That is a time when you might consider breaking down the task into smaller pieces and having specialized agents focus on the simpler tasks. This way, each step can be completed with higher quality. When doing this, you will also need to think about the topology of communication among these multiple agents. For the two agents, the process is straightforward; they simply take turns.

I say something, you say something, I say something. But for once, you have more than two, and the number of choices becomes very large; in fact, it can be an infinite number of choices. If you're just thinking about the three of us speaking, the order can be infinite. For example, you can have patterns like ABC, ABC, which is one type, or you can do AA, AB, AB, ABC, ABABC, right? Or ABC, ABABC. There is just an endless number of possible enumerations.

There needs to be an intuitive and natural way for multiple agents to work together. We provide a few basic conversation patterns or building blocks, including sequential chat, one-on-one chats, and group chats. Depending on how much control you want in the workflow, you can choose or mix these patterns for different purposes and different sub-parts of your workflow. There are patterns that offer more flexibility and others that provide more controllability. Additionally, there are patterns designed to help isolate the work of one agent from the work of another, such as one-on-one chats. This could be considered the second stage.

Regarding the models or tools used, having multiple agents allows for greater flexibility in choosing different models, configuring them in varied ways, and providing distinct tools for each agent to optimize performance. This setup makes it easier to optimize for multiple metrics simultaneously. While the primary goal is often to maximize quality, there may also be instances where you want to minimize costs or latency. A multi-agent setup facilitates this. Additionally, I have some guidelines related to the evolution of this system, along with some recent suggestions that correspond to the latest features in AG2. These practices are linked to two important features in the new version of AG2: the Captain Agent and the Swarm Agent.

They represent two different practices, both of which are needed in most cases. The captain agent represents a paradigm of a fully automatic way of creating agents and agent teams tailored to a specific task. The users just need to provide a task to the captain agent, and the captain agent will analyze the task, decompose it, and then create different agent teams on the fly for each step in the process. Thus, the entire process can be automated; the users only need to provide the input time, and all the intermediate steps are automated.

Users can also view the agents that are created or selected from the existing library. They can utilize existing agents or tools, or create new ones on demand. This is one paradigm. For relatively simple tasks, this approach can work well. However, when the task is too complicated or we don't have enough tools or agents already provided, it may fail to generate the right tools.

At that time, it often provides you with something useful as part of the solutions. Developers can then review the suggestions and incorporate any useful elements as a starting point, while also adding any missing components. This could include additional tools or specific roles that only domain experts are aware of, which the cabinet agent currently does not know. Consequently, developers can utilize the created or suggested agents and combine them with the swarm pattern. The swarm pattern acts as a high-level programming interface, similar to the OpenAI Swarm framework, but integrated with the rich functionalities within AG2.

So that all the conversation patterns can still be programmed using this one pattern, but more intuitively and in a simpler way. Now, you can take the suggested agents from the captain agent and make them work together using this one pattern, defining the necessary kinds of transitions or instructions on how to hand off to different agents. Additionally, they can share some context variables, allowing them to exchange information in a more structured way, leveraging the new structured output features. There are two different kinds of very useful ways to approach this, and you can combine them in your development process. This is the latest suggestion or guideline I believe can be used as a best practice. Over time, we will likely discover and share more insights. Another general guideline I find useful is to think about how human organizations solve problems because it seems reasonable to treat a language model as a human with, probably, medium intelligence.

We have already developed ways to approach different problems in society. For example, when handling HR, a typical company may have an HR team with various functional, smaller roles. I believe that mimicking what human organizations would do is generally a good starting point. As a quick follow-up, we talked a little bit about best practices. One of the best practices that I frequently get asked about is "human in the loop." When is the right time for a human to be in the loop?

And what have you seen in terms of your own projects or other, more complex enterprise-level projects? What is the right time for a human to be in the loop? Is there a standard definition for a certain complexity or importance of a decision? How do you define when that human should be involved? Yeah, I have some initial thoughts. I can share those, and you can add to them.

I think that depending on technological advancements, the type of environment for humans can vary. However, if we think very far ahead and consider what the end game might be, let's say we have agents that are so capable that they can handle nearly all the tasks we need them to complete. At that point, what do we still need from humans? It seems that the minimal things we would want humans to be involved in would be necessary. Additionally, there are questions about what humans should do when AI is not perfect and has not yet reached the level of intelligence we desire. What are the other tasks we would want humans to take on in that scenario?

So, there are two parts, I think, to this question. For the first part, what are the minimal kinds of things we want humans to be involved in? I think there are several very important aspects. For example, the intent—the original intent from humans regarding what kind of task we want to accomplish. I believe it's usually better for this to originate from humans. Even though we can sometimes imagine a world where AI can actively make suggestions without humans initiating the request, humans still need to make a decision about whether they really want that.

For the initial request, you need to start with a human. After that, it's often not just a single interaction. It's common for there to be multiple exchanges, as we don't typically provide just one initial intent and let AI handle everything else. There is natural ambiguity in how we express what we want. We often opt for brevity, but this can lead to a significant amount of disambiguation needed from AI agents.

And so, likely, we also need to iterate a few times until that intent is very clear for the AI to execute. So, that's the second thing I think humans need to be involved in. The third one is that once the AI returns with certain results or actions, if the result is not exactly what we need it to be, or when there's a need for human verification, the human needs to look at the result and provide feedback if there's anything left to be further addressed. So, that's feedback.

And sometimes, the fourth point I think is that we, as humans, need to override the decisions made by the AI agent and take control when it becomes necessary. These are the four very basic elements that likely require human involvement. However, depending on the quality of the AI techniques or the agent's capabilities, the degree of involvement can vary for each element. For example, if the AI agent consistently delivers high-quality results and satisfies expectations, then perhaps our involvement in the feedback process can be lessened. This is a basic analysis, but I also feel there are several other interesting aspects to consider beyond this. For instance, teaching—before the AI can handle everything we want, we may need it to learn specific tasks first.

They lack the knowledge and the skills, which means some level of teaching is required. We can envision building teachable agents that can learn from humans, who will impart the necessary information or knowledge. Over time, these agents can iterate and improve. Eventually, they can use the lessons we provide to complete follow-up tasks. However, initially, this interaction from humans is essential for the necessary teaching. We should also consider the opposite side of this process, which involves learning.

So sometimes humans, even though agents can perform tasks, still have an interest in learning how those tasks are accomplished. This interest can stem from curiosity, a desire to ensure correctness, or a wish to gain skills themselves. Humans want to improve and therefore seek to learn useful skills from AI agents. I think this represents a natural incentive for humans to keep engaging with these agents. When I consider this problem, I try to connect it to the way I think, essentially mimicking what a human would do.

And as I said, I think about larger models as a kind of human with medium-level intelligence. When we consider human involvement in the process, the first way to engage humans is to identify the gaps between these medium-level intelligence models and actual humans. There are several clearly defined areas where these gaps exist. First, in terms of intelligence, the most intelligent humans on Earth can far surpass the capabilities of these models. Tasks that require a high level of intelligence can be very challenging and necessitate that extra level of cognitive ability, whether for confirmation, verification, or other purposes. This higher level of intelligence is still crucial.

And probably some other levels that are unique to humans, the human race is likely focused on ethics and security. We do have these considerations, especially in high-stakes applications. It is important to consider getting humans in the loop to ensure that the solutions developed based on those AI models are accurate, ethical, and secure. So those are, I think, the places we should definitely consider. Okay.

So I appreciate all of those. I think they make sense. I want to change topics for a moment. Mm-hmm. You've both recently forked the original AutoGen code and started AG2.

I would love to hear about what led to that decision. And Xinguain, why don't you start and tell me why you decided to fork the code? Was there something that wasn't working for you in the environment it was previously in? What did you want to do? And where do you want to go with AG2?

Sure! Here’s the revised text with correct capitalization and spelling: "Yes, thank you, Matthew, for asking that. So, as we have discussed and mentioned, this AutoGen project has been a collaborative effort between Microsoft and many other contributors, including myself at Penn State University. In fact, there are other students from various universities involved as well. At the beginning, we collectively decided that we were okay with releasing the code under the Microsoft organization, as it remains open source and is licensed under the MIT License."

So, we think that's fine. Actually, it works well; it worked fine for a while, over a year. Although we are not from Microsoft—my students and many other collaborators are not from Microsoft—they have been contributing, and we have been working together well. However, as the project grows, complexities arise. First of all, there are challenges in managing the project. Within Microsoft, there may be people who want to steer the project in a particular direction, and there is also the involvement of leadership. I think you understand that, in a large corporation, many factors affect how a project develops. I'm not complaining or saying it's bad; it's just a reality.

And it actually slows down; it's a reality. It slows down the project a lot. A lot of times, our progress goes down, and there are external contributors. If I can interrupt for one second, speaking of speed, I feel like every other day I'm seeing a new post on X about a set of new features for AG2. So whatever has changed between Autogen and AG2, the reins are off.

You guys are moving so fast. It's so impressive. Sorry to interrupt, Xinguin. Please continue. Yeah, yeah, thank you.

Thank you, Matthew, for pointing that out. Yeah, that's one of the primary considerations. We want to move faster and have the time to make decisions and move forward together with the community instead of getting trapped by many other factors. So that's one important factor. Another factor is creating an environment that fosters autonomy in a more neutral place, as well as making it more convenient for others to contribute.

There are many contributors from other big companies who may not want to contribute to a project that is associated with Microsoft. So, that's another reason we want to make it much more convenient for others from all organizations to contribute to the project. Those are the two main reasons. Thank you for mentioning that we are making much faster progress. We are indeed very happy and excited about it.

And our community engagement has worked a lot as well. We now have contributors from Google, Meta, Microsoft, IBM, other small companies, and various universities. The community energy is super fascinating. Okay, so you're building AG2 now. By the way, for everybody watching, I will drop all the links to GitHub and the main website in the description below.

What are your short-term and long-term plans? What are the things coming that people should be excited about? In terms of short-term plans, I can share a few details that won't spoil the surprise. First of all, we want to ensure that this remains a community-driven project.

So we are constantly soliciting feedback and advice from the community and, more importantly, from our users on what we should do next. This is generally our principle right now. We want to use a use case-driven development approach and incorporate whatever is needed based on those use cases. We will try to support it. Currently, as she mentioned, we recently released our captain agent, swarm agent, knowledge graph, and structured output. We have a lot of updates already. In the short term, we are planning to extend AutoGen to EG2 for real-time capabilities to better support real-time use cases, such as real-time voice agents. This is something I feel very excited about. One thing that seems obvious and is likely to come in the future, but hasn't been fully realized yet, is agents that proactively accomplish tasks on your behalf.

Where, you know, the agent—I think agents in general right now do a lot—but you have to tell them, like, "Hey, go do this thing." They don't necessarily have easy access to your personal information or corporate information. You kind of have to set all of this up. When do you think we're going to reach a point where agents are more autonomous and more proactive in accomplishing things on your behalf, 24 hours a day? Yeah, that's a good question.

That's a very exciting question to think about because it opens up and saves so much time for a person. I think in terms of technology, we are already very close to reaching that. One tricky part, I believe, is the kind of human interaction interface, the UI/UX. Although we want the agent to automatically finish tasks, we still need to have the right authorization mechanism. For example, I wouldn't want the agent to send out an email that I don't approve of. While it's nice for the agent to help me send out emails, if it's a very important email, I would like to review it first. So, as long as this human-in-the-loop mechanism is in place and that contract or protocol is well-designed, then we are very close. I think, probably somewhere on Earth, someone has already invented this fully autonomous system. What do you think? Yeah, I think for special domains, as Jing said, I agree that there are no fundamental blockers technology-wise.

Because I've seen tasks that take a lot of time to run, and currently, agents are able to keep running. After they finish the work, they return the results. The only difference from what we described is that the user needs to provide the initial request. However, that step is supposed to be automatable as well. For example, we have another agent that can take on the role of the user in sending that request.

Once in a while, it simulates what the user will say, and then it triggers the remaining members of the Asian team to do the work. If there are no constraints on computer resources or if you don't need to worry about the bills, then that is doable. However, I think there are several reasons we haven't seen that running in reality. One reason may be related to cost concerns.

I think the main issue is that the quality is not guaranteed. When users send requests, they clearly know what task they want to accomplish. However, when the agent performs the task on their behalf, we cannot be sure that it will always meet their expectations. It may work well for a week, but the next day it could behave in an unexpected manner, leading to potentially undesirable consequences. This raises questions about who should take responsibility for those outcomes. This situation is somewhat similar to that of a self-driving car.

I think that, technology-wise, we have very good technology that may last for a while. However, many regulatory requirements and concerns about responsibility, such as how to separate and clarify who takes responsibility, prevent its widespread use. If it's just one individual taking all responsibility for themselves, they could potentially already implement this. But after addressing technical viability and meeting all the requirements, I still question why we really want this. Looking ahead to the next five to ten years, I believe we should be able to achieve a world where agents can complete most tasks—say, from the initial idea to suggesting tools and producing nice results—with minimal human supervision.

But at that time, we still want to make sure that if we want to delegate tasks to agents, we can do so; however, if we want to handle things ourselves or take over, we can do that as well. I think it's very important to ensure that flexibility. I believe that until that guarantee can be established, there will always be the question: Can we really let it go? So, I think once we reach that point—if you think a little further ahead—people may start to consider the question for everything they want to achieve: Should I do it myself, or should I delegate?

I think that would become a non-trivial question for humans to think about, because it's not necessary that I want everything to be done by my agent. I think that's an interesting kind of question to ponder, but that may be sometime later—maybe next year, or five years later, or ten years later. I'm really excited to see what the future of AG2 holds. I've said it a million times; I've kind of become known for saying I'm bullish on AG. I'm waiting for the time when agents can connect to my personal accounts and truly accomplish tasks. Xunwen, you specifically mentioned that you don't want agents to just write an important email and send it, but to draft it and have it ready for me to review and send. That would save me an immense amount of time.

So I'm very excited about that future. I want to thank you both very much for spending time with me today and chatting with me. This has been fascinating. I also want to thank you once again for contributing to open source and really starting the agentic framework wave. There are a lot of exciting things coming.

I'm going to drop all of the links in the description below. But once again, thank you very much for chatting with me today. Thank you very much, Matthew, for having us. Yeah, thank you. Thank you very much.

Thank you. It's a great pleasure talking to you. And yeah, it's a very unique experience. Bye.