Why Agents Are Stupid & What We Can Do About It

I am here with Daniel Jeffries. Daniel, can you hear me okay? I can hear you just fine. So we started out, and you were telling me that you've been a science fiction author. Is that true?

Can you tell me a little bit about that? How long have you been writing science fiction? About 30 years. So it's been a long time. I wrote my first story about the robots taking all the jobs 25 years ago, and I no longer believe it's actually real.

It's a good thing. Now, it's a good thing to write things down because then you disabuse yourself of the notion when you have enough time to think about it. Was that a gradual process where you began to stop fearing it? Would you say that the idea of robots taking over our jobs has dropped in your mind to a 0% chance, or has it gone from 100% to like 50%? Where would you place it?

Nothing has a 0% chance, except for living forever. I would say that, while nothing is truly a 0% chance, it’s pretty close to zero. In fact, when I started looking into historical fears, I found that we've had similar worries about the end of the world countless times throughout history. Since the beginning, even with stone tools, we've been concerned about jobs disappearing.

You know, that was a huge misunderstanding of the Luddite movement and the smashing of looms and all these kinds of things. It just never really pays off. The general retort to that is that this time it's different, but this time is always different. It's never truly different. However, is it the case, perhaps, that right now we’re dealing with, let’s say, autonomous agents—agents that are incredibly intelligent?

Like, does that change the calculus in some way? Well, we're certainly not dealing with agents that are incredibly intelligent yet. Maybe when they arrive, we can think about it. But at that point in time, when people started considering how cars or trains would go above 10 miles an hour, they began to wonder whether their heads would explode or if they would just drop dead. The point is, we're generally pretty bad at seeing around the corner and understanding the actual threats, and we tend to miss them.

A good example is something like ChatGPT. They spent a lot of time worrying about political manipulation. During the early days of its release, they focused a great deal on that, especially with the upcoming election. There was a lot of discussion about misinformation. Meanwhile, one of the biggest problems they faced was spammers using the platform.

Right. And they didn't think about that at all. That's really what ends up happening. We tend to focus on big, flashy threats because that's how humans are wired. We often overlook smaller threats that are actually worse, like the fact that more people die from cancer, smoking, and poor diets. These issues play out over a long period, in contrast to more immediate threats like terrorism or lawnmower accidents.

Right. We tend to focus on terrorism because it's big and flashy. Even though five people tragically die in such situations, the fact is that many more people are going to die due to terrorism. We're just not good at recognizing that for really terrible events.

Well, perhaps agents could help us understand it better or at least better calibrate the actual risks that they themselves impose. Have you seen... Well, let me just lead right into your talk because I think it's probably the best way to start the stage: "Illogical Logic: Why Agents Are Stupid and What We Can Do About It." I'm excited to hear what you have to say about this. I'll be back in 20 minutes.

Folks can be writing their questions in the chat below. In 20 minutes, I'll start to read them out loud for Daniel to react to. So, Daniel, you're sharing your screen. Let me just put this up, and the floor is yours. Thank you, sir. Let's get into it. So, the logic of agents: why do they suck and what can we do about it? If you think that AGI is just around the corner, well then try to build an agent. And I don't mean to talk to your PDF or break this website with a playwright agent. Those are good; they're useful tools, but they do not qualify as complex thinking and reasoning systems. I mean, try to build a real one. And what is a real agent?

An AI system that's capable of doing complex, open-ended tasks in the real world. Even more complicated than that is building one that can perform long-running open-ended tasks. I’m talking about tasks that can be done over many hours, many days, or many weeks, without any unrecoverable errors, similar to a person. The real world is messy. What do I mean by the real world?

It could be the physical world of a robot. You can see Atlas from Boston Dynamics facing the challenges of the real world. These are excellent robots, by the way, but they just can't do anything perfectly. Neither can anything else on the planet, because the real world is extremely complex. Alternatively, it could be the digital world, the online world, but even there, they just can't do anything perfectly.

Neither can anything else on the planet because the real world is super complex. Or it could be the digital world, the online world. When you start to face the never-ending complexity of real life, you begin to realize just how inadequate these systems currently are. Working with agents every day, as my team does, reveals the limitations of today's frontier models. We tend to classify the problems we encounter colloquially into three categories: big brain, little brain, and tool brain.

We tend to see the same problems again and again. That's what we affectionately call it: big brain, little brain, and tool brain.

Big brain has everything to do with higher-level reasoning, strategic long-term planning, abstraction, and expert understanding of common sense. So, when you hear phrases like "it's going to be a PhD-level intelligence," that's just the accumulation of knowledge—knowing how to abstract that knowledge and apply it to domains you have never seen before, and knowing when to use that knowledge. Understanding is completely different, which is why I don't like that analogy at all. As we're going to see later, these are basically the biggest problems that we run into. What’s interesting is that we are building GUI navigation agents—agents that can perform complex tasks by interfacing with a GUI in a multimodal way. Then we saw that Claude, a computer program, came out, and we quickly realized it had the same big brain problems that we had sort of faced. For example, it got bored and started looking at different photos instead of completing the tasks it was assigned. I sent Claude out to try to go to Google Flights, and it got lost on the European cookie pop-up. It couldn't figure out that it needed to scroll down to find the submit button and couldn't navigate to Kayak or Google Flights. So, it went to Kayak, but then the Google login pop-up was covering part of the screen where it needed to input information, and it couldn't figure that out. I was pretty happy to see that an organization with billions more in the bank than we have is still facing some of the same challenges. Here’s an example of a problem that is trivial for you to solve, but hard for a machine.

So I'm thirsty. There's a cascade of steps that, if you decide you want to go out and get some type of drink—whether at a bar or, let's say, at the grocery store, because you don't want to just drink water from the tap—you've got to get dressed, take your keys, take money, close the door, lock up, head to the store, find the water, buy it, and drink it. Now, Moravex Faridex says that reasoning for robots is relatively easy, and sensory-motor perception is hard. That's just wrong. They're both hard, especially when you try to generalize reasoning.

And in particular, I think we've made more progress on sensory-motor perception with the Atlas robots and the robots you're starting to see come out from Tesla, etc., such as self-driving cars, than we have actually made on reasoning. So the secondary problems involve little brain functions, which pertain to the tactical, in-the-moment actions where you have to decide. For example, when you're checking out, there's a subcluster of actions that occur: I'm going to pick the shortest line, and I'm going to wait in that line.

I'm going to step forward and answer questions like, "Do you want to beg or not?" "Are you going to make small talk with the checkout person?" All of these are subclusters of tasks, and you can think of those as tactical kinds of reasoning that are incredibly challenging. You can mess up on long-term reasoning, and you can mess up on tactical reasoning as well. The last bit is tool use.

So that's really the quality and precision of the tooling and the appendages, APIs, and the interfaces that you've provided. We have our hands and fingers, which are some of the most highly evolved appendages on the planet. We'll discuss those in more detail shortly. However, for a model or a robot to execute an action, if the big brain reasoning and little brain reasoning are dialed in, but the tool brain is malfunctioning, then it means that it won't be able to function effectively.

In other words, if my hand is broken, I can't pick up this banana. It doesn't matter if I know how to pick up the banana. I might be able to pick up the bottle in front of me, even if I know the steps, if my hand is broken. So, let's talk a little bit about reasoning.

And when we're talking about reasoning, we're discussing the thinking capabilities of the frontier models. O1, which I love, previewing Mini GPT-4, Claude 3, 5, and Llama: how do they make decisions? The answer is both amazing and, at times, absolutely horrible in an endless loop. It's even worse when it comes to agents. It's not a big deal if they make one mistake. For instance, if you get a prompt, type something in, and it's wrong, you can simply re-prompt it. However, it's not necessarily connected to the last prompt. In an agent, that can be disastrous because the errors can cascade.

It is a compounding problem. A mistake in one area of an agent can be disastrous because the errors can cascade. A mistake in one part of the task can carry through to the steps down the chain. There are all kinds of ways that this starts to show up. A great example is built-in memory; we're not even talking about a database or anything more advanced than that, but rather what's built into its weights. If you've ever coded — and I'm sure you have — with AI, you may notice that it always remembers the old-school OpenAI chat completion (`.create`) versus the newer version of the API (`client.chat.completion.create`) with lowercase.

And it'll occasionally revert to that old version because that's what it remembers. That's a mistake that cascades down. Suddenly, you're hunting for errors in the code because, ten steps in, it decided to revert to this older version of the API that doesn't make sense anymore. Here's a more complex example: we have these GUI navigation agents, and here's a screenshot of it.

I wanted to show a video, but you can find them on the Kintaros YouTube channel. In this case, I gave it a very simple test that it had probably done about ten times: go to Wikipedia and respond to a variation of prompts, such as telling me about a specific king and what he was king of. Sometimes they are all kings of England, but they might also be kings of Ireland and other regions. So, it got to the page with Edward the Confessor. All it had to do to complete the task was read the page, which contains the relevant information in the first paragraph, and tell me who he was king of. Instead, it decided it had to click the language button for reasons unknown. Then, it scrolled around a bit, trying to switch the language to English, even though it was already in English. It spent about two minutes trying to figure this out before finally providing the answer. This is a more complex example of the kinds of issues we see all the time with AI agents. They primarily lack common sense. If you've read "A Brief History of Intelligence," which is one of my absolute favorite books on neurobiology, neurohistory, and artificial intelligence, you would understand why. It is an incredible book that I read cover to cover.

I highlighted almost every page. I just stopped highlighting at some point because I meant to highlight every page. This is a great example. He says, you know, I threw a baseball 100 feet above my head. I reached up to catch it and jumped.

And if you give that prompt to GPT, 99 out of 100 times, maybe they fine-tuned it now because I put it out on Twitter. But you know, 99 out of 100 times, it would get it wrong. It would say something like, "It would jump up into the air, and I caught the ball, and I felt the roar of the crowd." This is because its common sense falls too high to catch. Now, if I change that and say, "Look, this is a fantasy, and you have superpowers and can jump," then you know it. The thing is, GPT just doesn't understand this type of thing, and there are all kinds of ways that these errors cascade into the agents themselves. Another issue is that there's no abstract reasoning; common sense is a built-in understanding of abstract patterns—it's a world model, and there's no world model. We've seen papers recently about fractured world models that they develop.

If you get cut once by a knife, you begin to understand the concept of pain and sharpness. Now, when you see a jagged rock, a spike on a fence, a pike, or a sword, you immediately recognize that they are dangerous. A multimodal model needs to process all these different images of knives. It also has to recognize other sharp objects, like jagged rocks. However, it doesn't easily generalize that shape or the concept of pain and sharpness to other items.

Another example is a fish. This came from a book where a comparative psychologist demonstrated the built-in understanding of three dimensions that fish, people, gorillas, and other animals possess—something that models often fail to replicate. For instance, if you show a picture of a frog to a fish and another animal and train them to touch their noses to it to receive food, and then you replace that frog with a totally different angle that they have never seen before, they will still recognize it as the frog. This occurs without having seen it from a hundred different perspectives, as a multimodal diffusion model would require. Additionally, our built-in understanding is remarkable. Our tools are incredibly refined. Your eyes can detect subtle variations in colors and process up to 36,000 pieces of information.

I could talk endlessly about the beauty of our hands, the most beautiful and exquisite fine-grained object manipulators in the known universe. They can throw a ball, paint a picture, perform precision cutting, and even carry out surgery. They are incredible tools, and we often take them for granted when we are building things. I'll give an example of how the tools break down in an agent.

We had a specialized issue. We found that frontier models were not very good at returning exact coordinates. However, that is starting to change now with the MoMo model, which is the Allen Institute of AI's model, and also our fine-tuned model based on PaLi. Additionally, Claude has started to be trained on this model, right? Generally, we found that these models were good at approximately telling us where things were located. So we decided to layer a grid over an image, and we would ask, "Okay, tell me where the components pose button is." They would respond, "Okay, it's number seven," and then we could zoom in. This usually worked pretty well, except with calendars, it turned into a disaster. As we zoomed in too far, we might ask, "Okay, where's the 29?" or "Where's the 30?" The problem is that these numbers were now obscured by the numbers 14 and 15. Therefore, we had to create a new type of calendar system—a new calendar recognizer. We used OCR (Optical Character Recognition) along with the position of the text. However, what we often found was that the OCR would miss two or three of the numbers for reasons unknown, even with the best OCR engines on the planet that you can pay for.

So it wouldn't have the coordinates for two, seven, and nine randomly on a page. We had to basically figure out mathematically where everything was placed, what numbers were showing, and then mathematically recreate the grid so it would know where to click. That provides a more precise tool, and it's the type of thing you have to do all the time with agents. So how do we fix this? There are three keys to making progress: reinforcement learning, learnable and generalizable algorithms, and scalable data.

We're going to discuss reinforcement systems. If you really look at Strawberry, when it came out, many people were saying, "This is AGI." In reality, it was exactly what I mentioned it would be: a deterministic policy that comes from a 10-year-old paper by DeepMind, where they taught it to play Atari games.

It's a deterministic policy. Once it learns to go left or right, it never goes in the other direction. If this is the ideal way to climb in Qbert, it starts on the left every time. This makes it good at hard reasoning, right? Like math and science, which both have definitive outcomes.

It doesn't make it good at fuzzy reasoning, but it is effective for building larger workflows with reinforcement learning. There are two kinds of reinforcement learning: reinforcement light and reinforcement strong, which I will discuss. The "bitter lesson" is also not very well understood. Generally, people think it means to just throw more computation at a problem, but that's a misreading. While it is correct to some extent, it also highlights what happens when people try to build these expert systems—we are even guilty of it ourselves. I think it's a necessary step to develop generalized agents, but what you end up with is that as soon as you remove all that expert knowledge and allow the machine to learn on its own, you need algorithms that are robust enough to permit general-purpose applications. These algorithms are really hard to build.

Backpropagation is one method, reinforcement learning is another, and then there's the transformer. These are sort of generalized algorithms that are quite challenging, but once you have them, you can achieve remarkable results, like AlphaGo Zero, which operates with no prior knowledge or expert training built in. It surpasses Lee Sedol's version of AlphaGo, which defeated Lee Sedol four games to one, by a staggering hundred to zero after just three days of training. That’s the goal: to develop a generalized system over time, although it can be difficult. We believe there's an eightfold path to better agents, which includes: tool interface models, processing shared memory, task-specific reasoning, hot-swappable skills, generalized task reasoning, and generalized reasoning. We will examine each of these components individually. The good news is that once you're down some of these paths, you start to tackle other pieces of the stack, which helps you understand the tasks you're facing. You can make mistakes, adjust, and move forward. That's applied AI, which is different from pure AI; it’s where you encounter the real world. With improved memory, you also achieve better workflow and tactical data, which you can loop back into the learning engine.

And all these things create a positive feedback loop. They compound. Once you have a rock-solid agentic memory platform that can store any arbitrary task structure, it applies to all the agents in your domain. With better memories, you gain better workflow data, which you can leverage for improved fine-tuning, and so on. This allows for cross-pollination between different domains.

And so we have several versions of our agent that have been released as open source. We've dropped RobbyG2; it was pretty good, but it’s not production-ready. Now we're working on G3, and we're almost done with it.

And in that part of it, we've created a five-part model of the brain: the Strategist, the Tactician, the Critic, the Translator, and the Arm. What's interesting is that the Strategist functions like the big brain, while the Tactician operates like the little brain, figuring out the small steps that need to be taken.

The critic checks in and says, "I don't know about this; maybe we need to go in a different direction. Here's a problem, or I think you're stuck in a loop. Let's break out of it." The translator takes these semantic actions of "I think I need to click the blue submit button" and turns them into coordinates through various tool-brain interfaces, and the arm takes the action. It can also be considered a multi-agent system, but we tend to think of it as parts of the brain. Now we have been working on a tool-brain soda click model. Initially, we started with Pali Gemma. We released the Wave UI dataset on Hugging Face and the original Pali Gemma model. We took it to about 63% accuracy at clicking on things. We found that the Momo model, the 7B version, is about 83% accurate, and the 72B version, which was trained for robotics on 2 million pairs, is about 90% accurate. They never really trained it on GUIs, but it seems to be really good at GUIs. We're currently in an open fine-tuning phase, and we plan to release this as a model that people can utilize, including on our interface, etc. But now it powers the tool-brain of our next-generation model that we're releasing soon. This is turning into a really important point. Most memories have focused on RAG systems, and developers often make the mistake of thinking that if you just pump a bunch of data into it, it will do everything. It will not. Storing data is only the most basic level of memory, and it's not even the most important aspect. The retrieval system is what makes a good memory store. When you are talking to somebody, your brain is running a bunch of routines about whether you know this person.

Is there an analogy for what they're saying? Do I understand it? It's hunting through your memory for related associations and memories, right? That is a retrieval algorithm, and that is where the real rubber meets the road when it comes to agents, and where we’re going in terms of what they learn and how they can pull information from memory. We built a system which is a sort of command-line interface, where we can create synthetic memories. We can annotate these memories, indicating whether they were a success or a failure. We can also human-annotate them, saying things like, "Hey, stop clicking the link button; you don't need to. Just go ahead and read the screen that’s there." That’s a failure, and then the system learns from these synthetic memories or from the annotated memories that it has already processed through those sequences. Additionally, the AI annotates its experiences as well, and we have a number of packages that help us address these aspects too. This kind of leads us to G4, which will involve shared memory. When one agent learns something, all of them in the swarm start to learn it as well. There’s also task-specific reasoning involved. And so, that’s where we begin.

We're building an agent learning system where you can create and demonstrate things to the agent's store of synthetic memories. We're giving it improved precision tooling to continuously enhance its reasoning abilities. In this case, you can start fine-tuning it with workflow data, which makes the big brain better and smarter. Thus, we have this fine-tuning pipeline, not just limited to the memories in the system. This also leads us to G5, where we begin discussing hot-swappable skills. This concept ties into reinforcement learning (RL) strong, where you can specify, "I want to become proficient in Amazon, so I'll load the corresponding skill." For example, if I want to navigate a particular database or Salesforce efficiently, I can load the appropriate skill for that as well. Reinforcement learning strong unleashes the agent to explore the world across various cloned websites and learn from those interactions. However, we want to ensure that it doesn't learn randomly. Many RL agents, as discussed in the paper "LLM for Teach," use the language model to set the reward policy and guide the agent's initial actions. This allows the agent to learn quickly and subsequently move beyond the flawed role model of the language model as it begins to explore and exploit. Therefore, you end up with a reward-driven agent. In conclusion, once you have these hot-swappable skills, we're also developing a real-time inference engine.

So you can swap based on the entropy; you can swap the skill at the time as it's generating the token. For example, you might realize, “I need to do a lookup,” or “I need to load this skill pack to improve this part before generating any more tokens.” This allows the agent to make better decisions, similar to how someone might suddenly open Google. That's actually it; we're almost there, and I'm running out of time. None of this is easy. Right now, in the world of AI agents, it often feels like we wanted to build a website but then discovered we had to invent PHP, MySQL, Linux, and Apache. Then, we have to build WordPress. After that, we could create a nice drag-and-drop editor or something similar. As the old spiritual joke goes, if you want to make a pizza, first you have to create the universe.

But you know, each day we're making progress, and slowly but surely, we're making our way to Rosie the Robot, and the world will never be the same. Daniel, I think I can say from reading the chat and from listening to the talk that this was an incredible, incredible talk. Thank you very much. I appreciate it. I also got myself a copy of "A Brief History of Intelligence" already.

Awesome book! Everyone should read it. It's just fantastic. It's a great movie. Well, no, thanks for the recommendation—I hadn't heard of it before, but once you mentioned it with such enthusiasm, I went and grabbed a copy while you were speaking.

So I have a few questions myself, but I see that people in the audience do too. Let me start by reading off a few of them. We have just a few minutes for this, so let's see how far I can go. Vibe is asking, "LLM agents are hyped, but can small teams with limited resources compete with big players?"

And if not, are there any strategies that can help adapt and succeed in a rapidly evolving field? Yes, you need to be a creative team. You should be an applied AI team, which means you have to get your hands dirty and try new things every day, including a lot of pursuits that are likely to fail. It's a scientific process that involves experimentation and learning from failures frequently.

And then you can absolutely fine-tune the larger models. We invented a number of algorithms, for instance, to make the agents better at clicking on things. When you are forced to adapt with limited resources, you often come up with brand new ideas that people who have $10 billion in the bank never considered. That's why they found a different cure for lung cancer in Cuba based on just antibiotics; they didn't have access to the larger Western medicine system. They were forced to adapt to a serious problem, but necessity is the mother of invention. They were able to compete in that way. You spoke a little bit about cascading errors. Here we have Tony asking, "Given cascading errors, does using a mixture of agent types approach to the big brain help?"

So I know that even when you break down the brain into multiple kinds of models, you made a point to say these are not different agents. At least, we don't consider them that way; we consider them different parts of a single brain. To think about the relationship between a mixture of agents and cascading errors, it's essentially the same thing, right? You're just dividing up the task into multiple components. So you can think of it as parts of the brain or multiple agents.

They could be different models on the backend. We end up having a swarm of models. We have two OCR models. We have the Molmo 7B model for clicking, and we have the large Frontier model, whether that's Quad, GPT-40, or a Volvo 72B running as the larger agent. You could split that even further if you want; it helps to a certain degree, but it is not a magical fix.

Everything that you see on Twitter as a magical fix is not a magical fix. As soon as you hear someone refer to it as a magical fix and the first reply is, "We're all cooked, AGI tomorrow," you can safely discount that by about 75%. It's very useful, but it doesn't solve all the problems. Sometimes, the agents can get into a method where they're just talking to each other and don't come up with a better solution. Just like when we tried letting the agent have more epochs to think about things; sometimes it would actually think itself out of the answer.

So, like five steps in, I'm thinking, "It's got it, right? It's got it in the loop." It thinks for another ten steps and gets the answer wrong. So, you know, it’s like a group of people talking themselves out of a solution. It sometimes helps, but it’s not perfect.

Tony is asking, with respect to these agentic reasoning challenges, do graph-based neurosymbolic AI approaches help at all? Potentially, I mean, I think so, but you want to consider how I'm qualitatively retrieving a memory that is useful to me. For example, we're experimenting with a scenario where I need to determine if I've clicked on a specific spot already. If I have, then that's great; just return that information so I don't have to think about it—I just need to click the submit button. Is it the same? That's a specific retrieval process. There's a sub-layer of that regarding which mechanism I'm using: this one, this one, or that one. But ultimately, it's really about what is the correct information that you're trying to retrieve for the task at hand. That's the real hard part. It does help to have some level of neurosymbolic AI in this context.

I would say it's not perfect. So, we're taking two approaches. One is inserting memories into the brain—synthetic memories or memories that are annotated. For instance, Robby G3 will gather the memories, both successes and failures; it will collect five of each.

And I found that sometimes it will take input from both successes and failures. It will grab five from each. I also found that it can sometimes reduce the steps from 25 down to eight, but occasionally it presents a different problem, right? Therefore, actually fine-tuning on that data will be more valuable for doing that. It might not be a retrieval problem.

It's helpful. All these different types of graphs and various approaches are very useful. Ultimately, it's about the knowledge you build that improves the information relevant to the test you are conducting. We have a bunch more questions, and I'm not sure if I should go through them quickly because I think we need to leave in a couple of minutes.

But let me try to take a stab at a more abstract reading of some of the questions that are here. On one hand, I hear you saying that this is ultimately a scientific approach. You have to engage in dialogue with the reality of the situation as much as possible. You should just ship and continue to improve over time, as there isn't a silver bullet.

You're just going to have to continue to try lots of different tactics to improve over time. There isn't a silver bullet. You will need to explore various approaches to enhance the agents continuously. On the other hand, we have the more valuable lesson: let us invest in generalizable functions that are somewhat broad and algorithms that are general-purpose, which can help us learn effectively. Is there a trade-off between those two?

And if so, how do you feel about where to land in making that trade-off? If you're a small team, you're going to have a lot of trouble competing with the frontier labs for generalizable intelligence. What you need to do is plan for the fact that you're going to get upgraded intelligence at various points, and you will have to probe that black box to determine whether it's useful or not. You can think of it as a graphics card upgrade or a Linux kernel upgrade that you get for free. If you're going to be outmoded by that, you don't want to build the rest of your system around it. The rest of it is sort of like a middleware for those agents, and that’s where the friction happens—integrating it into your stack. It's the same kind of friction you see in any sort of development. So, you know, the challenge is that you want to build expert system stuff, but you need to be willing to let it go right as soon as there's a generalized version. That said, there are some generalized things that I think the open source community can really start to excel at. If you look at the work from groups like Entropics and some others that are working on entropy to try to predict the next token, I think that's the kind of thing where smaller labs can experiment with a million different people trying it in different ways, allowing them to start to outpace the frontier models. This is contingent on them having access to good open source models, and we hope the AI doomsayers don't end up restricting all models while allowing unfettered war machines and surveillance spots at the government level.

Let's cross our fingers that doesn't happen. Daniel, thank you very much. If you can, please linger in the chat and perhaps respond to some of the folks who had asked questions; that would be wonderful. Sounds good. Thank you very much for joining us.