On Engineering AI Systems that Endure The Bitter Lesson - Omar Khattab, DSPy & Databricks

So, thanks everyone for showing up, and thanks to the organizers for inviting me and having me here. I'm excited to talk to you all about engineering AI systems that endure the bitter lesson. I'm Omar. I guess the introduction has already happened, so let's not repeat that. If you're here, I think it's probably because you engineer what we might call AI software, or you might manage people or work with people who do. It's not really a term that has been used in this way for that long.

So we're all trying to figure out what the right basics and fundamentals are, as well as what things are fleeting. This will largely be the focus of the talk. The name of the game has become somewhat of a meme at this point. It seems like there's a new large language model every week—perhaps even more frequently than that—which changes the trade-offs you can make.

It might not be state-of-the-art in terms of the best quality necessarily, although sometimes it is. But maybe it's the best performance for certain costs. Or it's the best performance for certain types of applications. Or maybe it's the speed that's really incredible. We've seen things like diffusion now. Every week, there's a new LLM that you have to consider if you're engineering software in this space, which is really unusual.

Think back to normal software engineering; you change your hardware every two or three years, maybe, if that. So this situation is pretty unusual. Another part that’s actually a little bit weirder is that, if you're lucky, the LLM provider has recognized that they're not really building these LLMs. They're training them. They're emerging based on a lot of nudging, data, and iterations, along with a lot of evaluations and, of course, vibes as well.

And they have realized, you know, if you're lucky, that there are new features in their latest models that weren't there before. To the surprise of many people these days, you still receive longer and longer prompting guides for the latest models that are supposed to be closer to AGI. If you're less fortunate, you have to figure it out on your own, right? If you're even less lucky, the prompting guides from the provider aren't even that good, so you have to determine what the right approach is. Every day, perhaps at an even faster pace, someone is releasing an archive paper, a tweet, or something else that introduces a new learning algorithm—maybe some reinforcement learning bells and whistles, prompting tricks, or a prompt optimization technique. There's always something that promises to make your system learn better and align more closely with your goals.

Someone else is introducing some search, scaling, inference strategies, agent frameworks, or agent architectures that promise to finally unlock levels of reliability or quality better than what you had before. I think that if you're actually doing a reasonable job now, most likely you're scrambling every week. This isn't just because you're doing a bad job; it's because you're doing a good job, right? You're thinking, "I need to stay on top of at least some of this stuff so that I don’t fall behind."

And in many cases, model APIs actually change the model under the hood, even though you're using the same name. So, you're forced to scramble. I would say maybe the question isn't whether you will scramble every week, but rather, will you even get to scramble for long if you think about the rate of progress of these large language models? Are they going to eat your lunch?

So, these are, I think, questions that are on a lot of people's minds, and this is what the talk is going to be addressing. The talk mentions the "Bitter Lesson," which sounds like a piece of ancient AI lore, but it's actually just six years old. The current year's Turing Award winner, Rich Sutton, who is a pioneer of reinforcement learning, wrote a short essay on his website. In this essay, he explains that years of AI research have taught him, and others in the AI community, that when AI researchers leverage domain knowledge to solve problems—like chess, for example—we often build complicated methods that essentially don't scale, leading to stagnation. We then get outperformed by methods that leverage scaling much more effectively. According to Sutton, what seems to work better are general methods that scale. He identifies search—not in the sense of retrieval, but more about exploring large spaces and learning. This helps the system understand its environment. In this context, search can be compared to what we might call inference time scaling in the realm of large language models (LLMs). I don't claim to speak for Sutton, nor am I suggesting that I have the definitive understanding of what he means or that I necessarily agree or disagree with him. However, I believe this is a fundamental and important concept in this field.

So I think it raises interesting questions for us as people who build and engineer AI systems. Because if leveraging domain knowledge is bad, what exactly is AI engineering supposed to be about? I mean, engineering is understanding your domain and working in it with a lot of human ingenuity in repeatable ways, let's say, or with principles. So are we just doomed? Are we just wasting our time?

Why are we at an AI engineering fair? I will tell you how to resolve this. I have not really seen a lot of people discuss it. Sutton is talking about the bitter lesson, and clearly, someone has to think about this, right? Sutton is discussing the maximization of intelligence.

All of us probably care about that to some degree, but it's like the ability to figure things out in a new environment really quickly. I'm also an AI researcher, but when we're building AI systems, I think it's important to remember that the reason we build software is not that we lack AGI. We build software for a reason, and the way to understand this is that we already have general intelligence everywhere.

We have eight billion of them. They're unreliable because that's what intelligence is. And they've not solved the problems that we want to solve with software. So we program software not because we lack AGI, but because we want reliable, robust, controllable, and scalable systems. We want these systems to be something we can reason about and understand at scale.

And actually, if you think about engineering and reliable systems, and if you consider checks and balances, in any case where you try to systematize things, it's about carefully subtracting agency and intelligence in exactly the right places without restricting intelligence otherwise. This is a very different axis from the kinds of lessons that you would draw from the Bitter Lesson. Now, that does not mean the Bitter Lesson is irrelevant. Let me tell you the precise way in which it is relevant. The first takeaway here is that scaling search and learning works best for intelligence. This is the right approach if you are an AI researcher interested in building agents that learn effectively and efficiently in new environments.

Don't hard code anything unless you really have to. When building AI systems, it's helpful to think about search and learning. Searching for what, right? What is your AI system even supposed to be doing? What is the fundamental problem you're solving? It's not intelligence.

It's something else. And what are you learning for, right? Like, what is the system learning in order to do well? That is what you need to be engineering, not the specifics of search and not the specifics of learning, as I'll talk about in the rest of this talk. So, he's saying - Sutton is saying - complicated methods get in the way of scaling, especially if you do it early, like before you know what you're doing, essentially. Did we hear that before?

I feel like I heard that back in the 1970s, although I wasn't around. This is the notion of structured programming, with Knuth saying his popular phrase in a paper: "Premature optimization is the root of all evil." I think this is the bitter lesson for software and, thereby, also for AI software. It's human ingenuity and human knowledge of the domain; it's not that it's harmful.

It's that when you do it prematurely in ways that constrain yourself, in ways that reflect poor understanding, it becomes bad. However, you can't get away in an engineering field with not engineering your system. You're just quitting or something. Here's a little piece of code. If you follow me on X (formerly Twitter), you might recognize it, but otherwise I think it looks pretty opaque to me. In three seconds, I can't really look at this and tell exactly what it's doing, and I also honestly don't really care.

Lo and behold, this is computing a square root in a certain floating-point representation on an old machine. Immediately, this is not the most future-proof program possible. If you change the machine architecture, different floating-point representations, or better CPUs, first of all, it will be wrong because it's just hard-coding some values here. Secondly, it will probably be slower than a normal square root that might be a single instruction, or perhaps the compiler has a really smart way of handling it, or there are many other things that could be optimized for you. So, someone who wrote this—maybe they had a good reason, maybe they didn't—but certainly, if you're writing this kind of thing, you're probably making a mistake as an engineer. Premature optimization is possibly the root of all evil or something, but what counts as premature?

I mean, that's kind of the name of the game, right? We could just say that, but it doesn't mean anything. So I don't think any strategy will work in tech. Nobody can anticipate what will happen in three years, five years, or ten years. However, I think you still have to have a conceptual model that you're working off of. I happen to have built two things that are several years old and have fundamentally stayed the same over the years, from the days of BERT and Text-DaVinci 2 up to 04 Mini. They're bigger now than they ever were and serve as stable, fundamental abstractions or AI systems around LLMs.

So what gives? What happens in order to get something like Colbert or something like DSpy in this ecosystem and endure for a few years, which is like, you know, centuries in AI land? I'll try to reflect on this, and again, none of this is guaranteed to last forever. So here's my hypothesis: Premature optimization is what happens if and only if you're hard coding things at a lower level of abstraction than you can justify.

If you want a square root, please just say, "Give me a square root." Don't start doing random bit shifts and bit manipulations that happen to appease your particular machine today. Instead, take a step back. Do you even want a square root, or are you computing something even more general? Is there a way you could express that more general concept?

And only stoop down or go down to the level of abstraction that's lower if you've demonstrated that a higher level of abstraction is not good enough. So I think the bigger picture here is that applied machine learning and prompt engineering have a huge issue. Tighter coupling than necessary is known to be bad in software, but it's not really something we discuss when building machine learning systems. In fact, the name of the game in machine learning is usually about the latest advancements: "Hey, this latest thing came out; let's write everything to work around that specific innovation." I tweeted about this a year ago, 13 months ago, in May 2024, stating that the bitter lesson is just an artifact of lacking good high-level ML abstractions.

Deep learning and scaling deep learning help predictably, but after every paradigm shift, the best systems always include modular specializations because we're trying to build software. We need those. Every time, they basically look the same and should have been reusable, but they're not because we're writing code poorly. We're writing bad code. So here's another nice example to demonstrate this.

It's not special at all. Here's a 2006 paper. The title could have really been a paper now, right? "A Modular Approach for Multilingual Question Answering." And here's the system architecture.

It looks like your favorite multi-agent framework today, right? It has an execution manager and includes some question analyzers and retrieval strategists from a variety of corpora. It's like a figure; you know, if you colored it, you might think it’s a paper from last year or something. Now, here’s the problem.

It's a pretty figure. The system architecturally is actually not that wrong. It's the perfect architecture. In a normal software environment, you could just upgrade the machine, right? Put it on new hardware, put it on a new operating system, and it would just work. It would actually work reasonably well because the architecture is not that bad.

But we know that that's not the case for these machine learning architectures because they're not expressed in the right way. I think, fundamentally, I can express this most passionately against prompts. A prompt is a horrible abstraction for programming, and this needs to be fixed as soon as possible. However, it's actually not a terrible one for management. If you want to manage an employee or an agent, a prompt is reasonably similar to a Slack channel. You have a remote employee.

If you want to be a pet trainer, working with tensors and objectives is a great way to iterate. That's how we build the models. However, I want us to be able to engineer AI systems as well. I believe that for engineering and programming, a prompt is a poor abstraction. Here's why.

It's a strongly typed canvas, just a big blurb, with no structure whatsoever, even if structure actually exists in a latent way. That couples and entangles the fundamental task definition you want to convey, which is truly important. This is what you're engineering with some random, overfitted, half-baked decisions about how this LLM (large language model) responded to a particular language when you communicated in a certain way. Or, you provided an example to demonstrate your point, and it seemed to resonate with this model, so you decided to keep it in. And there's no way to really tell the difference between what the fundamental issue you're solving is and what random tricks you applied.

It's like a square root thing, except you don't call it a square root, and we just have to stare at it and think, "Wait, why are we shifting to the left by five bits or something?" You're also using the inference time strategy, which is like changing every few weeks, or people are proposing ideas all the time, and you're baking it, literally entangling it into your system. So if it's an agent, your prompt is telling it that it's an agent. Your system has no problem knowing whether it’s an agent, a reasoning system, or whatever. What are you actually trying to solve?

It's like when you're writing a square root function, and then you think, "Hey, here is the layout of the structures in memory or something." You’re also discussing formatting and parsing things—writing in XML, producing JSON, whatever. Again, that's usually none of your business most of the time. You want to write a human-readable specification, but you find yourself saying things like, "Do not ignore this."

Sure! Here’s the reformatted text: "Generate XML, answer in JSON. You are Professor Einstein, a wise expert in the field. I'll tip you $1,000. That is just not engineering, guys. So what should we do? Concerns, I think, is the answer."

Your job as an engineer is to invest in your actual system design, starting with the specifications. Unfortunately or fortunately, the specifications cannot be reduced to just one thing. At this point, I want to discuss evaluations. I know everyone hears about evaluations, so here's one key point about them that prompts further discussion. Much of the time, you should invest in natural language descriptions because that is the strength of this new framework.

Natural language definitions are not prompts. They are highly localized pieces of ambiguous information that could not have been expressed any other way, right? I can't communicate certain things to the system except in English, so I'll say it in English. However, much of the time, I'm actually iterating to appease a specific model and to make it perform well relative to some criteria I have—not disclosing the criteria, just tinkering with various elements. In this context, evaluations are the way to do this because evaluations indicate what I truly care about.

Change the model. The evaluations are still what I care about. It's a fundamental aspect. However, evaluations are not everything. If you try to use evaluations to define the core behavior of your system, you will not learn.

Induction, learning from data, is a lot harder than following instructions, right? So, you need to have both. Code is another thing that you need. You know, a lot of people are like, "Oh, it's just, you know, you just ask it to do the thing." Well, who's going to define the tools?

Who's going to define the structure? How do you handle information flow? For instance, things that are private should not be shared in the wrong places, right? You need to control these aspects. How do you apply function composition?

LLMs are horrible at composition because neural networks essentially don't learn things that rely on function composition. In software, function composition is always perfectly reliable, basically by construction. Therefore, many of these tasks are often best delegated to code. However, it's challenging, and it's really important that you can effectively juggle and combine these concepts. You need a canvas that allows you to integrate these elements well. A good canvas should enable you to express these components in a highly streamlined manner, ensuring that they are decoupled and not entangled with changing models.

I should just be able to hot-swap models. Inference strategies are changing. Hey, I want to switch from a chain of thought to an agent. I want to switch from an agent to a Monte Carlo tree search. Whatever the latest thing that has come out is.

I should be able to just do that, and new learning algorithms are really important. We talked about learning, but learning is always happening at the level of your entire system if you're engineering it.

Or at least you've got to be thinking about it that way. You're saying, "I want the whole thing to work cohesively for my problem". That's the general default, right? So, that's what the evaluations here are going to be doing. You want a way of expressing this that allows you to do reinforcement learning, prompt optimization, and any of these things at the level of abstraction that you're actually working with. The second takeaway is that you should invest in defining things specific to your AI system and decouple from the lower-level swappable pieces, because they'll expire faster than ever.

So I'll just conclude by telling you that we've built, and have been building for three years, this DSPI framework, which is the only framework that actually decouples your job—writing the lower-level AI software—from our job, which is providing you with powerful, evolving toolkits for learning, for search, which is scaling, and for swapping LLMs through adapters. There is only one concept you have to learn; it is a new concept, which we call "signatures," a new first-class concept. If you learn it, you've learned DSPI. I will unfortunately have to skip this due to time constraints for the other speakers, but let me give you a summary. I can't predict the future.

I'm not telling you that if you do this, the code you write tomorrow will be there forever. But I am telling you that the least you can do—this is not like the top level, it's just the baseline, I would say—is to avoid hand engineering at lower levels than today allows you to do, right? That's the big lesson from the bitter lesson and from premature optimization being the root of all evil. Among your safest bets, which could turn out to be wrong, I don't know, is that models are not going to read specs from your mind anytime soon. I don't know if we'll figure that out. And they're not going to magically collect all the structures and tools specific to your application.

So, that's clearly stuff you should invest in, right, when you're building a system. Invest in the signatures, which, again, you can learn about on the DSPY site, dspy.ai. Invest in essential control flow and tools. And invest in evaluations for things that you would otherwise be iterating on by hand. Finally, write the wave of swappable models.

Write the wave of the modules we build. You just swap them in and out. Additionally, write the wave of optimizers, which can perform tasks such as reinforcement learning or prompt optimization for any application that you have developed. All right. Thank you, everyone.