Stanford Webinar - Large Language Models Get the Hype, but Compound Systems Are the Future of AI

It's wonderful to be here with you all. Large language models get the hype, and this is certainly true. I firmly believe that the future lies in compound systems. However, I want to emphasize that I also feel that the present consists of compound systems.

I think we are all kind of intuitively aware, if we're working in the field, that we only ever interact with systems, but models get all the headlines. That core insight can get clouded out. Part of what I want to do today is raise your awareness of the fact that we are constantly dealing with systems. If that is getting lost in the shuffle, it is understandable because we are inundated with headlines that focus on large language models and other foundation models. This trend really kicked off with the GPT-3 paper, of course, and that makes perfect sense because the headline result from that paper is that they did a significant thing in the world.

They trained a model with 175 billion parameters, which is more than an order of magnitude larger than any model that had come before it. The central insight is that the simple act of scaling allowed them to design systems of a kind that we had never seen before. This really kicked off the trend of announcing lots of exciting developments. Here's another example of this: when Google announced its PaLM model, a key point in the announcement was that it consisted of 540 billion parameters, significantly larger than any model we had seen before. I think PaLM is a system in the sense that I mean for today's talk, but they made the announcement in terms of a model.

And they even did that for their Gemini system. I think, again, Gemini is a paradigm case of innovation at the level of model artifacts, but also at the systems that we build around them so that they do exciting things. Gemini is a system, but of course, they described it as a model in keeping with this trend. Even OpenAI, which I think of as an organization that really has reoriented around full software systems—where the paradigm example of that would be ChatGPT, which is a kind of experience powered by lots of interesting models—tends to announce things in terms of models.

So, GPT-4.0, the new flagship model. I think these are really systems; they centered it on the model. And of course, that's just the big tech players. If you're like me, it feels like every time you open up Twitter, your feed is just full of breathless announcements about some exciting new model series that is supposed to change everything. So, if you're living in this kind of world with me, it makes sense that you would end up thinking in terms of models.

But this is the central lesson here: we only ever deal with systems. Let me try to make this vivid for you. On the slide, a large language model is displayed. You can imagine that you downloaded it from somewhere as a pre-trained artifact, or maybe you spent millions of your own money to train this model yourself. In some sense, you feel like this is going to be a really exciting and highly performant artifact.

But currently, it is just sitting on disk, completely inert. All it can do is occupy disk space at this point. To get this model to do anything at all, you have to do at least these two things: you have to prompt it to enter some kind of internal state, and then, if you want it to be a generative system, you have to define some kind of sampling method.

You have to get this model to talk because, intrinsically, all it really does is represent things in an abstract space. Having made a choice about a prompt and a sampling method, you now have a system. Part of the main thesis for today is that both of these choices are non-trivial and will define the kind of system that you end up with. The prompt, the model, and the sampling method form a kind of minimal system, but it is already an important one. And then, of course, in the more modern context, we are taking minimal systems like that and giving them access to calculators, programming environments, databases, and maybe even web APIs and the web itself.

I think at that point, we can all see that this is a software system. The language model might have a privileged place here, as a kind of hub for all the things that are going to happen, but it is just one component. The capabilities of this system will be defined by how all of these elements work together in concert, and not solely by the language model. That is kind of the essence of my thesis for today. In my group and in related groups at Berkeley and various other places, we have been thinking in these terms for quite a while now.

At the start of this year, we published a blog post titled "The Shift from Models to Compound AI Systems." Today, I want to provide you with an updated perspective on the thesis presented in that blog post. I would like to emphasize that this blog post was published on February 18, 2024. The reason I want to highlight this date is that, in AI terms, it feels like a long time ago. I’m increasingly hearing discussions along these lines. In particular, Sam Altman, at a recent OpenAI developer event, reflected on the future of AI and mentioned that he expects a shift from discussing models to discussing systems. This resonates with the title of our blog post.

I don't know whether he was influenced by us or whether this is a case of convergent thinking or what, but we did say this first. I guess the main point I want to make is, well, goodbye, Sam. We've been thinking in these terms for a very long time. Why is this important? I’m going to try to support these main claims here, but let me just give them to you at a high level. The first is that this is important just from the perspective of investing in all of the right things as you're developing an AI solution.

Here's an analogy: when you design a system in this mode, you are doing something analogous to designing a Formula One race car. I think we can all recognize that an F1 race car is much more than just its engine. It's also about aerodynamics, friction, the driver, the controls for the driver, and everything else. All of those elements have to come together to create a truly exceptional race car.

And too often in AI, we are behaving like Formula 1 race car designers who focus obsessively on the engine. I don't know much about F1 racing, but I'm confident that if all you ever did was think about the engine for this car, you would not end up with a good race car. By the way, you can probably tell that I generated these images using ChatGPT. It was sort of funny; I had trouble prompting it to simply show me a picture of an F1 engine. It kept adding wheels to the engine, resulting in pictures like this.

And I decided that this is almost a comical embodiment of the thing that I'm worried about today. We're trying to design F1 race cars, and we end up doing that by putting wheels on the engine. Do not do this; it's not going to lead to a good overall system.

Got to these terms. Here's another important point: building the best system for your goals and constraints is almost certainly going to emphasize system components. I maintain that, for example, a small model that is embedded in a smart system is always going to be better than a large model embedded in a simplistic system. I believe this is true even if you're just aiming for raw accuracy or performance in some sense, but it is absolutely true if you also care about things like cost, latency, safety, and privacy.

Once those considerations come into play, you will want a system that can provide you with all of those guarantees. In that context, a small model might be the only choice available to you, especially if cost is a factor. Finally, we can broaden the perspective a bit and consider safety and regulation. I will address this at the end of today's talk. Much of the discussion in these areas has focused on model artifacts.

And I think that is simply a mistake. I think we need to think about regulating entire systems, not just models. I believe that if we focus on the models, we're inevitably going to let some really dangerous things through and also end up over-regulating things that could actually be very productive. However, if we think in terms of systems, I believe we have a better chance of getting these things right. So, let's dive in a bit now to the system components.

And I'm going to stick with the simplest ones for today because I actually think those are the most consequential, especially in this era of in-context learning. But let's start with the one that seems, on the surface, to be the simplest of all. This is the method that you're going to use for sampling when you want your model to generate. Here is that minimal system. I imagine that you've got a prompt that came in, and now we're going to think about getting this model to talk.

And, of course, there are many methods. We could use greedy decoding, which involves generating the most probable next token based on what has been input so far. We consider this a kind of default for generation, but it is not the only method available; it is just one option among many. We could also consider top-p sampling. This approach involves sampling tokens from the most probable choices according to the model's distribution.

We could also employ a technique like beam search, which involves sampling tokens from the most probable options according to the model's distribution. This method is more exploratory. Additionally, we could prioritize token diversity in these generations, which represents an even higher-level ideal we could impose. Furthermore, we could even insist that the output we generate conforms to valid JSON, introducing a significant consideration in our approach.

And that is just the tip of the iceberg. In fact, if you start to look around in this literature, you will find that it is a space full of innovative ideas. I won't have time to go through these in detail, but many of these methods are doing things like making use of the gradients to gather much more information about the forward-backward flow in the model. A lot of them focus on ensuring that the generated output conforms to a high-level grammar, such as that used in logic or computer programming environments. Here’s a really nice paper that attempts to do this efficiently and also offers an overview of several prior methods aimed at ensuring model generations conform to the specifications of computer code and similar applications. Additionally, here’s a very recent paper that adds parameters to the model in an effort to adaptively find a good temperature so that the model can either be creative or constrained, depending on the task you want it to solve at any given moment. This is a really innovative way to approach sampling.

And if you'll permit me one step further, we could expand on what we mean by sampling for generation and really delve into aspects that resemble creative exploration. I've categorized this under the heading of majority completion strategies. This is a straightforward example of a pattern we are observing frequently now in technology and research. Imagine you have a prompt that involves a particularly challenging reasoning task. Instead of asking the model to generate the answer in one step, which might be too difficult, you could instead have it create a reasoning path and then produce an answer.

But you could do that by sampling multiple reasoning paths, and different reasoning paths might lead to different answers. So, if I do this a bunch of different times, I get a distribution over the answers. I might then say that the actual generated output in this context will be the answer that was the most common outcome, given all the diverse reasoning paths. This is a way of letting the model explore and engage in some reasoning during generation before insisting on a final answer.

Now, this isn't strictly speaking just a sample of Datalink strategy, but we might trick the user a little bit and hide from them all of those intermediate steps so that it looks like we have simply sampled an answer over here on the right for their input. That’s just one simple instance of the many things that you could do as reasoning paths lead to other answers, which lead to other reasoning paths, and so forth, before finally producing the answer at the end. So, that's the tip of the iceberg, I think, even though that’s a pretty rich array of ideas. This is the essence of this, though: There is no one true sampling method for your model. This is a highly consequential step.

In some sense, you are making the model speak, which it does not really do intrinsically. The choice that you make here is going to be highly consequential for the overall system that you design. I emphasize this because we too often don't consider the sampling method, even though you can see that it really matters for the behavior of the overall system, interacting in complex ways with the language model that you've chosen as your basis. So that's sampling. Let's think now about prompting.

And prompting is really the heart of modern AI system development. Probably many of you out there have engaged in prompt engineering. You've worked very hard on prompts and have seen that you can achieve complex and interesting behaviors from the resulting system. That is certainly the core of all of this. If you've done it long enough, you might have discovered that it can also be quite heartbreaking.

I want to discuss that with you as well. Let's reflect, though, for a minute on the origins of all this. We've started to take it for granted, but it's a new and very unusual idea. I think the origins of this really trace back to the GPT-2 paper from 2019 by Radford et al.

There are some precedents before this, but this is the paper where it really crystallized, I believe. In that paper, they state, "We demonstrate that language models can perform downstream tasks in a zero-shot setting without any parameter or architecture modification." They go on to say, "To induce summarization behavior, we add the text 'TL;DR' after the article and generate 100 tokens." Regarding in-context learning, I want to confide in you all that when I first read this way back in 2019, I did not properly understand what it was saying.

I was so cognitively biased against this kind of thing working that I thought surely somewhere hidden in this paper was a description of some fine-tuning method that they used to get that token TL;DR to produce summarizations. But now I see, of course, that they really meant what they wrote, which was perfectly clear. I was just not primed to really understand it. They could get it to summarize simply by relying on this language model to do in-context learning. Really amazing.

And in that paper, they also tried translation, question answering, text completion, and reading comprehension. Performance is variable across them, but mostly, they can get a signal, and it really is a striking exploration of this idea. We now take it for granted, but at the time, at least for me, it was very surprising. Then fast forward just one year, and we get GPT-3, right? We've gone from 1.2 billion parameters for GPT-2 to now up to 175 billion.

The GPT-3 paper is an incredible exploration of what scaling leads to. It’s full of cases where they demonstrate successful, complex in-context learning. Here’s an example of that: for question answering, we simply prompt the model with a context passage. Then, we provide it with a few demonstrations, which are also part of the prompt we’re creating here.

The prompt is meant to demonstrate that we want the behavior to be such that the answer is a substring of the context passage. We may provide a few examples, and then we present our target question. The major discovery is that, with context, the model can learn to imitate this behavior and answer questions as substrings of the passage. This is truly striking behavior. The paper already identifies a general pattern for this approach: we start with some context or instructions, followed by a list of demonstrations that further exemplify the behavior we want to see.

And then finally, there is a target down here. They apply this template to a lot of different tasks, from QA to reading comprehension to machine translation. Here's a quick example: I could say, "Please unscramble the letters into a word and write that word as my instruction or context." I would give a few demonstrations, and then, at least at the time of GPT-3, I might get something that was at least close to the behavior I wanted to see.

And so forth and so on. So, there's already a template for building systems in this modern mode. It's very exciting. But, as I said, if you have done some work with prompting models, you have already encountered that there can be a real dark side to all of this. This is a paper that kind of crystallizes it for me.

This is from a group at Berkeley and the University of Washington. "Quantifying Language Model Sensitivity to Spurious Features in Prompt Design," or how I learned to start worrying about prompt formatting. This is a really dramatic case of the model's sensitivity to the prompt choices that you make. This is a sample table from the paper. They are looking at LLaMA-27B.

The paper explores a variety of other models. Just to take one example here, Task 280 is an instruction-following task. They have two prompt formats for it, and those formats differ only in whether or not they include colons after the words "passage" and "answer." That minor change leads to an 80-point difference in the performance of one model in the same language. This pattern is repeated across many tasks in the paper. This illustrates that with the wrong prompting strategy, you could make any model look arbitrarily bad.

But the essence of this for today is that I think it shows that it just doesn't make sense to ask what it means to evaluate this model in the context of these tasks. The only way to make sense of this question is to ask how the model paired with a prompting strategy is going to perform, and to think in those terms. That is already systems thinking. So, get rid of that question and think right away in terms of systems. You should be asking yourself, "What is the optimal prompt-model combination, given whatever goals you have?"

And let me just tell you a few more stories about this. I'm going to kind of elevate the discussion, starting with the chain of thought. This is a lovely paper called "Echo Prompt." It's an empirical exploration of different strategies for engaging in the classic step-by-step chain of thought reasoning. Here's a sample table from the paper.

You can see that they've just got different variants of that phrasing. Let's repeat the question, reiterate the question, and so forth and so on. This is a glimpse of the results, which show wide variation in terms of the performance of the model based on how we frame the chain of thought reasoning step. This is for Code Da Vinci 2, which is an older model.

But again, I think this reproduces for newer models, and it shows that it just doesn't make sense to ask this question. If you ask, "What does it mean to evaluate this as a model?"—even just for chain of thought—you'll find that you have to be thinking in terms of model prompt combinations, or in this case, model chain of thought combinations. Again, that is systems-level thinking, not model-level thinking. I believe that this will bring so much clarity to your own development cycles.

So, let's get rid of that question and take things one step further. I came across a nice tweet where someone noted, "Fun fact: I realized I was still using old GPT-4 for tool calling and as part of an agent. I updated to 4.0, and immediately broke things." I think if you've been in this business long enough, you've seen this happen yourself.

And this nice response tweet says: "I had the same exact experience where going from 3.5 to 4.0 mini triggered a fresh prompt engineering exercise as if it's a new project. I think this shows that the model and the prompt are inextricably linked. We write these prompts in English, but in fact, they are more like an effort to communicate with this sort of alien creature, the language model. We can deceive ourselves by thinking that our understanding is going to translate directly into the performance of the system. We would like to get away from this, but I think the first step toward that is thinking about these two things as inextricably linked."

And by the way, a fun fact here, which Petra pointed out to me: I didn't know this at the time of quoting this tweet, but this Twitter user is an alum of my natural language understanding course. So maybe it's not surprising that they are a fan, as you can see here, of DSPi, the programming library. One more final story about this: You might have seen that people found the Apple intelligence prompts. These are in system files that ship with the OS.

And the prompts are fascinating to read because you can kind of glimpse the development cycles that they had to go through to get the Apple intelligence models to behave in the intended way. They're full of elements that are somewhat like special pleading with the model to do what they want. You can tell that these prompts are very tightly knit to the models that they shipped. Again, this makes me want to take the perspective that even though they are in English, they are much more like compiled binaries, which are meant to be paired with a particular model. This reflects systems thinking, and I think it shows that these components are really interacting in tightly knit ways to deliver the performance that we want. We can't really separate the prompt as a system component from the model as a system component.

And this is a kind of funny thing to reflect on. This does lead us to the DSPI library that we've been trying to promote and encourage people to work on. It's a humorous moment because some of the core lessons of artificial intelligence seem to have been forgotten. So let's just step back and reflect on this. Throughout AI, we've been successful in part because we have adopted these lessons.

Modular system design, data-driven optimization, and generic architectures are the essence of what has enabled us to move so quickly, especially in the deep learning era leading to the exciting development of large language models. I think these time-tested lessons have been nicely embodied in libraries like Torch, Theano, Chainer, and especially PyTorch. These tools encapsulate these concepts and help us move more swiftly as a result. However, the current moment is somewhat amusing. At this time, we rely heavily on prompt templates and make manual adjustments to prompts, which leads to a complete dependence on the models we are using. As illustrated in the preceding stories, the prompting strategy, iterated on over time by hand, becomes tightly intertwined with whatever model we happen to be working with.

And this is kind of tragic for me because these time-tested lessons have taken us so far, and it's surprising that we seem to have forgotten them as we entered this new mode of AI system development. That does bring us to DSpy. DSpy is a model; it's a programming library for moving away from prompt engineering and toward language model programming. This approach really honors the insight that when you have a prompt, a language model, and a sampling strategy, you have designed a software system. We would like, in essence, to bring the core insights of artificial intelligence and software engineering to this new mode of AI system development.

If you haven't seen DSpy before, let me just give you a quick sense of how it works. At the top here, I've got some code that sets up some high-level tools that will define the system. For this example, I've chosen Turbo as my model. I've also created an index of Wikipedia, and I could have other tools at my disposal if I wanted. Those will be set up at a high level.

And then this here on the left, "dspy.predict question to answer," is a minimal system in DSPi for doing basic question answering. That's all it takes: just one line. What happens under the hood when I actually use that system is that it gets compiled down into an actual prompt to a language model, of course, because that's how we communicate with these language models. But you can see here that there's a real difference between the code that I wrote and the thing that gets compiled down. The thing that I actually prompt the language model with is very tightly knit to the particulars of that language model and could look different depending on the tools that I had set up at the top here.

So we've abstracted away; we've removed some of the model dependence that I was worried about before. That's a very simple program. I could also write a much more complex one. This is a program for doing multi-hop question answering, where I might want to gather evidence from a bunch of different passages in order to answer a question. As you can see here, it is mostly Python code.

It's also adhering very tightly to the design principles of PyTorch. It's meant to allow you to freely express in code the kind of system that you want to develop. The really nice aspect of all of this, in addition to these design principles, is that as a final step, I could optimize this program. In doing that, I would try to find a prompting strategy that was truly successful, given the labeled examples that I have and independent of whatever tools I had chosen. What I'm showcasing at the bottom is an option that would allow you to simultaneously optimize the instructions, as well as the few-shot demonstrations that you were using.

Those are both crucial aspects of successful prompting. This approach shifts the burden of finding effective methods onto the optimizer, revisiting the time-tested lesson of data-driven optimization. To illustrate how significant this can be and the importance of expressing systems in these terms, I want to highlight a few results from the original DSPI paper. This will emphasize various aspects of why it's essential to think in systems terms. For evaluation, I will use the program I am going to write and the optimizer as a framework. These will be paired with the language model to specify a complete system.

And for the two models, I'll have Turbo and Llama 213b to show very different sizes and types of language models. At the top here, I have a real baseline system. It's the one that I showed you before, where I just go from questions to answers. For my optimizer, I'm just going to randomly select a few few-shot demonstrations to help the model understand what kind of behavior I want to see. For this baseline, we get about 34 for Turbo and 27.5 for Llama 2, where our metric here is exact match on the answer that we want to see. So, that's a baseline system.

If I go up and just use retrieval-augmented generation, with a very simple DSPi program, I already get a boost just from gathering relevant information. If I implement bootstrap few-shot optimization, where I'm using the DSPi program to generate full examples, which could include retrieved passages and include those in the prompt as examples of the behavior I want to see, I achieve significant gains from my baseline, all the way up to 42 and 38 here. We could also consider using React agents. This is a very interesting proposal for having the model reflect and think about how to solve the task and break it down into pieces. While this approach is less successful for this problem, it still demonstrates the power of systems thinking, as I have combined my program with different optimization strategies and again observed real gains over the baseline. Interestingly, the human reasoning prompts—those that were carefully crafted by hand—actually underperform compared to the prompts generated through simple bootstrapping and in this React context, which further illustrates the power of data-driven optimization over trying to develop your own prompting strategy intelligently.

And then finally, at the bottom here, if we move to a program that is designed to do multi-hop reasoning, it is designed to gather evidence from multiple passages and use that to provide answers. We get really good systems here. We have gone all the way from 34, whereas our baseline is up to almost 55 for turbo, and we have seen even larger gains from 27.5 all the way up to 50 for the smaller model. This really shows the power not only of intelligent system design—where we consider the prompting strategy and the optimizer—but also illustrates the potential we can achieve with small models. We have almost closed the gap here, and we observed larger gains for the smaller of the two models.

And that's a bit of a digression, but this is so important in the current moment that we think about designing systems that can get the most juice possible out of small models. This is a really insightful post from an analyst at Theory Ventures, titled "Small but Mighty AI." It includes the observation that 77% of enterprise usage of models is at the 13 billion parameter size or smaller. So, it's not the largest models that get all the headlines, but actually much smaller ones. For a glimpse into why that is happening, you could just think about the latency numbers included in this blog post. If you've worked at all on industrial systems, you'll know that it's very nice to be at the space of around 18 milliseconds for latency. However, as we move up to latencies that are more like above 50 milliseconds, and certainly all the way up to 750 milliseconds, we encounter real headaches. At some level, those headaches are going to translate into high costs, and of course, that might be prohibitive.

You could have a wonderful solution, but if it costs too much for people to use it relative to what you're gaining in the rest of the organization, it's just a non-starter. Again, that creates pressure to choose the smallest models. However, to get any value out of those small models, we really need to think about the systems that we are designing around them. Your prompt will be a deciding factor in your system's performance. That's the aspect that I want to emphasize the most here. Finally, I want to talk a little bit about tool access because, as I mentioned before, that is where we really begin to think in terms of entire systems, not just in terms of models.

So, this is the step where I'm going to actually bring in calculators, programming environments, databases, the web, and web APIs, and so forth. I think it's really clear at this point that these are systems, so I thought it would be nice to, instead of diving into the technical details, think about the overall consequences of designing systems in these terms. As some food for thought for you, let me pose a few different questions. I'm going to offer you choices between two types of systems, and you can reflect for yourselves on which system you would prefer for whatever you're trying to do out there in the world. First question: Which is more reliable?

A giant large language model that embeds a snapshot of the entire web as of today, and then it's frozen, or a tiny language model working with an up-to-date web search engine— which would be more reliable? Here's a second question: Which one would you prefer in a general sense? A giant large language model doing contextless autocomplete on your phone via a centralized service, sending these completion messages back and forth, or a small model doing that same autocomplete task locally on your phone or device, using your own chat history? Which would you prefer?

How about this? Which one is more dangerous: GPT-4 with no access to databases or the web, or a 10-billion-parameter language model that has been instructed to log into websites and has tool usage giving it access to the web? Which would be more dangerous? Finally, what do you expect to see in 2026? Massive foundation models that perform math, retrieval, reasoning, and so forth entirely in terms of their parameters and standard computations, or systems consisting of multiple models and tools working together in a coordinated fashion to handle tasks like math, retrieval, and reasoning?

Maybe in the Q&A, we can discuss these questions together. Let me start to wrap up here by offering you a few thoughts on the high-level consequences of all of this for technology and society. Let's begin with the recent legislation, SB 1047, which was vetoed by California's Governor Gavin Newsom. SB 1047 sought to do many things, but one of the main objectives was to introduce new regulations based on the size of models. In particular, SB 1047 aimed to regulate models that cost more than $100 million to train and had 10^26 FLOPS performed during training—a massive scale, indeed. Models of smaller sizes were largely exempt from regulation under SB 1047. Newsom's veto and the rationale he provided are quite interesting.

He notes that smaller, specialized models may emerge as equally or even more dangerous than the models that were targeted by the legislation. I'm not sure what prompted him to say this, but I think it is really wise. It reflects the observation that we could take small models and embed them in complex systems that, as systems, could do things that are truly surprising. They could be productive, but they could also be quite dangerous. The combination of those elements working together in a system could be more dangerous than a very expensive model that is simply sitting on a disk, unable to access the web and engage in other activities that could lead to trouble.

So ultimately, I think this was a wise decision and points toward the idea that future legislation should be oriented around systems, not around models. We could also think about the consequences for research, and in particular for the kinds of comparative evaluations that we conduct. There are lots of leaderboards out there that are meant to rank different language models. This is one from Hugging Face, this is Chatbot Arena, and we have HELM from Stanford.

And all of the entrants in these competitions are nominally listed as individual models. But of course, you can't actually evaluate a model; you can only evaluate a system. Under the hood, we need to have at least a prompting strategy, a model, and some procedure for generation—the minimal system. I can't help but feel that this is not quite what we want to be evaluating. For my Formula 1 race car analogy, this is akin to running races that are really just engines with wheels.

If you want to run a race of engines with wheels, that might be a sensible thing for you to do, but we should be clear-sighted about the fact that it is probably not the thing that we think we are doing. What we really want to do is think about these language models as components in larger systems. Therefore, I would exhort the community to reorient all of these leaderboard evaluations around entire systems. We could give a privileged place to the language models, if we want to, as important components, but we should really consider all the pieces working together in constellation, because that is the relevant aspect to address when we think about these technologies being deployed out in the world. Here is the final slide, which contains a prediction about the future.

So, this is also something that you might think about and ask questions regarding. I'm just reflecting on these last five or so years and what has happened. What I think we see already are a few different notions of scaling in play as driving forces behind all of the progress that we've seen happen so fast. In 2020, with the GPT-3 paper, we kind of began the era of scaling unsupervised training. That is, taking a really big language model and having it learn to imitate all the data on the web and all the data you can find. Indeed, we continue to live in an era in which scaling up these processes is showing some gains.

But I believe that, on its own, this is not showing the kind of gains that we're seeing overall in AI. Those gains now are being driven, at least in part, by scaling up in a different sense. This is scaling up instruction fine-tuning. Starting in about 2022, with especially ChatGPT, we saw the power of having large teams of very smart humans create good input-output pairs that we can use to update models so that they learn particular things and acquire particular skills. And again, we continue to live in an era in which scaling these things up is leading to gains.

But we think we see that it's not a silver bullet. This has very recently led to a rise in the first theme that I mentioned. If you consider scaling and sampling for generation, we are now observing very sophisticated forms of sampling for generation that you could think of as a kind of scaling up of inference time processing—search that you might do on your way to producing an answer to a user query. I believe this trend will continue from 2024 onward. Now, here’s my final prediction: as we think about the future, we are going to see systems scaling up in transformative ways. This will occur due to the fact that we can take perfectly good language models—perhaps even small ones, if we are considering high-volume services—and give them access, in productive ways, to a variety of tools and resources that enhance their capabilities as systems. That's my prediction about the future.

I think you should join me in moving from LLM thinking to full-on system thinking. I believe it will make you more productive in your work and lead to bigger gains, just like I'm predicting for 2025 and beyond. So, I'll stop there. Thank you very much. I'm eager to hear your questions and comments. Thank you so much, Chris.

Amazing! It's so wonderful to hear from you again. I always learn so much. We have a bunch of questions, and hopefully, more will come up during this Q&A part of the session. Maybe the first one is more about understanding the kind of compound systems and their relationship to generative agents.

So this question has come up a few times: How do generative agents fit into this whole concept? Agents, yes. They could be a technique used to take a language model and enable it to perform tasks that it couldn't accomplish through simple generation alone. Additionally, they could be the key to allowing the language model to utilize tools and leverage their outputs. I believe this truly represents systems thinking. I would encourage people not to be purists in this regard. If you consider this as a software system, you could design an agent that relies entirely on the model to handle complex tasks in generation and effectively address the problems you present to it.

But you could also write some code that would help bridge the gaps between the language model's capabilities and the things that you want to see. If you're designing a system, that's a very natural thing to play with. I think we could test this by looking at the results that you achieve. Thank you so much. Near the beginning of the talk, you mentioned that modern AI systems are already producing multiple reasoning paths in the background.

How does the system produce reasoning paths? And now, how can we give users more access to the inner workings of these systems to help them evaluate the results? That's an interesting question. There are a few layers to this. The concept I was referring to with the majority completion really took off with the advent of chain of thought methods, like "let's think step by step."

And that was just the simple observation that if you had a model go through that, prompting it with "Let's think step by step" and allowing it to generate tokens in response, the process of producing all those tokens in ways that we don't fully understand often led it to more reliable answers. In retrospect, we can see that this was just the tip of the iceberg in terms of what's possible. For instance, I could have it perform a chain of thought reasoning to generate an intermediate step, which could itself generate further chain of thought reasoning. I could have numerous inference paths leading to a variety of different outcomes, and then I could consider conducting a statistical analysis of all those outputs to decide on a final generation. This creates a real playground of different ideas and techniques to explore. It truly embodies systems thinking, as you must consider the prompt that was received, the structure of the overall system—which might even involve multiple language models working together—and also how you're going to perform these generations, including the sampling method you'll use.

And it adds that twist of how you are going to decide on the final answer. In my illustration, it was the most commonly occurring answer across all the reasoning paths, but you could even think about different variants of that idea. I think we have seen a kind of culmination of this in the OpenAI announcements of its 401 models, which are clearly doing a lot of inference-time work before they produce an answer for you. Part of the question was how we could expose more of that. I believe OpenAI is not going to expose more of this.

I think they regard these as trade secrets, but you could explore with smaller models to see what kind of behaviors you can coax out of them. I believe a lot of this is going to start being explored in the research literature going forward. Thank you so much. The next question is about the challenges to solve or to reach system-level scaling. What do you see as still missing or the most challenging aspects to get there? What would you point out?

So, that's kind of like what is going to happen in 2025 and beyond in terms of scaling systems. I mean, they will just get ever more complex. You could think of a paradigm case of this as Google Search. We have some precedent for this, where at some level, it began from a very simple search technology that was just indexing pages on the web. By now, in 2024, that is probably such a complex software system that no one individual could even begin to understand it, but it still functions as a result of teams of people and lots of dynamic behavior. We're just at the starting point for these generative AI systems; they look like Google did in probably the year 2001. Over the next few decades, I think we're going to see incredible systems being built up.

And I think the thing to watch is when people actually do provide lots of tools and tool access. Essentially, these language models could become like you or me as we cruise around on the web, capable of trying different pages, logging into different systems, and communicating with people on social networks. When that really starts to happen in a very freeform way—perhaps even in ways that the system designers don't fully understand—we are sure to see consequential things happen in the world. You can probably hear in my tone of voice that I believe some of these developments will be productive, while others may be quite problematic. However, one can predict that it’s going to be consequential.

There is also a follow-up question regarding that. What are the recommended guardrails? How should we think about not only the positive consequences but also the potential negative consequences? How can we establish the guardrails, and what factors should we consider at this stage? That is a wonderful question. I'm not a regulator or a lawyer, but if I were, this would be where I would focus all of my attention.

As I said, I think focusing on the language models is going to miss the mark. However, I believe that if we concentrate on regulating the systems, we're more likely to get it right for a few reasons. First, we already have legislation that governs how these systems can behave, indeed how any software system can behave. I frankly think that much of that will carry over into the generative AI realm more than we currently give it credit for. We could also consider human aspects, such as guarantees that these systems need to identify themselves as non-human when interacting with us. This helps us, as people, figure out how to calibrate our responses to them as agents, and it could also aid in controlling the situations we allow them to enter. Additionally, there might need to be some fundamental restrictions on whether or not we allow these models to log into certain websites or interact freely on specific social networks.

I'm not sure; I'm taking a kind of wait-and-see attitude, and I guess I'm hoping that the initial disasters are not cataclysmic so that we can learn from them and figure out how to respond as a society. Thank you so much. There are a few questions about consequences for individuals. So we talked briefly about how this can inspire technology and society. How can people think about this for themselves? There was actually a question that was phrased, "How might this influence a normal person like me in the next five years?"

Like, what would you say to that? Oh, I think it's going to impact all of our lives. I think if we weren't even thinking about AI, it would still impact our lives because we’re going to see more and more systems that can help us in our daily routines. And again, a language model on its own is not going to help you very much at all because it is kind of inert. But a language model embedded in a system that has prompting strategies and access to tools could be something that helps you with low-level tasks in your life.

It could also help you with interacting, companionship, the discovery of new ideas, and creative expression. I think we're going to find systems that can assist us with all of these aspects, which is the bright point. Additionally, we may see advancements in areas like education. I believe we're going to discover systems that can support us in all of these endeavors.

And that's the bright side. Additionally, when it comes to education, I believe there is great potential for customized educational experiences at low costs, thanks to generative AI. However, we must also be cautious of bad actors who may attempt social engineering tactics to obtain our usernames and passwords. This represents the darker side of granting AI the ability to log into websites. If left unchecked, these systems might exhibit surprising and potentially harmful behaviors in pursuit of their objectives. Therefore, individuals in society need to remain vigilant regarding AI systems that may run amok. Many people are asking how they can learn more about this topic.

One of the groups of people I am observing is very technical; they are trying to learn more about DSP, why it matters, and what you are recommending. The other group consists of leaders and business executives who are uncertain about how to grasp the implications of this technology for their business and leadership strategies. Do you have recommendations for both groups? Oh, fascinating!

In terms of general education for DSPI, we have a Discord server and plenty of tutorials. I believe one wonderful way to get involved in an open-source project like this is to file issues or even create pull requests that may lead to contributions. This is a great way to introduce yourself to the community, make a positive impact, and learn about the topics that are on people's minds. Perhaps the first step is to join the Discord server. It's thriving and will give you a sense of the kinds of projects people are working on. Some members are collaborating in teams, and you might consider joining forces with them.

So, that's wide open. The research community is also wide open. I do think there are wonderful things on YouTube that could help you think about different prompting strategies, agents, and tool usage. A lot of the themes that I touched on today can be unpacked into entire courses, frankly. YouTube is pretty good about that. In terms of understanding what's happening in the industry, unfortunately, it seems like things are getting increasingly closed, and we're losing a lot of insight into exactly the decisions they're making and why they're making those decisions, even at the level of research innovation.

If you are a leader in an organization considering how to define a generative AI strategy, it may be worth its own separate learning lecture. However, I can offer some advice. The main point is to think early on about what success looks like and what kind of testing you will conduct. This should be a guided process with clear goals in mind. You should also consider designing a system that balances your objectives with the known risks of deploying a system in the current era. Great!

Thank you very much for your question. There are people asking how to stay updated with information in the field. As you mentioned, you showed a nice slide about Twitter, now known as X. How do you personally get information and keep up to date with current developments? If someone has maybe ten minutes a day to keep up with the latest in the field, what would you recommend they follow? I do feel a sense of loss about this because four or five years ago, Twitter was my go-to resource. My timeline was full of people announcing and discussing research papers, which made it a great way to filter and understand what was important and what innovative developments were occurring. However, due to changes at Twitter—now X—it feels less vibrant in that regard. It seems as though the communities have spread out into Blue Sky, Threads, and Mastodon, making it less reliable. One positive aspect of the current situation, which is somewhat meta, is the rise of generative AI. In particular, systems that utilize retrieval-augmented generation for research papers allow you to gain an understanding of an area simply by typing a common-sense question, such as "What is deep learning?" or "What is chain-of-thought?" into search engines like ChatGPT, which now includes search functionality.

I saw that there's a new tool from the Semantic Scholar Institute that does this, and I think it could be very productive in terms of getting a sense of what's happening in the literature and where to begin regarding papers to check out, and so forth. NLP is relatively straightforward for this because it's a very organized community in terms of its literature. If you go to the ACL Anthology, it pretty much includes every NLP paper, and you can use the search functions together with citation information to understand what's most influential in the NLP field and what the latest developments are. This also presents a nice opportunity to promote the course that Petra and I teach, Natural Language Understanding, which includes a project development phase involving techniques for building a good literature review, forming an experimental protocol, and then writing a paper, possibly accompanied by some associated code.

And that is a kind of guided way to do a focused research project and get a sense of the rhythms of research in the domain. Associated code. Great, thank you. Maybe we can take one or two more questions, preferably shorter ones.

One of the questions I'm encountering is whether DSPI is gaining traction in the business world. My engineering team is very focused on long chains, but I'm trying to help them see what DSPI offers. We've actually received this question a few times regarding long chains and where to look next. Do you have any recommendations or input on this topic? Yes, I would suggest checking out...

So, dspy.ai is the website. It has documentation and also lists out use cases. One nice aspect of those use cases is that many of them are blog posts from various organizations, ranging from JetBlue to very new and small startups that have been using dspy in different ways. This could give you a sense of the coding patterns, the kinds of problems that people have tackled, and also plenty of starter code.

And then I think, at a high level, whether you use DSpy or not, it could be LangChain. However, I believe the most important thing to do if you're just starting out is to make some principled choices. It's very tempting at this point to begin with some prompt templates, and I think that can be very productive in terms of teaching you things. However, the problem is that you might look back in six months and find that you have designed, in some sense, a system entirely around these prompt templates. Now, any changes you want to make may be almost impossible, and they could have unintended consequences throughout the system.

And then you have that dreaded moment, which I alluded to, where somebody says you can't use Claude anymore. You have to use these OpenAI models or these LLaMA models. What you discover is that now everything is broken, and that really does mean going back to step zero and rewriting all these prompt templates. You have to avoid that failure mode.

If you're already entrenched in prompt templates, you might just have to live with that and try to find a way out. However, if you're just starting out, consider engaging with tools that allow you to express these concepts as proper software systems. I believe DSPI is a great choice for that. It's particularly tailored to individuals who are experienced in machine learning and incorporates many of the PyTorch principles I mentioned earlier. There may be a bit of a learning curve at the beginning, but I think the investment will pay off by ultimately leading you to a system that is very flexible, adaptable, and capable of responding to new requirements and changes in the underlying environment that you're working in. Thank you.

And one last kind of summary question: If people should take away one important thing from this talk—one fact or piece of information that you believe is the most crucial, even if they forget everything else—what would it be? What would you recommend? I think it really is to avoid the trap of thinking entirely in terms of models. We're all doing it, and I feel like it's a trick we're playing on ourselves because we always focus on what is currently popular.

The only way is to just avoid the trap of thinking entirely in terms of models. We're all doing it, and I feel like it's a trick we're pulling on ourselves because we always talk about the latest model releases. We talk about things that are clearly software systems, like ChatGPT, as though they were models when, in fact, they are not. If you embrace the fact that it's a system, you will concentrate your energy not just on model choice or its properties, but also on the other aspects that are so consequential. In that way, you'll be like an F1 race car design team that, of course, is focusing on much more than just the engine. It is an effort to get all of those complicated pieces to work in concert to achieve something really difficult. Intrinsically, I think that's a better perspective to have.

And then, if you think about that note regarding what's happening in the industry, where most of the energy is focused on small models, this is especially important. With a small model, you can't rely on simplistic system design or a simple prompting strategy. You have to do everything you can to get that relatively small model to achieve significant outcomes in the world. This really places more pressure on system design. However, as you can tell, I believe that this pressure is actually just a huge opportunity. Thank you so much, Chris. Thank you for making the time.

Thank you for all the wonderful questions. We couldn't get to all of them, but we tried to cover most of them at a higher level. We appreciate everybody joining. We will post this on YouTube and share the recording of this session.

So, thank you again, and have a wonderful day. Thank you, Petra, and thanks to everyone for all those questions. This was really a wonderful discussion.