Deep Dive into LLMs like ChatGPT

Hi everyone. I've wanted to make this video for a while. It is a comprehensive yet general audience introduction to large language models like ChatGPT. What I'm hoping to achieve in this video is to provide you with mental models for understanding what this tool is. It is, in many respects, magical and amazing.

It's really good at some things, not very good at other things, and there are also a lot of sharp edges to be aware of. So, what is behind this text box? You can put anything in there and press enter. But what should we be putting there? And what are these words generated back?

How does this work? And what are you talking to exactly? I'm hoping to cover all those topics in this video. We're going to go through the entire pipeline of how this stuff is built, but I'm going to keep everything accessible to a general audience. So, let's take a look at how you build something like ChatGPT.

And along the way, I'm going to discuss some of the cognitive and psychological implications of tools. Okay, so let's build ChatGPT. There will be multiple stages arranged sequentially. The first stage is called the pre-training stage, and the first step of the pre-training stage is to download and process the internet.

Now, to get a sense of what this roughly looks like, I recommend looking at this URL here. A company called Hugging Face collected, created, and curated a dataset called FineWeb. They go into a lot of detail in this blog post on how they constructed the FineWeb dataset. All of the major LLM providers, like OpenAI, Anthropic, and Google, will have some equivalent internally to something like the FineWeb dataset. So, roughly, what are we trying to achieve here? We're trying to gather a large amount of text from the internet, from publicly available sources.

We are trying to create a vast quantity of very high-quality documents. Additionally, we want a wide diversity of documents to ensure that the models contain extensive knowledge. Therefore, we aim for a large variety of high-quality documents and we want many, many of them. Achieving this is quite complicated, and as you can see, it involves multiple stages to do well. Let's take a look at what some of these stages look like in a moment.

For now, I would like to note that, for example, the FineWeb dataset, which is fairly representative of what you would see in a production-grade application, actually ends up being only about 44 terabytes of disk space. You can easily get a USB stick for around a terabyte, or I think this could almost fit on a single hard drive today. So, this is not a huge amount of data at the end of the day. Even though the internet is very, very large, we are working with text and filtering it aggressively. Thus, we end up with about 44 terabytes in this example.

So let's take a look at what this data looks like and what some of these stages are. The starting point for many of these efforts, and something that contributes most of the data by the end, is data from Common Crawl. Common Crawl is an organization that has been scouring the internet since 2007. As of 2024, for example, Common Crawl has indexed 2.7 billion web pages. They have crawlers that traverse the internet, starting with a few seed web pages and then following all the links. By continuously following links and indexing the information, they accumulate an enormous amount of data from the internet over time.

So this is usually the starting point for a lot of these efforts. Now, this Common Crawl data is quite raw and is filtered in many different ways. Here, they document—this is the same diagram—the kind of processing that happens in these stages. The first thing here is something called URL filtering. What this refers to is that there are block lists of URLs or domains from which you don't want to collect data. Usually, this includes things like malware websites, spam websites, and marketing websites—essentially URLs or domains that you want to avoid.

So, usually, this includes things like malware websites, spam websites, marketing websites, racist websites, and similar content. There are a ton of different types of websites that are eliminated at this stage because we don't want them in our dataset. The second part is text extraction. You have to remember that all these webpages contain the raw HTML saved by these crawlers. When I go to inspect, this is what the raw HTML actually looks like.

You'll notice that it has all this markup, like lists and other elements, and there's CSS and various other components. This is almost computer code for these web pages. But what we really want is the text; we just want the content of this web page without the navigation and other distractions. A lot of filtering, processing, and heuristics go into adequately isolating just the valuable content from these web pages. The next stage here is language filtering.

So, for example, FineWeb filters using a language classifier. They try to determine the language of every single web page and only keep web pages that have more than 65% English content, as an example. This highlights a design decision that different companies can make for themselves regarding the fraction of various languages to include in their dataset. For instance, if we filter out all of the Spanish content, it is likely that our model will not perform well in Spanish later on because it has not encountered enough data in that language.

Different companies can focus on multilingual performance to varying degrees. For example, FineWeb is quite focused on English, so their language model, if they end up training one later, will be very good at English, but may not perform well in other languages. After language filtering, there are a few other filtering steps, such as deduplication and others. This concludes with, for example, personally identifiable information (PII) removal. PII includes information like addresses and social security numbers; efforts are made to detect and filter out webpages containing such data from the dataset as well. There are many stages in this process, and while I won't go into full detail, it is a fairly extensive part of the pre-processing, resulting in the FineWeb dataset, for example.

So when you click on it, you can see some examples of what this actually looks like, and anyone can download this from the Hugging Face webpage. Here are some examples of the final text that ends up in the training set. One example is an article about tornadoes in 2012, discussing various tornadoes that occurred that year. The next one is about how you have two small, yellow, nine-volt battery-sized adrenal glands in your body. This is a rather unusual medical article. You can think of these as essentially web pages on the internet, filtered for text in various ways. Now we have a vast amount of text—40 terabytes of it.

And that now is the starting point for the next stage. Now, I wanted to give you an intuitive sense of where we are right now. So, I took the first 200 web pages here, and remember, we have tons of them. I just took all that text and put it all together, concatenating it. This is what we end up with: raw text, raw internet text, and there's a ton of it, even in these 200 web pages.

So I can continue zooming out here, and we just have this massive tapestry of text data. This text data has all these patterns, and what we want to do now is start training neural networks on this data so the neural networks can internalize and model how this text flows, right? We have this giant tapestry of text, and now we want to get neural nets that mimic it. Okay, now before we plug text into neural networks, we have to decide how we're going to represent this text and how we're going to feed it in. The way our technology works for these neural networks is that they expect a one-dimensional sequence of symbols, and they require a finite set of symbols that are possible. Therefore, we need to decide what those symbols are and then represent our data as a one-dimensional sequence of those symbols. Right now, what we have is a one-dimensional sequence of text; it starts here, goes there, and then comes here, etc. This is a one-dimensional sequence, even though on my monitor, of course, it's laid out in a two-dimensional way. But it goes here, and then it comes here, etc.

So this is a one-dimensional sequence, even though, on my monitor, it's laid out in a two-dimensional way. It goes from left to right and top to bottom, right? It's a one-dimensional sequence of text. Now, this being computers, of course, there's an underlying representation here. So if I do what's called UTF-8 encoding on this text, I can get the raw bits that correspond to this text in the computer. And that looks like this. For example, this very first bar here represents the first eight bits.

So what is this thing, right? It’s a representation that we are looking for in a certain sense. We have exactly two possible symbols, 0 and 1, and we have a very long sequence of them. Now, as it turns out, this sequence length is actually going to be a very finite and precious resource in our neural network, and we don't want extremely long sequences of just two symbols. Instead, what we want is to trade off the size of this vocabulary and the resulting sequence length. So, we don’t want just two symbols with extremely long sequences.

We're going to want more symbols and shorter sequences. One naive way of compressing or decreasing the length of our sequence is to consider a group of consecutive bits, for example, eight bits, and group them into a single unit called a byte. Because these bits can be either on or off, if we take a group of eight of them, there are only 256 possible combinations of how these bits can be on or off. Therefore, we can re-represent this sequence as a sequence of bytes instead. This sequence of bytes will be eight times shorter, but now we have 256 possible symbols.

So every number here goes from 0 to 255. Now, I really encourage you to think of these not as numbers, but as unique IDs or unique symbols. So maybe it’s a bit better to actually think of these; if you were to replace every one of these with a unique emoji, you'd get something like this. Essentially, we have a sequence of emojis, and there are 256 possible emojis. You can think of it that way.

Now, it turns out that in the production of state-of-the-art language models, you actually want to go even beyond this. You want to continue to shrink the length of the sequence because, again, it is a precious resource in exchange for more symbols in your vocabulary. The way this is done is by running what's called the Byte Pair Encoding algorithm. Essentially, we are looking for consecutive bytes or symbols that are very common. For example, it turns out that the sequence 116 followed by 32 is quite common and occurs very frequently.

So, what we're going to do is group this pair into a new symbol. We will mint a symbol with an ID of 256 and rewrite every single pair 11632 with this new symbol. Then, we can iterate this algorithm as many times as we wish. Each time we mint a new symbol, we decrease the length while increasing the symbol size. In practice, it turns out that a pretty good setting for the vocabulary size is about 100,000 possible symbols.

So in particular, GPT-4 uses 100,000 and 277 symbols. This process of converting from raw text into these symbols, or as we call them, tokens, is referred to as tokenization. Let's now take a look at how GPT-4 performs tokenization, converting text to tokens and tokens back to text, and what this process actually looks like. One website I like to use to explore these token representations is called TickTokenizer. Here, come to the dropdown and select CL100KBASE, which is the GPT-4 base model tokenizer.

And here on the left, you can input text, and it will show you the tokenization of that text. For example, "hello space world" turns out to be exactly two tokens: the token "hello," which has the token ID 15,339, and the token "space world," which has a token ID of 1, 15,339. Additionally, "spaceworld" corresponds to the token ID 1, 9, 1, 7. So we have "hello," "space world."

Now, if I were to join these two, for example, I'm going to get two tokens again, but it's the token "h" followed by the token "hello" without the "h." If I put in two spaces between "hello" and "world," it creates a different tokenization. There's a new token "220" here. You can play with this and see what happens. Also, keep in mind that this is case-sensitive. So, if this is a capital "H," it is something else.

Or if it's "Hello, world," then this actually ends up being three tokens since there are just two tokens. Yeah, you can play with this and get an intuitive sense of how these tokens work. We're actually going to loop back to tokenization a bit later in the video. For now, I just wanted to show you the website, and I wanted to demonstrate that this text will essentially be seen by GPT-4 as follows: for example, if I take one line here, this text will be a sequence of length 62.

This is the sequence here, and this is how the chunks of text correspond to these symbols. And again, there are 100,277 possible symbols, and we now have one-dimensional sequences of those symbols. So, yeah, we're going to come back to tokenization, but that's where we are for now. What I've done now is taken this sequence of text that we have in the dataset and re-represented it using our tokenizer into a sequence of tokens. And this is what that looks like now.

So, for example, when we go back to the FineWeb dataset, they mentioned that not only does it take up 44 terabytes of disk space, but it also contains about a 15 trillion token sequence. Here, these are just some of the first one, two, or three thousand tokens from this dataset. But remember, there are 15 trillion tokens in total. It's important to keep in mind that all of these represent small text chunks; they are like the atoms of these sequences. Additionally, the numbers presented here may not make any sense.

They're just unique IDs. Okay, so now we get to the fun part, which is the neural network training. This is where a lot of the heavy lifting happens computationally when you're training these neural networks. In this step, we want to model the statistical relationships of how these tokens follow each other in the sequence.

So, what we do is come into the data and take windows of tokens. We select a window of tokens from this data fairly randomly, and the window's length can range anywhere from zero tokens up to a maximum size that we determine. For example, in practice, you might encounter a token window of about 8,000 tokens. In principle, we can use arbitrary token window lengths, but processing very long window sequences would be computationally expensive. Therefore, we decide that 8,000, or sometimes 4,000 or 16,000, is a good number and crop it there.

Now, in this example, I'm going to be taking the first four tokens just so everything fits nicely. So these tokens, we're going to take a window of four tokens: "this," "bar," "view," and "ing," as well as "space" and "single," which are these token IDs. What we're trying to do here is basically predict the token that comes next in the sequence. So, 3962 comes next, right? What we do here is that we call this the context.

These four tokens provide context, which feeds into a neural network. I will go into detail about what is inside this neural network shortly. For now, it's important to understand the input and output of the neural network. The input consists of sequences of tokens of variable length, anywhere between zero and a maximum size, such as 8,000. The output, in turn, is a prediction of what comes next.

Since our vocabulary has 100,277 possible tokens, the neural network will output exactly that many numbers, and all of those numbers correspond to the probability of each token being the next in the sequence. It makes guesses about what comes next. In the beginning, this neural network is randomly initialized, which we will explore shortly, meaning it's a random transformation.

The probabilities at the very beginning of the training are somewhat random. Here, I have three examples, but keep in mind that there are 100,000 numbers in total. The probability of the token "space direction" is currently 4%, while "11,799" is at 2%. Lastly, the probability of "3962," which corresponds to "post," is 3%.

Now, of course, we've sampled this window from our data set. So we know what comes next. We know, and that's the label, that the correct answer is that 3962 actually comes next in the sequence. So now we have this mathematical process for updating the neural network. We have a way of tuning it, and we're going to go into a little bit of detail shortly.

But basically, we know that this probability of 3% is too low; we want this probability to be higher, and we want the probabilities of all the other tokens to be lower. We have a method for mathematically calculating how to adjust and update the neural network so that the correct answer has a slightly higher probability. If I perform an update to the neural network, the next time I input this particular sequence of four tokens, the neural network will be slightly adjusted. It might determine that "post" has a probability of maybe 4%, "case" now maybe has 1%, and "direction" could become 2%, or something similar. Thus, we have a way of nudging, or slightly updating, the neural network to give a higher probability to the correct token that follows in the sequence.

And now you just have to remember that this process happens not just for this token here; these four fed in and predicted this one. This process occurs simultaneously for all of these tokens in the entire dataset. In practice, we sample little windows, little batches of windows, and at every single one of these tokens, we want to adjust our neural networks so that the probability of that token becomes slightly higher. This all happens in parallel in large batches of these tokens, and this is the process of training the neural network.

It's a sequence of updating it so that its predictions match the statistics of what actually happens in your training set. Its probabilities become consistent with the statistical patterns of how these tokens follow each other in the data. So let's now briefly get into the internals of these neural networks to give you a sense of what's inside. As I mentioned, we have these inputs that are sequences of tokens.

In this case, this is four input tokens, but this can be anywhere between zero up to, let's say, a thousand tokens. In principle, this can be an infinite number of tokens; it would just be too computationally expensive to process an infinite number of tokens. So, we crop it at a certain length, which becomes the maximum context length of that model. Now, these inputs \(x\) are mixed up in a giant mathematical expression together with the parameters or the weights of these neural networks. Here, I'm showing six example parameters and their settings, but in practice, these modern neural networks will have billions of these parameters.

And in the beginning, these parameters are completely randomly set. Now, with a random setting of parameters, you might expect that this neural network would make random predictions, and it does. Initially, the predictions are entirely random. However, through the process of iteratively updating the network—what we call training a neural network—the setting of these parameters is adjusted so that the outputs of our neural network become consistent with the patterns observed in our training set. You can think of these parameters as similar to knobs on a DJ set.

And as you're twiddling these knobs, you're getting different predictions for every possible token sequence input. Training a neural network simply means discovering a setting of parameters that seems to be consistent with the statistics of the training set. Now, let me just give you an example of this giant mathematical expression, just to give you a sense of it. Modern networks are massive expressions with probably trillions of terms. But let me just show you a simple example here.

It would look something like this. I mean, these are the kinds of expressions, just to show you that it's not very scary. We have inputs \( x \), like \( x_1 \), \( x_2 \), in this case, two example inputs, and they get mixed up with the weights of the network, \( w_0 \), \( w_1 \), \( w_2 \), \( w_3 \), etc. This mixing involves simple operations like multiplication, addition, exponentiation, division, etc. It is the subject of neural network architecture research to design effective mathematical expressions that have many convenient characteristics.

They are expressive, they are optimizable, they are parallelizable, etc. But at the end of the day, these are not complex expressions; basically, they mix up the inputs with the parameters to make predictions. We are optimizing the parameters of this neural network so that they come out consistent with the training set. Now, I would like to show you an actual production-grade example of what these neural networks look like. For that, I encourage you to visit this website that has a very nice visualization of one of these networks. This is what you will find on this website.

And this neural network, which is used in production settings, has a special kind of structure. This network is called the transformer. This particular one, as an example, has roughly 85,000 parameters. At the top, we take the inputs, which are the token sequences. Then, information flows through the neural network until the output, which is the logit softmax. These are the predictions for what comes next, meaning which token will come next.

And then, there is a sequence of transformations and all these intermediate values that are produced inside this mathematical expression as it predicts what comes next. As an example, these tokens are embedded into what is called a distributed representation. Every possible token has a vector that represents it inside the neural network. First, we embed the tokens, and then those values flow through this diagram. All of these are very simple mathematical expressions individually, including layer norms, matrix multiplications, softmax functions, and so on.

So here's kind of like the attention block of this transformer, and then information flows through into the multilayer perceptron block, and so on. All these numbers here represent the intermediate values of the expression, and you can almost think of these as the firing rates of these synthetic neurons. However, I would caution you not to think of it too much like neurons, because these are extremely simple compared to the neurons you would find in your brain. Your biological neurons are very complex dynamical processes that have memory and so on. There's no memory in this expression.

It's a fixed mathematical expression from input to output with no memory; it's just stateless. These are very simple neurons in comparison to biological neurons, but you can still loosely think of this as a synthetic piece of brain tissue, if you'd like to consider it that way. Information flows through, and all these neurons fire until we reach the predictions. Now, I'm not going to dwell too much on the precise mathematical details of all these transformations.

Honestly, I don't think it's that important to get into. What's really important to understand is that this is a mathematical function. It is parameterized by some fixed set of parameters, like, say, 85,000 of them, and it is a way of transforming inputs into outputs. As we adjust the parameters, we get different kinds of predictions. We then need to find a good setting of these parameters so that the predictions align with the pattern seen in the training set.

We are getting different kinds of predictions, and then we need to find a good setting of these parameters so that the predictions sort of match up with the patterns seen in the training set. So, that's the transformer. Okay, I've shown you the internals of the neural network, and we talked a bit about the process of training it. I want to cover one more major stage of working with these networks, and that is the stage called inference.

In inference, what we're doing is generating new data from the model. We want to examine the patterns that it has internalized in the parameters of its network. Generating from the model is relatively straightforward. We start with some tokens that serve as a prefix, which represents what we want to initiate with. For example, let's say we want to start with the token 91.

Well, we feed it into the network. And remember that the network gives us probabilities, right? It provides us with this probability vector here. So what we can do now is essentially flip a biased coin. We can sample a token based on this probability distribution.

So the tokens that are given a high probability by the model are more likely to be sampled when you flip this biased coin. You can think of it that way. We sample from the distribution to get a single unique token. For example, token 860 could come next when we're generating from the model.

Now, 860 is a relatively likely token. It might not be the only possible token in this case; there could be many other tokens that could have been sampled. But we can see that 860 is a relatively likely token, as an example. Indeed, in our training example here, 860 does follow. So, let's now say that we continue the process.

So after 91, there's a 60. We append it, and we again ask, what is the third token? Let's sample, and let's just say that it's 287, exactly as here. Let's do that again. We come back in. Now we have a sequence of three, and we ask, what is the likely fourth token?

And we sample from that and get this one. Now, let's say we do it one more time. We take those four, we sample, and we get this one. This 13,659 is not actually 3,962 as we had before. Instead, this token is the token for "article." So, we are viewing a single article. In this case, we didn't exactly reproduce the sequence that we saw here in the training data.

So, keep in mind that these systems are stochastic. They involve sampling and flipping coins. Sometimes we get lucky and reproduce a small chunk of the text from the training set, but other times we generate a token that was not verbatim part of any of the documents in the training data. As a result, we get something akin to remixes of the data we encountered during training, because at each step, we can flip and select a slightly different token. Once that token is chosen, if you sample the next one and continue the process, you quickly begin to generate token streams that are quite different from the token streams found in the training documents. Statistically, they will have similar properties, but they are not identical to the training data.

They're kind of inspired by the training data. In this case, we received a slightly different sequence. Why would we get "article"? You might imagine that "article" is a relatively likely token in the context of "bar," "viewing," "single," etc. It's also conceivable that the word "article" followed this context window somewhere in the training documents to some extent.

And we just happened to sample it here at that stage. So basically, inference is just predicting from these distributions one at a time. We continue feeding back tokens and getting the next one. We're always flipping these coins, and depending on how lucky or unlucky we get, we might observe very different kinds of patterns based on how we sample from these probability distributions. So that's inference.

In most common scenarios, downloading the internet and tokenizing it is a preprocessing step. You do that a single time. Once you have your token sequence, you can start training networks. In practical cases, you would try to train many different networks with various settings, arrangements, and sizes. Therefore, you'd be involved in a lot of neural network training.

And then, once you have a neural network and you train it, and you have a specific set of parameters that you're happy with, you can take the model and perform inference. This allows you to generate data from the model. When you're using ChatGPT and conversing with a model, that model was trained by OpenAI many months ago, probably. It has a specific set of weights that work well. When you're interacting with the model, all of that is just inference. There is no further training involved. Those parameters are held fixed, and you're essentially engaging with the model.

You're providing it with some tokens, and it's completing token sequences. That's what you observe being generated when you actually use the model. You're giving it some tokens, and it's completing token sequences, which is what you see generated when you use the model on ChatGPT. So, that model does more than just inference alone.

So, let's now look at an example of training an inference that is kind of concrete and gives you a sense of what this actually looks like when these models are trained. The example that I would like to work with, and that I'm particularly fond of, is that of OpenAI's GPT-2. GPT stands for Generative Pre-trained Transformer, and this is the second iteration of the GPT series by OpenAI. When you are talking to ChatGPT today, the model that underlies all of the magic of that interaction is GPT-4, the fourth iteration of that series. Now, GPT-2 was published in 2019 by OpenAI in this paper that I have right here.

And the reason I like GPT-2 is that it is the first time a recognizably modern stack came together. All of the pieces of GPT-2 are recognizable today by modern standards; it's just that everything has gotten bigger. Now, I'm not going to be able to go into the full details of this paper, of course, because it is a technical publication. They are recognizable today by modern standards.

It's just that everything has gotten bigger. Now, I'm not going to be able to go into the full details of this paper, of course, because it is a technical publication, but some of the details I would like to highlight are as follows. GPT-2 was a transformer neural network, just like the neural networks you would work with today. It had 1.6 billion parameters, right? So these are the parameters that we looked at here.

It would have 1.6 billion of them. Today, modern transformers would have a lot closer to a trillion or several hundred billion, probably. The maximum context length here was 1,024 tokens. So, when we are sampling chunks of windows of tokens from the dataset, we're never taking more than 1,024 tokens. Therefore, when you are trying to predict the next token in a sequence, you will never have more than 1,024 tokens in your context in order to make that prediction.

Now, this is also tiny by modern standards. Today, the token and context lengths would be a lot closer to a couple of hundred thousand or maybe even a million. This means you would have a lot more context and significantly more tokens in history, allowing for much better predictions of the next token in a sequence. Finally, GPT-2 was trained on approximately 100 billion tokens, which is also fairly small by modern standards.

As I mentioned, the FineWeb dataset that we looked at here has 15 trillion tokens, so 100 billion is quite small. Now, I actually tried to reproduce GPT-2 for fun as part of a project called lm.c. You can see my write-up of that in this post on GitHub in the lm.c repository. In particular, the cost of training GPT-2 in 2019 was estimated to be approximately $40,000. But today, you can do significantly better than that.

And, in particular, it took about one day and approximately $600. However, this wasn't even a significant effort. I believe you could really reduce this to about $100 today. Now, why have the costs decreased so much? Well, first, these data sets have become much larger, and the methods we use to filter, extract, and prepare them have become much more refined.

And so, the dataset is of a much higher quality. That’s one thing. But really, the biggest difference is that our computers have gotten much faster in terms of hardware, and we'll look at that in a second. Additionally, the software used to run these models has greatly improved, allowing us to maximize speed from the hardware as much as possible. As everyone has focused on these models and tried to run them very quickly, this software has also evolved significantly. Now, I won't be able to delve into the full details of this GPT-2 reproduction, as this is a long technical post. However, I would like to provide you with an intuitive sense of what it looks like to actually train one of these models as a researcher. What are you looking at, and what does the process look like?

What does it feel like? Let me give you a sense of that a little bit. Okay, so this is what it looks like. Let me slide this over. What I'm doing here is training a GPT-2 model right now.

And what's happening here is that every single line, like this one, represents one update to the model. Remember that we are essentially making the prediction better for each of these tokens, and we are updating the weights or parameters of the neural network. Therefore, every single line is one update to the neural network, where we change its parameters slightly so that it is better at predicting the next token in the sequence. In particular, every single line here is improving the prediction for 1 million tokens in the training set. We have essentially taken 1 million tokens from this dataset and tried to enhance the prediction of each token as the next in a sequence across all 1 million of them simultaneously.

At every single one of these steps, we are making an update to the network. The number to watch closely is called "loss." The loss is a single number that indicates how well your neural network is performing at that moment. It is designed so that a low loss is good. You will notice that the loss is decreasing as we make more updates to the neural network, which corresponds to making better predictions on the next token in the sequence.

And so the loss is the number that you are watching as a neural network researcher. You are kind of waiting, twiddling your thumbs, drinking coffee, and making sure that this looks good so that with every update, you are improving and the network is getting better at prediction. Now, here you see that we are processing 1 million tokens per update.

Each update takes about 7 seconds, roughly. Here, we are going to process a total of 32,000 steps of optimization. So, 32,000 steps with 1 million tokens each is about 33 billion tokens that we will process. Currently, we are only at step 420 out of 32,000. Therefore, we are still just a bit more than 1% done, as I have only been running this for 10 or 15 minutes or so.

Now, every 20 steps, I have configured this optimization to perform inference. So what you're seeing here is the model predicting the next token in a sequence. You start it randomly, and then you continue feeding in the tokens. We are currently running this inference step, and this is the model predicting the next token in the sequence.

And every time you see something appear, that's a new token. So let's just look at this, and you can see that this is not yet very coherent. Keep in mind that this is only 1% of the way through training, and so the model is not yet very good at predicting the next token in the sequence. What comes out is actually somewhat gibberish, right? But it still has a little bit of local coherence.

Here’s a reformatted version with correct capitalization and spelling: "So, since she is mine, it's part of the information; I should discuss my father, the great companions Gordon showed me while sitting over it, and so on. I know it doesn't look very clear, but let's actually scroll up and see what it looked like when I started the optimization. All the way back here at step one. After 20 steps of optimization, you can see that what we're getting here looks completely random. This is, of course, because the model has only had 20 updates to its parameters."

And so, it's generating random text because it's a random network. You can see that, at least in comparison to this, the model is starting to perform much better. Indeed, after completing the entire 32,000 steps, the model will have improved to the point where it's generating fairly coherent English, and the token streams are correct, making up English much more effectively. This has to run for about a day or two more, and at this stage, we just need to ensure that the loss is decreasing. Everything is looking good, and we just have to wait.

And now let me turn to the story of the computation that's required. Because, of course, I'm not running this optimization on my laptop; that would be way too expensive. We have to run this neural network, and we have to improve it. We need all this data and so on. So, you can't run this effectively on your computer because the network is just too large.

So, all of this is running on the computer that is out there in the cloud. I want to address the computing aspect of training these models and what that looks like. Okay, so the computer that I'm using for this optimization is an 8xH100 node. This means there are eight H100s in a single node or a single computer. I am renting this computer, and it is located somewhere in the cloud. I'm not sure where it is physically, actually.

The place I like to rent from is called Lambda, but there are many other companies that provide this service. When you scroll down, you can see that they have some on-demand pricing for computers that have these H100s, which are GPUs. I’m going to show you what they look like in a second. For example, on-demand, you can rent eight NVIDIA H100 GPUs, and this machine costs $3 per GPU per hour. You can rent these, and then you get a machine in the cloud where you can train your models. These GPUs look like this.

So, this is one H100 GPU. This is kind of what it looks like. You slot this into your computer, and GPUs are a perfect fit for training neural networks because they are very computationally expensive but exhibit a lot of parallelism in the computations. This means you can have many independent workers working simultaneously to solve the matrix multiplications that are essential for training these neural networks. This is just one of these H100s, but actually, you would use multiple of them together. You could stack eight of them into a single node, and then you can stack multiple nodes to create an entire data center or system. So, when we look at a data center, we start to see things that look like this.

Right? So we have one GPU that scales to eight GPUs, which then connects to a single system, and eventually expands to many systems. These form the larger data centers, which are, of course, much more expensive. What's happening is that all the big tech companies really desire these GPUs to train their language models because they are incredibly powerful. This demand is fundamentally what has driven Nvidia's stock price to reach $3.4 trillion today, for example, and explains why Nvidia has experienced such significant growth.

So, this is the Gold Rush. The Gold Rush involves acquiring GPUs, gathering enough of them so they can collaborate to perform this optimization. What are they all doing? They are collaborating to predict the next token on a dataset like the FineWeb dataset. This computational workflow is extremely expensive.

The more GPUs you have, the more tokens you can try to predict and improve upon. You'll process this dataset faster, allowing you to iterate more quickly and develop a larger network, and to train a bigger network, and so on. This is what all those machines are doing, and this is why it is such a big deal. For example, this is an article from about a month ago, which highlights why it's significant that Elon Musk is acquiring 100,000 GPUs in a single data center.

And all of these GPUs are extremely expensive, will consume a tremendous amount of power, and are all focused on predicting the next token in the sequence to improve the network's performance. As a result, they likely generate much more coherent text than what we're seeing here, and at a much faster pace. Unfortunately, I do not have a couple of tens or hundreds of millions of dollars to spend on training a really large model like this. However, we are fortunate to have big tech companies that routinely train these models and release some of them once the training is complete. These companies have invested a significant amount of computational resources to train these networks, and they release the models at the end of the optimization process, making it very useful since they have done a lot of computation on our behalf.

There are many companies that routinely train these models, but not many of them release what are called base models. The model that comes out at the end is referred to as a base model. What is a base model? It is a token simulator, specifically an internet text token simulator. By itself, it is not yet useful, because what we want is something referred to as an assistant.

We want to ask questions and have it respond with answers. These models won't do that. They just create sort of remixes of the internet; they generate internet pages. Therefore, the base models are not very often released because they are only a first step among several others that we still need to take in order to develop an assistant.

However, a few releases have been made. For example, the GPT-2 model released the 1.5 billion parameter model back in 2019. The GPT-2 model is a base model. Now, what is a model release? What does it look like to release these models? This is the GPT-2 repository on GitHub.

Well, you need two things basically to release a model. Number one, we need the Python code, which usually describes the sequence of operations that are performed in the model. If you remember back to this transformer, the sequence of steps taken in this neural network is detailed in this code. This code implements what's known as the forward pass of the neural network. Therefore, we need the specific details of exactly how they wired up that neural network.

So, this is just computer code, and it's usually just a couple of hundred lines of code. It's not that crazy, and this is all fairly understandable and usually quite standard. What's not standard are the parameters. That's where the actual value lies. What are the parameters of this neural network?

Because there are 1.6 billion of them, we need the correct settings or a really good setting. That's why, in addition to this source code, they release the parameters, which in this case is roughly 1.5 billion parameters. These are just numbers; one single instance has roughly 1.5 billion parameters.

So it's one single list of 1.5 billion numbers, the precise and good setting of all the knobs such that the tokens come out well. You need those two things to get a base model release. Now, GPT-2 was released, but that's actually a fairly old model, as I mentioned. The model we're going to turn to is called Llama 3, and that's the one I would like to show you next. So, Llama 3, similar to GPT-2, has 1.6 billion parameters trained on 100 billion tokens.

LLaMA3 is a much bigger and more modern model. It was released and trained by Meta, and it consists of 405 billion parameters trained on 15 trillion tokens. It operates in a similar way to its predecessors, but on a much larger scale. Meta has also made a release regarding LLaMA3, which was part of an accompanying paper. This paper provides extensive details about the largest base model they released, which is the LLaMA 3.1, with its 405 billion parameters. This is the base model, and in addition to it, there is foreshadowing for later sections of the video, indicating that they also released an instruct model.

And the instruction means that this is an assistant. You can ask it questions, and it will give you answers. We will cover that part later. For now, let's just look at this base model, this token simulator, and let's play with it. Let's try to think about what this thing is, how it works, and what we get at the end of this optimization if we let it run until the end for a very big neural network with a lot of data.

So, my favorite place to interact with the base models is this company called Hyperbolic, which is basically serving the base model of the 405B LLaMA 3.1. When you go to the website, you might need to register and so on. Make sure that, in the models, you are using LLaMA 3.1 405 billion base—it must be the base model. Additionally, the max tokens indicate how many tokens we are going to generate, so let's decrease this a bit to avoid wasting compute.

We just want the next 128 tokens and leave the other stuff alone. I'm not going to go into the full detail here. Now, fundamentally, what's going to happen here is identical to what happens during inference for us. So this is just going to continue the token sequence of whatever prefix you're going to give it. I want to first show you that this model here is not yet an assistant.

So you can, for example, ask it, "What is 2 plus 2?" It's not going to respond with, "Oh, it's 4. What else can I help you with?" It's not going to do that because "What is 2 plus 2?" is going to be tokenized, and those tokens will act as a prefix. Then the model will calculate the probability for the next token, essentially functioning as a glorified autocomplete.

It's a very, very expensive autocomplete of what comes next, depending on the statistics of what it saw in its training documents, which are basically web pages. So let's just hit enter to see what tokens it comes up with as a continuation. Okay, so here it kind of actually answered the question and started to go off into some philosophical territory. Let's try it again. So let me copy and paste, and let's try again from scratch. What is 2 plus 2?

Okay, so it goes off again. I want to stress one more thing: the system, I think, starts from scratch every time you put it in. The system here is stochastic. For the same prefix of tokens, we're always getting a different answer. The reason for that is that we receive a probability distribution and sample from it, which leads to different samples each time. As a result, we often end up exploring different territory afterward. In this case, I'm unsure what this is.

Let's try one more time. So it just continues on. It’s just doing the stuff that is available on the internet, right? It’s just kind of like a regular thing, regurgitating those statistical patterns. So, first things first: it’s not an assistant yet.

It's a token autocomplete. Second, it is a stochastic system. The crucial thing is that, even though this model is not yet very useful for a lot of applications, it is still valuable because, in the task of predicting the next token in the sequence, the model has learned a lot about the world and has stored all that knowledge in the parameters of the network. So remember that our text looked like this, right? Internet web pages.

And now all of this is sort of compressed in the weights of the network. You can think of these 405 billion parameters as a kind of compression of the internet. You can consider the 405 billion parameters as similar to a zip file; however, it's not a lossless compression—it's a lossy compression. We are left with a sort of gestalt of the internet from which we can generate information. Now, we can elicit some of this knowledge by prompting the base model accordingly.

So, for example, here's a prompt that might work to elicit some of that knowledge that's hiding in the parameters. Here's my top 10 list of the top landmarks to see in Paris. I'm doing it this way because I'm trying to prime the model to continue this list. Let's see if that works when I press enter. Okay, it started the list, and it's now providing some of those landmarks.

And now notice that it’s trying to provide a lot of information here. You might not be able to fully trust some of the information presented. Remember that this is all just a recollection of various internet documents. Consequently, things that occur very frequently in the internet data are likely to be remembered more accurately compared to those that happen infrequently. Therefore, you can’t fully trust some of the information here, as it is merely a vague recollection of internet documents and is not stored explicitly in any of the parameters.

It's all just a recollection. That said, we did get something that is probably approximately correct, and I don't actually have the expertise to verify that this is roughly accurate. However, you can see that we've elicited a lot of the knowledge from the model, and this knowledge is not precise and exact. It is vague, probabilistic, and statistical. The kinds of things that occur often are more likely to be remembered in the model.

Now I want to show you a few more examples of this model's behavior. The first thing I want to show you is this example. I went to the Wikipedia page for "zebra," and let me just copy-paste the first sentence here. Now, when I click enter, what kind of completion are we going to get?

So, let me just hit enter. There are three living species, etc., etc. What the model is producing here is an exact regurgitation of this Wikipedia entry. It is reciting this Wikipedia entry purely from memory, and this memory is stored in its parameters. It is possible that, at some point within these 512 tokens, the model will stray away from the Wikipedia entry.

But you can see that it has huge chunks of it memorized here. Let me see, for example, if this sentence occurs by now. Okay, so we're still on track. Let me see, for example, if this sentence occurs by now. Okay, so we're still on track.

Let me check here. On track, it will eventually stray away. Okay, so this thing is just recited to a very large extent. It will eventually deviate because it won't be able to remember exactly. Now, the reason that this happens is that these models can be extremely good at memorization, and usually, this is not what you want in the final model.

And this is something called regurgitation. It is usually undesirable to cite things directly that you have been trained on. The reason this happens is that for many documents, such as Wikipedia, when these documents are deemed to be of very high quality as a source, it is often the case that when training the model, it will preferentially sample from those sources. Essentially, the model has likely gone through several epochs on this data, meaning it has seen this webpage multiple times, perhaps around ten times or so. It's somewhat similar to you; if you read a text many times, say a hundred times, you will be able to recite it. The same principle applies to this model.

If it sees something way too often, it's going to be able to recite it later from memory. However, these models can be a lot more efficient per presentation than a human. So, it is probable that it has only seen this Wikipedia entry ten times, but essentially, it has remembered this article exactly in its parameters. Okay, the next thing I want to show you is something that the model has definitely not seen during its training. For example, if we go to the paper and then navigate to the pre-training data, we will see here that the dataset has a knowledge cutoff at the end of 2023, meaning it will not have seen documents after this point.

And certainly, it has not seen anything about the 2024 election and how it turned out. Now, if we prime the model with the tokens from the future, it will continue the token sequence and make its best guess according to the knowledge it has in its own parameters. So, let's take a look at what that could look like. The Republican Party nominated Trump, okay, President of the United States from 2017. And let's see what it says after this point.

So, for example, the model will have to guess the running mate and who they're running against, etc. Let's hit enter. Here are some scenarios: Mike Pence was the running mate instead of J.D. Vance, and the ticket was against Hillary Clinton and Tim Kaine.

So, this is kind of an interesting parallel universe, potentially, of what could have happened according to the alarm. Let's get a different sample. So, with the identical prompt, let's resample. Here, the running mate was Ron DeSantis, and they ran against Joe Biden and Kamala Harris. This is, again, a different parallel universe.

So the model will take educated guesses and continue the token sequence based on this knowledge. What we're seeing here is referred to as a hallucination; the model is simply making its best guess in a probabilistic manner. The next thing I would like to show you is that even though this is a base model and not yet an assistant model, it can still be utilized in practical applications if you are clever with your prompt design. For example, here’s something we would call a few-shot prompt. I have ten words, or ten pairs, with each pair consisting of an English word followed by its translation in Korean. We have ten of these pairs. At the end, we have "Teacher:" followed by where we will request a completion, say, just five times. These models possess what we call in-context learning abilities. This refers to the model's capacity to identify and learn patterns within the given context, as it processes the data.

And it knows to continue that pattern. This is referred to as in-context learning. It takes on the role of a translator, and when we hit completion, we see that the teacher's translation is correct. This is how you can build apps by being clever with your prompting, even though we still only have a base model for now. It relies on what we call in-context learning ability, which is achieved by constructing what's known as a few-shot prompt.

Okay, and finally, I want to show you that there is a clever way to actually instantiate a whole language model assistant just by prompting. The trick to it is that we're going to structure a prompt to look like a webpage that features a conversation between a helpful AI assistant and a human. The model will then continue that conversation. To write the prompt, I turned to ChatGPT itself, which is kind of meta. I told it that I want to create an OLM assistant, but all I have is the base model.

So, can you please write my prompt? This is what it came up with, which is actually quite good. Here’s a conversation between an AI assistant and a human. The AI assistant is knowledgeable, helpful, and capable of answering a wide range of IT questions, etc. Additionally, it’s not enough to just provide a description.

It works much better if you create this few-shot prompt. So here are a few terms for the human assistant. We have several conversation terms, and then at the end, we will put the actual query that we would like. Let me copy and paste this into the base model prompt.

And now let me create the human column. This is where we put our actual prompt: "Why is the sky blue?" Now, let's run it. Assistant: The sky appears blue due to the phenomenon called Rayleigh scattering.

Etc. So you see that the base model is just continuing the sequence, but because the sequence resembles this conversation, it takes on that role. However, it's a bit subtle because here it just ends with the assistant and then hallucinates the next question by the human, etc. It will just continue going on and on. But you can see that we have accomplished a task. If you just took this, "Why is the sky blue?"

And if we just refresh this and put it here, then, of course, we don't expect this to work with a base model. Who knows what we're going to get? We're just going to get more questions. So this is one way to create an assistant, even though you may only have a base model. This is a brief summary of the process, even if you only have a base model.

Okay, so this is a brief summary of the things we talked about over the last few minutes. Let me zoom out here. This is what we've discussed so far: We wish to train LLM assistants like ChatGPT. We've covered the first stage of that, which is the pre-training stage. It essentially boils down to taking internet documents, breaking them up into tokens—these atoms of small text chunks—and then predicting token sequences using neural networks.

The output of this entire stage is this base model. It involves the setting of the parameters for this network. This base model is essentially an internet document simulator at the token level, allowing it to generate token sequences that exhibit the same statistical patterns as internet documents. We observed that we could utilize it in some applications, but we need to improve further. Our goal is to create an assistant.

We want to be able to ask questions, and we want the model to give us answers. Therefore, we need to move into the second stage, which is called the post-training stage. We take our base model, our internet document simulator, and hand it off to post-training. Now, we will discuss a few ways to perform the post-training of these models. These stages in post-training will be computationally much less expensive.

Most of the computational work, all of the massive data centers, and all of the heavy compute, which involve millions of dollars, are part of the pre-training stage. However, we are now moving into a slightly cheaper, yet still extremely important stage called post-training. In this stage, we transform the large language model (LLM) into an assistant. Let's take a look at how we can adjust our model so that it doesn't just sample internet documents but can provide answers to questions. In other words, we want to start thinking about conversations. These conversations can be multi-turn, meaning there can be multiple exchanges, and they generally involve a dialogue between a human and an assistant.

And so, for example, we can imagine the conversation could look something like this. When a human says, "What is two plus two?" the assistant should respond with something like, "Two plus two is four." When a human follows up and says, "Star" instead of "plus," the assistant could respond with something like this. Similarly, this is another example showing that the assistant could also have some kind of personality, making it seem nice. In the third example, I'm illustrating that when a human asks for something we don't wish to help with, we can produce what's called a refusal.

We can say that we cannot help with that. In other words, what we want to do now is to think through how an assistant should interact with a human, and we want to program the assistant's behavior in these conversations. Since this involves neural networks, we won't be programming these explicitly in code. We won't be able to program the assistant in that traditional way. Instead, everything is done through neural network training on datasets.

And so, because of that, we are going to be implicitly programming the assistant by creating datasets of conversations. These are three independent examples of conversations in a dataset. An actual dataset, which I will show you examples of, will be much larger. It could contain hundreds of thousands of multi-turn conversations that are very long and cover a diverse range of topics. Here, I am only showing three examples. The way this works is that the assistant is being programmed by example.

And where is this data coming from? For example, two times two equals four, which is the same as two plus two, and so on. Where does this information originate? This comes from human labelers. We provide them with some conversational context and ask them to provide the ideal assistant response for that situation.

A human will write out the ideal response for an assistant in any situation. Then, we will have the model train on this and imitate those types of responses. The way this works is as follows: we will take our base model, which we produced in the pre-training stage. This base model was trained on internet documents. Now, we will discard that dataset of internet documents and substitute it with a new dataset, which will consist of conversations.

And we're going to continue training the model on these conversations, using this new dataset of conversations. What happens is that the model will very rapidly adjust and learn the statistics of how this assistant responds to human queries. Later, during inference, we will be able to prime the assistant and obtain the response, mimicking what the human labelers would do in that situation, if that makes sense. We're going to see examples of that, and this will become a bit more concrete. I also wanted to mention that during this post-training stage, we will continue training the model, but the pre-training stage can, in practice, take roughly three months of training on many thousands of computers.

The post-training stage will typically be much shorter, lasting about three hours, for example. This is because the data set of conversations that we are going to create manually is much smaller than the data set of text available on the internet. Therefore, this training will be very short. Fundamentally, we will take our base model and continue training using the exact same algorithm and everything else, except we will be swapping out the data set for conversations. Now, the questions are: Where are these conversations? How do we represent them? How do we get the model to interpret conversations instead of just raw text? What are the outcomes of this kind of training? Additionally, what insights can we gain in a psychological sense when we discuss the model? So, let's turn our attention to those questions. First, let's start by discussing the tokenization of conversations.

Everything in these models has to be turned into tokens because it's all about token sequences. So, how do we convert conversations into token sequences? That is the question. To achieve this, we need to design some form of encoding. This is somewhat similar to the TCP/IP packet on the internet, where there are precise rules and protocols for representing information. Everything is structured in such a way that all this data is organized clearly, allowing for a common understanding among everyone involved.

And so it's the same thing now happening in LLMs. We need some kind of data structures, and we need to have some rules about how these data structures, like conversations, get encoded and decoded to and from tokens. I want to show you how I would recreate this conversation in the token space. If you go to TechTokenizer, I can recreate this conversation in the token space. This is how it is represented for the language model.

So here we are, iterating a user and an assistant in this two-turn conversation. What you're seeing may look messy, but it's actually relatively simple. The process of converting it into a token sequence at the end is somewhat complicated, but ultimately, this conversation between the user and assistant consists of 49 tokens. It forms a one-dimensional sequence of 49 tokens. And these are the tokens, okay?

And all the different LLMs will have a slightly different format or protocols, and it's a little bit of a Wild West right now. For example, GPT-4O does it in the following way: You have this special token called "IM_START," which is short for "Imaginary Monologue Start." Then, you have to specify—I'm not actually sure why it's called that, to be honest—whose turn it is. For example, "USER," which is represented by the token "1 4 2 8." After that, you have internal monologues separated, and then it's the exact question—the tokens of the question. Finally, you have to close it with "IM_END," marking the end of the imaginary monologue. So basically, the question from a user, "What is 2 plus 2?" ends up being the token sequence of these tokens.

And now the important thing to mention here is that "I am start" is not text, right? "I am start" is a special token that gets added. It's a new token that has never been trained on before. It is a new token that we create in a post-training stage and introduce. Therefore, these special tokens, like "I am sep," "I am start," etc., play a critical role.

They are introduced and interspersed with text so that they help the model learn that, "Hey, this is the start of a turn for..." Who is it the start of a turn for? The start of the turn is for the user. Then, this is what the user says, and then the user ends. After that, it's a new start of a turn.

And it is by the assistant. And then what does the assistant say? Well, these are the tokens of what the assistant says, etc. And so this conversation is now turned into this sequence of tokens. The specific details here are not actually that important. All I'm trying to show you in concrete terms is that our conversations, which we think of as kind of like a structured object, end up being turned via some encoding into one-dimensional sequences of tokens.

And so, because this is a one-dimensional sequence of tokens, we can apply all the methods we used before. Now, it's just a sequence of tokens, and we can train a language model on it. We are predicting the next token in the sequence, just like before, and we can represent and train on conversations. Now, what does it look like at test time during inference? Suppose we've trained a model on datasets of conversations, and now we want to perform inference. What does this look like when you're using ChatGPT?

Well, you come to ChatGPT, and you have, say, a dialogue. The way this works is basically like this: if this was already filled in, you might ask, "What is 2 plus 2?" The answer is 4. Now, you might issue a question like "What if it was multiplied instead?" What basically ends up happening on the servers of OpenAI is that they input "I'm start assistant," "I'm sep," and this is where they end it, right here.

So they construct this context, and now they start sampling from the model. At this stage, they will go to the model and ask, "What is a good first token? What is a good second token? What is a good third token?" This is where the LLM takes over and creates a response.

Here's a reformatted version of your text with correct capitalization and spelling: For example, a response might look something like this. It doesn’t have to be identical, but it will have a similar flavor if this kind of conversation was included in the dataset. That’s roughly how the protocol works, although the specific details are not important. My goal is to demonstrate that everything ultimately becomes a one-dimensional token sequence. Therefore, we can apply everything we've already learned, but we are now training on conversations and also generating conversations as well.

Okay, so now I would like to turn to what these data sets look like in practice. The first paper that I would like to show you and the first effort in this direction is a paper from OpenAI in 2022. This paper is called InstructGPT, referring to the technique they developed. It was the first time that OpenAI discussed how to take language models and fine-tune them on conversations. This paper contains a number of details that I would like to take you through.

The first stop I would like to make is in Section 3.4, where they discuss the human contractors they hired, in this case from Upwork or through Scale AI, to construct these conversations. There are human labelers involved whose professional job it is to create these conversations. These labelers are asked to come up with prompts and then to also complete the ideal assistant responses. These are the kinds of prompts that people came up with; they are human labelers.

Sure! Here’s the reformatted version: "Please list five ideas for how to regain enthusiasm for my career. What are the top 10 science fiction books I should read next? There are many different types of prompts here. For example, translate this sentence to Spanish, etc. There are several suggestions that people have come up with."

They first come up with the prompt, and then they also answer that prompt, providing the ideal assistant response. How do they know what the ideal assistant response is that they should write for these prompts? When we scroll down a little further, we see an excerpt of labeling instructions given to the human labelers. The company developing the language model, such as OpenAI, writes labeling instructions for how humans should create ideal responses. Here, for example, is an excerpt of these labeling instructions. At a high level, you are asking people to be helpful, truthful, and harmless. You can pause the video if you'd like to see more here.

But at a high level, basically just answer, try to be helpful, try to be truthful, and avoid answering questions that we don’t want the system to handle later in ChatGPT. Roughly speaking, the company develops the labeling instructions. Usually, they are not this short; typically, they are hundreds of pages long, and people have to study them professionally. Then, they write out the ideal assistant responses following those labeling instructions.

So, this is a very human-heavy process, as it was described in this paper. Now, the dataset for InstructGPT was never actually released by OpenAI, but we do have some open-source reproductions that were trying to follow this kind of setup and collect their own data. One example I’m familiar with is the effort of Open Assistant from a while back. This is just one of many examples, but I want to highlight it. These are people on the internet who were asked to create conversations similar to what OpenAI did with human labelers.

And so, here's an entry from a person who came up with this prompt: "Can you write a short introduction to the relevance of the term 'monopsony' in economics? Please use examples, etc." Following this, the assistant will provide a response.

And so then the same person, or a different person, will actually write out this ideal response. This is an example of how the conversation could continue. Now, explain it to a dog, and then you can try to come up with a slightly simpler explanation or something like that. This then becomes the label, and we end up training on this. During training, we, of course, won't have full coverage of all the possible questions that the model will encounter at test time during inference.

We can't possibly cover all the potential prompts that people will be asking in the future. However, if we have a dataset of a few of these examples, then the model, during training, will begin to take on the persona of a helpful, truthful, and harmless assistant. It's all programmed by example. These are all instances of behavior. If you have conversations that exhibit these example behaviors, and you gather enough of them—like 100,000—the model starts to understand the statistical patterns and adopts the personality of this assistant. Now, it's possible that when you get the exact same question during testing, the answer may be recited exactly as it was in the training set. However, more likely, the model will produce a response that captures a similar vibe and understands the type of answer you are seeking.

So that's what we're doing. We're programming the system by example, and the system adopts this persona of a helpful, truthful, and harmless assistant, which is reflected in the labeling instructions that the company creates. Now, I want to show you that the state of the art has advanced in the last two or three years since the InstructGPT paper. In particular, it's not very common for humans to be doing all the heavy lifting on their own anymore. This is because we now have language models, and these language models are helping us create datasets and conversations.

So it is very rare for people to literally write out a response from scratch. It is much more likely that they will use an existing response to come up with an answer, and then edit it or modify it. There are many different ways in which large language models (LLMs) have started to permeate this post-training stack. LLMs are basically used pervasively to help create these massive data sets of conversations. I want to highlight UltraChat as one example of a more modern data set of conversations.

It is to a very large extent synthetic, but I believe there's some human involvement; I could be wrong about that. Usually, there will be a little bit of human input, but there will be a huge amount of synthetic assistance. This is all constructed in different ways, and UltraChat is just one example of many SFT datasets that currently exist. The only thing I want to show you is that these datasets now contain millions of conversations. These conversations are mostly synthetic, but they're likely edited to some extent by humans.

And they span a huge diversity of areas, and so on. These are fairly extensive artifacts by now, and there are all these SFT mixtures, as they're called. So you have a mixture of many different types and sources, and it's partially synthetic and partially human. It's kind of evolved in that direction since. But roughly speaking, we still have SFT datasets, and they are made up of conversations.

We're training on them just like we did before. And I guess the last thing to note is that I want to dispel a little bit of the magic of talking to an AI. When you go to ChatGPT and you give it a question, then hit enter, what comes back is statistically aligned with what's in the training set. These training sets really just consist of humans following labeling instructions. So what are you actually talking to in ChatGPT? Or how should you think about it?

Well, it's not coming from some magical AI, broadly speaking. It’s coming from something that is imitating human labelers, based on labeling instructions written by these companies. So, you're kind of imitating this; it's almost as if you're asking a human labeler. Imagine that the answer given to you from ChatGPT is some kind of simulation of a human labeler. It’s like asking what a human labeler would say in this kind of conversation.

And it's not just that the human labeler is a random person from the internet; these companies actually hire experts. For example, when you are asking questions about code and similar topics, the human labelers involved in the creation of these conversation datasets are usually educated and skilled individuals. You are, in a way, asking a question of a simulation of those experts, if that makes sense. So you're not talking to a magical AI; you're interacting with an average labeler. This average labeler is probably fairly highly skilled, but you're essentially communicating with an instantaneous simulation of the type of person that would be hired in the construction of these datasets.

So let me give you one more specific example before we move on. For instance, when I go to ChatGPT and ask, "Recommend the top five landmarks you see in Paris," and then I hit enter. Okay, here we go. When I hit enter, what's coming out here? How do I think about it? Well, it's not some kind of magical AI that has gone out and researched all the landmarks and then ranked them using its infinite intelligence, etc.

What I'm getting is a statistical simulation of a labeler that was hired by OpenAI. You can think about it roughly this way. If this specific question is in the post-training dataset somewhere at OpenAI, then I'm very likely to see an answer that is probably very, very similar to what that human labeler would have put down for those five landmarks. How does the human labeler come up with this? Well, they go off and explore the internet, doing their own research for about 20 minutes, and then they come up with a list.

Now, if they come up with this list and this is in the dataset, I’m very likely to see what they submitted as the correct answer from the assistant. However, if this specific query is not part of the post-training dataset, then what I'm encountering here is a little more emergent because the model understands this statistically. The kinds of landmarks in a streaming set are usually the prominent landmarks—those that people typically want to see and that are frequently discussed on the internet. Remember that the model already possesses a wealth of knowledge from its pre-training on the internet. Therefore, it has likely seen numerous conversations about pairs, landmarks, and the types of things that people enjoy seeing.

And so it's the pre-training knowledge that has been combined with the post-training dataset that results in this kind of imitation. So that's roughly how you can think about what's happening behind the scenes here in a statistical sense. Okay, now I want to turn to the topic of LLM psychology, as I like to call it, which explores the emergent cognitive effects of the training pipeline that we have for these models. In particular, the first aspect I want to discuss is, of course, hallucinations. You might be familiar with model hallucinations.

It's when LLMs make things up. They totally fabricate information, etc. And it's a big problem with LLM assistance. This issue existed to a large extent with early models from many years ago. I think the problem has gotten a bit better because there are some mitigations that I'm going to discuss in a moment.

For now, let's just try to understand where these hallucinations come from. Here’s a specific example of three conversations that you might think are part of your training set. These are reasonable conversations that you could imagine being included. For instance, when asked, "Who is Tom Cruise?" the response would be, "Tom Cruise is a famous American actor and producer," etc.

Who is John Bartholomew? So, this turns out to be a U.S. senator, for example. Who is Genghis Khan? Well, Genghis Khan was blah, blah, blah. And so, this is what your conversations could look like during training. Now, the problem with this is that when the human is writing the correct answer for the assistant, in each of these cases, the human either knows who this person is or they research them on the internet.

And they come in and write this response that has a confident tone. What happens, basically, is that at test time, when you ask for someone who is... This is a totally random name that I just came up with, and as far as I know, this person does not exist. I simply generated it randomly. The problem arises when we ask, "Who is Orson Kovac?" The assistant will not just say, "Oh, I don't know." Even if the assistant and the language model itself might know internally that this person is not someone with which it is familiar, some part of the network might still recognize that in some sense. However, the model will not respond by saying, "I don't know who this is," because it statistically imitates its training set.

In the training set, questions of the form "Who is blah?" are confidently answered with the correct response. Therefore, the model adopts the style of the answer and does its best to provide a response. It will give you the statistically most likely guess. Essentially, it may fabricate information because, as mentioned earlier, these models do not have internet access and are not conducting research.

These are statistical token tumblers, as I call them. It's just trying to sample the next token in a sequence, and it's going to basically mix things up. So, let's take a look at what this looks like. I have here what's called an inference playground from Hugging Face, and I am purposefully picking on a model called Falcon 7B, which is an older model from a few years ago now.

So, it's an older model, so it suffers from hallucinations. As I mentioned, this has improved over time recently. But let’s say, who is Orson Kovac? Let’s ask Falcon 7B and start. Run. Oh yeah, Orson Kovac is an American author and science fiction writer.

Okay. This is totally false. It's a hallucination. Let's try again. These are statistical systems, right?

So we can resample. This time, Orson Kovac is a fictional character from a 1950s TV show. It's total BS, right? Let's try again. He's a former minor league baseball player. Okay, so basically, the model doesn't know.

And it's given us lots of different answers because it doesn't know. It's just kind of sampling from these probabilities. The model starts with the tokens "Who is Orson Kovac, assistant?" and then it comes in here, getting these probabilities and sampling from them, ultimately generating responses. The information is actually statistically consistent with the style of the answers in its training set. It operates this way, but we experience it as made-up factual knowledge.

But keep in mind that the model essentially doesn't know; it's just imitating the format of the answer. It isn't going to go off and look it up because it is, again, just imitating the answer. So how can we mitigate this? For example, when we go to ChatGPT and I ask, "Who is Orson Kovac?" I'm now asking the state-of-the-art model from OpenAI. This model will tell you that it is actually smarter because, as you briefly saw, it said "searching the web."

We're going to cover this later. It's actually trying to discuss tool use and kind of just came up with some kind of story, but I want to clarify that Bruce Orson Kovacs did not use any tools. I don't want it to do a web search. There's a well-known historical figure named Orson Kovac, so this model is not going to make up information.

This model knows that it doesn't know, and it informs you that it doesn't appear to be a person that this model recognizes. So, somehow, we have sort of improved hallucinations, even though they remain an issue in older models. It makes total sense why you would be receiving these kinds of answers if this is what your training set looks like. So, how do we fix this? Well, clearly, we need some examples in our dataset where the correct answer for the assistant is that the model doesn't know about some particular fact.

But we only need to have those answers produced in the cases where the model actually doesn't know. The question is: how do we know what the model knows or doesn't know? Well, we can empirically probe the model to figure that out. Let's take a look at how Meta dealt with hallucinations for the LLaMA 3 series of models as an example. In this paper they published for Meta, they delve into hallucinations, which they refer to as factuality. They describe the procedure by which they essentially interrogate the model to determine what it knows and doesn't know, helping to clarify the boundaries of its knowledge.

And then they add examples to the training set for those situations where the model doesn't know the answer, indicating that the correct response is that the model doesn't know. This may seem like a simple task in principle, but it effectively addresses the issue. The reason it resolves the problem is that the model might actually possess a fairly accurate understanding of its self-knowledge within the network. Remember when we examined the network and all the neurons inside it?

You might imagine there's a neuron somewhere in the network that lights up when the model is uncertain. However, the problem is that the activation of that neuron is not currently connected to the model's ability to express in words that it doesn't know. Even though the internals of the neural network recognize uncertainty, because there are some neurons representing that, the model will not surface this uncertainty. Instead, it will take its best guess, aiming to sound confident, just as it learned from the training set.

So we need to basically interrogate the model and allow it to say "I don't know" in cases where it doesn't have an answer. Let me walk you through what Metaroughly does. For example, today, Dominic Hasek is the featured article. I just went there randomly. What they do is take a random document from the training set, select a paragraph, and then use a large language model (LLM) to construct questions about that paragraph.

So, for example, I did that with ChatGPT here. I said, "Here's a paragraph from this document. Generate three specific factual questions based on this paragraph, and give me the questions and the answers." The large language models (LLMs) are already good enough to create and reframe this information. If the information is in the context window of this LLM, this actually works pretty well. It doesn't have to rely on its memory.

It's right there in the context window, and it can basically reframe that information with fairly high accuracy. For example, it can generate questions for us, such as, "For which team did he play?" Here's the answer: "How many cups did he win?"

Etc. Now, what we need to do is address some questions and answers, and we want to interrogate the model. Roughly speaking, we'll take our questions and present them to our model, which could be, for example, LLaMA from Meta, but let's use Interrogate Minstrel 7B here as an example. So, does this model know the answer to this question?

Let's take a look. So he played for the Buffalo Sabres, right? The model knows this. The way you can programmatically make a decision is essentially by taking this answer from the model and comparing it to the correct answer. Once again, the models are sophisticated enough to do this automatically.

So, there's no human involvement here. We can basically take the answer from the model, and we can use another LLM as a judge to verify if that answer is correct. If it is correct, that means the model probably knows. What we're going to do is this a few times. So, okay, it knows it's the Buffalo Sabres.

Let's try again. Buffalo Sabres. Let's try it one more time: Buffalo Sabres. We asked three times about this factual question, and the model seems to know, so everything is great.

Now let's try the second question. How many Stanley Cups did he win? Again, let's interrogate the model about that, and the correct answer is two. Here, the model claims that he won four times, which is not correct. It doesn't match the fact that the answer is two.

So the model doesn't know. It's making things up. Let's try again.

So here, the model again is kind of like making stuff up, right? Let's try again. It says he did not even win during his career. So obviously, the model doesn't know. The way we can programmatically tell is by interrogating the model three times and comparing its answers—maybe three times, five times, or whatever it is—to the correct answer.

And if the model doesn't know, then we know that the model doesn't know the answer to this question. What we do is take this question and create a new conversation in the training set. When the question is, "How many Stanley Cups did he win?" the correct answer would be, "I'm sorry, I don't know" or "I don't remember."

We interrogated the model and observed that this is indeed the case. When you apply this approach to various types of questions and documents, you give the model an opportunity, based on its training set, to refuse to provide answers rooted in its knowledge. If there are only a few examples of such refusals in the training set, the model can learn to associate this knowledge-based refusal with an internal neuron somewhere within its network that we assume exists. Empirically, this appears to be true. The model can learn the association that when this neuron of uncertainty is activated, it signifies a lack of knowledge, allowing it to express a response such as, "I'm sorry, but I don't think I remember this." If these examples are present in the training set, they significantly mitigate the risk of hallucination. That is roughly speaking why ChatGPT is capable of behaviors like this.

So, these are the kinds of mitigations that people have implemented, which have improved the factuality issue over time. Okay, so I've described mitigation number one for essentially addressing the hallucination issue. Now, we can actually do much better than that. Instead of just saying that we don't know, we can introduce an additional mitigation number two to give the language model an opportunity to be factual and actually answer the question. Now, what do you and I do if I were to ask you a factual question and you don't know?

What would you do in order to answer the question? Well, you could search the internet, figure out the answer, and then tell me what that answer is. We can do the exact same thing with these models. Think of the knowledge inside the neural network, stored within its billions of parameters, as a vague recollection of things that the model has seen during its training, particularly during the pre-training stage, which occurred a long time ago. In this sense, you can think of that knowledge as something you read a month ago.

And if you keep reading something, then you will remember it, and the model remembers that. However, if it's something rare, you probably don't have a very good recollection of that information. What you and I do is simply go and look it up. When you go and look it up, you are essentially refreshing your working memory with information, which enables you to retrieve it, discuss it, and so on. Therefore, we need some equivalent that allows the model to refresh its memory or recollection.

And we can do that by introducing tools for the models. The way we are going to approach this is that instead of just saying, "Hey, I'm sorry, I don't know," we can attempt to use tools. We can create a mechanism by which the language model can emit special tokens. These are tokens that we are going to introduce—new tokens. For example, I have introduced two tokens and a format or a protocol for how the model is allowed to use these tokens. Instead of answering the question when the model does not know, instead of just saying "I don't know, sorry," the model now has the option to emit the special token "search start," and this query will go to a search engine like Bing.com in the case of OpenAI or Google Search or something similar.

So, it will emit the query and then emit "search end." What will happen here is that the program, which is sampling from the model and running the inference, will see the special token "search end." Instead of sampling the next token in the sequence, it will pause generating from the model. It will open a session with Bing.com, paste the search query into Bing, and retrieve all the text that comes back. This text will be processed, potentially represented again with some other special token or notation. It will then copy and paste that text into the area I tried to illustrate with brackets. Once this text is added, it enters the context window.

The text from the web search is now inside the context window that will feed into the neural network. You should think of the context window as being similar to the working memory of the model. The data in the context window is directly accessible by the model; it directly feeds into the neural network.

So it’s no longer a vague recollection. It’s data that is in the context window and is directly available to the model. Now, when it samples new tokens afterward, it can easily reference the data that has been copy-pasted in there. That's roughly how these tools function, and web search is just one of the tools.

We're going to look at some of the other tools in a bit, specifically how these tools function. Web search is just one of the tools we will discuss. Essentially, you introduce new tokens and create a schema that allows the model to utilize these tokens and call special functions, such as web search functions.

And how do you teach the model how to correctly use these tools, like "web search start," "search end," etc.? Well, again, you do that through training sets. We need to have a collection of data and conversations that demonstrate to the model, by example, how to use web search. What are the settings in which you are using the search, and what does that look like? Here is an example of how you start a search, end a search, etc.

And if you have a few thousand examples of that in your training set, the model will actually do a pretty good job of understanding how this tool works. It will know how to structure its queries. Of course, because of the pre-training data set and its understanding of the world, it also understands what a web search is. It has a fairly good native understanding of what constitutes a good search query. So everything just works together. You provide a few examples to show it how to use this new tool, and then it can lean on it to retrieve information and put it in the context window.

And that's equivalent to you and me looking something up. Because once it's in context, it's in the working memory, and it's very easy to manipulate and access. So that's what we saw a few minutes ago when I was searching on ChatGPT for "Who is Orson Kovac?" The ChatGPT language model decided that this is some kind of rare individual or something like that. Instead of giving me an answer from its memory, it decided that it would sample a special token that would perform a web search.

And we briefly saw something flash. It was like using a web tool or something similar. It quickly displayed that, and then we waited for about two seconds before it generated this. You can see how it's creating references here, citing sources.

So what happened here is that it went off and conducted a web search. It found these sources and URLs, and the text of these web pages was all stuffed in between here. It's not shown here, but the text is basically included in that section. Now it sees that text and references it, indicating that it could belong to these citations or those citations, etc. That's what happened, and that's why, when I asked, "Who is Orson Kovats?" I could also say, "Don't use any tools," and that was enough to convince ChatGPT not to use any tools and just rely on its memory and recollection. I also went ahead and asked ChatGPT, "So how many Stanley Cups did Dominik Hasek win?" ChatGPT confidently stated that he won twice. It relied on its memory, presumably because it has enough confidence in its weights, parameters, and activations to retrieve this information just from memory.

But you can also, conversely, use web search to verify. For the same query, it actually goes off and searches, finding a variety of sources. It gathers all this information, which gets copy-pasted into the document. Then it cites everything again.

And it actually cites the Wikipedia article, which is the source of this information for us as well. So that's tools, web search. The model determines when to search, and that's kind of how these tools work. This is an additional kind of mitigation for hallucinations and factuality. I want to stress one more time this very important psychological point: knowledge in the parameters of the neural network is a vague recollection.

The knowledge in the tokens that make up the context window is the working memory. It roughly works like how it does in our brains. The information we remember represents our parameters, while the things we just experienced, like a few seconds or minutes ago, can be imagined as being in our context window. This context window is constructed as you have a conscious experience around you. This has several implications for your practical use of language models.

So, for example, I can go to ChatGPT and do something like this: I can say, "Can you summarize chapter one of Jane Austen's Pride and Prejudice?" right? This is a perfectly fine prompt, and ChatGPT actually provides a relatively reasonable response here. The reason it does that is because ChatGPT has a pretty good recollection of a famous work like Pride and Prejudice.

It's probably seen a ton of stuff about it. There are probably forums about this book. It's likely that many have read various versions of this book. It kind of sticks in your memory because, even if you've read this or articles about it, you'd have enough of a recollection to discuss it. However, when I interact with language models and want them to recall specific things, it usually works better if I just provide the information directly.

So I think a much better prompt would be something like this: Can you summarize for me Chapter One of Jane Austen's Pride and Prejudice? I am attaching it below for your reference. Then I will place a delimiter here and paste it in. I found that just copy-pasting it from a website I discovered is effective. By pasting Chapter One here, the model has direct access to it in the context window and doesn't need to recall it; it can access it exactly as provided.

And so this summary can be expected to be of significantly high quality or higher quality than a summary just because it's directly available to the model. I think you and I would work in the same way. If you wanted to, you would produce a much better summary if you had reread this chapter before you had to summarize it. That's essentially what's happening here, or its equivalent. The next psychological quirk I'd like to talk about briefly is that of the knowledge of self.

So what I see very often on the internet is that people do something like this: they ask language models questions like, "What model are you, and who built you?" Basically, this question is a little bit nonsensical. The reason I say that is that, as I tried to explain with some of the under-the-hood fundamentals, this thing is not a person, right? It doesn't have a persistent existence in any way.

It sort of boots up, processes tokens, and shuts off. It does that for every single person. It just builds up a context window for the conversation, and then everything gets deleted. This entity is kind of restarted from scratch with every single conversation, if that makes sense. It has no persistent self; there's no sense of self, and it follows the statistical regularities of its training set.

So it doesn't really make sense to ask it, "Who are you?" or "What built you?" etc. By default, if you do what I described, you're going to get some pretty random answers. For example, let's focus on Falcon, which is a fairly old model, and see what it tells us. It seems to be evading the question by responding with "Talented engineers and developers."

Here it says, "I was built by OpenAI, GPT-3 model." It's totally making stuff up. Now, the fact that it's built by OpenAI here, I think a lot of people would take this as evidence that this model was somehow trained on OpenAI data or something like that. I don't actually think that that's necessarily true. The reason for that is that if you don't explicitly program the model to answer these kinds of questions, then what you're going to get is its statistical best guess at the answer.

And this model had a SFT (Supervised Fine-Tuning) data mixture of conversations. During the fine-tuning, the model sort of understands, as it's training on this data, that it's taking on the personality of a helpful assistant. It doesn’t know exactly how to label itself; it just kind of adopts this persona of a helpful assistant. Remember that the pre-training stage involved gathering documents from the entire internet. ChatGPT and OpenAI are very prominent in these documents, and I think what’s likely happening here is that this is just its hallucinated label for what it is.

This is its self-identity: it is ChatGPT by OpenAI. It states this because there is a vast amount of data on the internet containing answers like this that originate from OpenAI and ChatGPT. Therefore, this is its label for what it is. However, you can override this. As a developer, if you have a large language model (LLM), you can actually override it.

And there are a few ways to do that. So, let me show you this Olmo model from Allen AI. This is one LLM. It's not a top-tier LLM or anything like that, but I like it because it's fully open source. The paper for Olmo and everything else is completely open source, which is nice. So here we are looking at its SFT mixture.

So this is the data mixture for the fine-tuning. This is the conversations dataset, right? The way they are solving it for the ULMO model shows that there is a variety of content in the mixture, with a total of 1 million conversations here. However, we have OLMO 2 hard-coded. If we check that, we see that there are 240 conversations.

And look at these 240 conversations. They're hard-coded. "Tell me about yourself," says the user. And then the assistant replies, "I'm Olmo, an open language model developed by AI2, the Allen Institute for Artificial Intelligence, etc. I'm here to help, blah, blah, blah."

What is your name? The Olmo Project. These are all kinds of cooked-up, hard-coded questions about Olmo 2 and the correct answers to provide in these cases. If you take 240 questions like this, or conversations, put them into your training set, and fine-tune with it, then the model will actually be expected to reproduce this information later. If you don't provide this, then it's likely just a ChatGPT by OpenAI.

And there's one more way to sometimes do this: basically, in these conversations, where there are exchanges between a human and an assistant, there is sometimes a special message called a system message. At the very beginning of the conversation, it is not just between the human and the system; there is also a system. In the system message, you can actually hard code and remind the model that, "Hey, you are a model developed by OpenAI and your name is ChatGPT-4. You were trained on this date, and your knowledge cut-off is this." Essentially, it helps document the model a little bit. Then, this information is inserted into your conversations. So, when you go on ChatGPT, you see a blank page, but actually, the system message is kind of hidden in there, and those tokens are in the context window.

And so, those are the two ways to program the models to talk about themselves. Either it's done through data like this, or through system messages and similar methods. Essentially, these are invisible tokens within the context window that remind the model of its identity. However, it's all somewhat arbitrary and added on in a way that doesn't truly reflect a deep understanding, unlike how it might have been before.

I want to now continue to the next section, which deals with the company’s computational capabilities, or rather, the native computational capabilities of these models in problem-solving scenarios. In particular, we have to be very careful with these models when we construct our examples of conversations. There are many sharp edges here that are quite elucidative. Is that a word? They are intriguing to examine as we consider how these models think.

So consider the following prompt from a human. Suppose that we are building out a conversation to enter into our training set of conversations. We will train the model on this. We are teaching it how to solve simple math problems. The prompt is: Emily buys three apples and two oranges.

Each orange costs two dollars. The total cost is thirteen. What is the cost of apples? This is a very simple math question. Now there are two answers here: one on the left and one on the right.

They are both correct answers. They both state that the answer is three, which is correct. However, one of these two is significantly better for the assistant than the other. If I were a data labeler creating one of these responses, one would be a really poor answer for the assistant, while the other would be acceptable. I would like you to pause the video and consider why one of these two is a significantly better answer than the other.

And if you use the wrong one, your model will actually perform poorly in math and could lead to negative outcomes. This is something you need to be cautious about in your labeling documentation when training people to create ideal responses for the assistant. So, the key to this question is to truly realize and remember that when the models are training and also inferring, they operate in a one-dimensional sequence of tokens from left to right. This is the image I often have in my mind: I envision the token sequence evolving from left to right.

To always produce the next token in a sequence, we are feeding all these tokens into the neural network. This neural network then provides us with the probabilities for the next token in the sequence, right? So this picture here is the exact same image we saw earlier, and it comes from the web demo that I showed you before. This is the calculation that essentially takes the input tokens at the top, performs operations with all the neurons, and gives you the probabilities for what comes next. Now, the important thing to realize is that, roughly speaking, there is a finite number of layers of computation that occur here.

So, for example, this model here has only one to three layers of what's called attention and MLP. A typical modern state-of-the-art network might have around 100 layers or something similar. However, there are only 100 layers of computation or something like that to go from the previous token sequence to the probabilities for the next token. Therefore, there's a finite amount of computation that occurs for every single token. You should think of this as a very small amount of computation, and this amount of computation is almost roughly fixed for every token in this sequence. That's not entirely accurate because the more tokens you feed in, the more expensive this forward pass will be for this neural network, but not by much.

So you should think of this, and I believe it's a good model to keep in mind: this is a fixed amount of compute that will occur in this box for every single one of these tokens. This amount of compute cannot possibly be too large because there aren’t that many layers moving from the top to the bottom here. There's not that much computation happening in this process. Therefore, it’s hard to envision a model that can perform arbitrary computation in a single forward pass to generate a single token.

And so what that means is that we actually have to distribute our reasoning and our computation across many tokens, because every single token is only spending a finite amount of computation on it. We want to distribute the computation across many tokens, and we can't have too much computation or expect too much computation out of the model in any single individual token, because there's only so much computation that happens per token. Okay, roughly, we have a fixed amount of computation here. So that's why this answer here is significantly worse. The reason for that is to imagine going from left to right here, and I copied and pasted it right here.

The answer is 3, etc. Imagine the model having to go from left to right, emitting these tokens one at a time. It has to say, or we're expecting it to say, "The answer is $". And then right here, we're expecting it to basically cram all the computation of this problem into this single token. It has to emit the correct answer: 3.

And then, once we've emitted the answer, 3, we're expecting it to acknowledge all these tokens. However, at this point, we have already produced the answer, and it is already in the context window for all the tokens that follow. So anything here is just a kind of post hoc justification for why this is the answer. The answer has already been created; it is already in the token window.

So it's not actually being calculated here. If you are answering the question directly and immediately, you are training the model to essentially guess the answer in a single token. That approach is not effective because of the finite amount of computation that occurs per token. That's why the answer on the right is significantly better; we are distributing this computation across the answer. We are actually guiding the model to gradually reach the answer.

From left to right, we're obtaining intermediate results. We're saying that the total cost of oranges is 4, so 13 minus 4 equals 9. We are creating intermediate calculations, and each of these calculations is not particularly expensive on its own. Essentially, we are making somewhat of an educated guess about the difficulty that the model can handle with each individual token. There can never be too much computational work associated with any single token; otherwise, the model won't be able to process that effectively later on.

And so, we are teaching the model to spread out its reasoning and computation over the tokens. In this way, it encounters only very simple problems with each token, which can then accumulate. By the time it reaches the end, it has all the previous results in its working memory, making it much easier to determine the final answer, which is 3. This is a significantly better label for our computation. On the other hand, trying to do all the computation in a single token is really bad. That's an important consideration to keep in mind. Usually, you don't have to think about it explicitly because the people at OpenAI have labelers who manage this. They ensure that the answers are spread out appropriately.

And so, actually, OpenAI will kind of do the right thing. When I ask this question for ChatGPT, it's actually going to go very slowly. It's going to be like, "Okay, let's define our variables, set up the equation," and it's kind of creating all these intermediate results. These are not for you; these are for the model.

If the model is not creating these intermediate results for itself, it's not going to be able to reach 3. I also wanted to show you that it's possible to be a bit mean to the model. We can just ask for things. So, as an example, I gave it the exact same prompt and said, "Answer the question in a single token. Just immediately give me the answer, nothing else."

And it turns out that for this simple prompt, it was actually able to do it in a single go. So it created a single response; I think this is two tokens, right? Because the dollar sign is its own token. So basically, this model didn't give me a single token. It gave me two tokens, but it still produced the correct answer.

And it did that in a single forward pass of the network. Now, that's because the numbers here, I think, are very simple. So I made it a bit more difficult to be somewhat challenging to the model. I said Emily buys 23 apples and 177 oranges, and then I just made the numbers a bit bigger.

And I'm just making it harder for the model. I'm asking it to do more computation in a single token. And so I said the same thing. Here it gave me five, but five is actually not correct.

So the model failed to perform all these calculations in a single forward pass of the network. It couldn't go from the input tokens and produce the result in just one pass through the network. Then I said, "Okay, don't worry about the token limit and just solve the problem as usual." By doing this, it generates all the intermediate results, simplifies them, and each of these intermediate results and calculations is much easier for the model to handle.

And it sort of—it's not too much work per token. All of the tokens here are correct, and it arises with a revised resolution, which is 7. It just couldn't squeeze all of this work into a single forward-passing network. So I think that's kind of a cute example and something to think about. I believe it is also somewhat elucidative in terms of how these models work.

The last thing I would say on this topic is that if I were in practice, trying to actually solve this in my day-to-day life, I might not fully trust that the model has carried out all the intermediate calculations correctly. In fact, what I would likely do is something like this: I would come here and say to use code. That's because code is one of the tools that ChatGPT can utilize. Instead of relying on mental arithmetic, like the calculations here, I don't completely trust it. This concern is particularly heightened when the numbers become very large, as there is no guarantee that the model will perform those calculations correctly.

Any one of these intermediate steps might, in principle, fail. We're using neural networks to perform mental arithmetic, similar to how you do mental arithmetic in your brain. Just like you can sometimes screw up some of the intermediate results, it's actually quite remarkable that it can even handle this kind of mental arithmetic. I don't think I could manage this in my head, but basically, the model is functioning as if it is doing it in its head. However, I don't trust that, so I wanted to use tools.

You can say things like "use code." I'm not sure what happened there. Use code. As I mentioned, there's a special tool, and the model can write code, which I can inspect to ensure that it is correct. It doesn't rely on its mental arithmetic; instead, it uses the Python interpreter, which is a simple programming language, to write out the code that calculates the result.

And I would personally trust this a lot more because this came from a Python program, which I think has a lot more correctness guarantees than the mental arithmetic of a language model. So, just another potential hint is that if you have these kinds of problems, you may want to ask the model to use the code interpreter. Just like we saw with the web search, the model has a special kind of tokens for calling. It will not actually generate these tokens from the language model; it will write the program and then send that program to a different part of the computer that runs the program and brings back the result. Then, the model gets access to that result and can tell you that, okay, the cost of each apple is seven. So, that's another kind of tool, and I would use this in practice for yourself.

So, that's another kind of tool. I would use this in practice for yourself. And it's, yeah, it's just less error-prone, I would say. That's why I called this section "Models Need Tokens to Think." Distribute your computation across many tokens, ask models to create intermediate results, or whenever you can, lean on tools and tool use instead of allowing the models to do all of this in their memory.

So, if they try to do it all from memory, don't fully trust it and prefer to use tools whenever possible. I want to show you one more example of where this actually comes up, and that's in counting. Some models actually are not very good at counting for the exact same reason: you're asking for way too much from a single individual token.

Let me show you a simple example of that. How many dots are below? Then I just put in a bunch of dots, and ChatGPT responds with the number of dots. It tries to solve the problem in a single token. So, in a single token, it has to count the number of dots in its context window.

And it has to do that in a single forward pass of a network. In a single forward pass of a network, as we talked about, there's not that much confidence that can occur there. Just think of that as very little computation happening. So if I just look at what the model sees, let's go to the LLM and the tokenizer. It sees this.

How many dots are below? It turns out that this group of what I think are 20 dots is a single token. Then, this other group, whatever it is, forms another token. For some reason, they break up like this. I don't actually understand the details of the tokenization process, but it turns out that the model basically sees the token IDs, one after the other.

And then from these token IDs, it's expected to count the number. And spoiler alert, it's not 161. It's actually, I believe, 177. So here's what we can do instead: we can say to use code.

And you might expect that, like, “Why should this work?” It's actually kind of subtle and interesting. So when I say "use code," you might wonder about its effectiveness. It's a more nuanced topic than it initially seems.

So when I say use code, I actually expect this to work. Let's see. Okay, that is correct. What happens here is that I've actually, though it doesn't look like it, broken down the problem into smaller problems that are easier for the model. I know that the model can't count.

It can't do mental counting, but I know that the model is actually pretty good at doing copy-pasting. So what I'm doing here is, when I say "use code," it creates a string in Python for this, and the task of basically copy-pasting my input from here to there is very simple because, for the model, it sees this string as just these four tokens or whatever it is. So it's very simple for the model to copy-paste those token IDs and kind of unpack them into dots here. It creates this string, then it calls `pythonroutine.count`, and finally, it comes up with the correct answer. So the Python interpreter is doing the counting.

It's not the model's mental arithmetic that does the counting. Again, it's a simple example: models need tokens to think and shouldn't rely on their mental arithmetic. That's why models are not very good at counting. If you need them to perform counting tasks, always ask them to lean on the tool. Moreover, the models have various cognitive deficits here and there, which are important to be aware of over time. For instance, models are not very good with various spelling-related tasks.

They're not very good at it. I told you that we would loop back around to tokenization, and the reason for this is that the models don't see characters; they see tokens. Their entire world is made up of tokens, which are these little text chunks. They don't perceive characters the way our eyes do.

And so, very simple character-level tasks often fail. For example, I'm giving it the string "ubiquitous," and I'm asking it to print only every third character, starting with the first one. We start with 'u,' and then we should go every third character. So, one, two, three—'q' should be next, etc. So, I see that this is not correct.

And again, my hypothesis is that the mental arithmetic here is failing, number one, a little bit. But number two, I think the more important issue is that if you go to the Tokenizer and look at "ubiquitous," we see that it is three tokens, right? So you and I see "ubiquitous" and can easily access the individual letters because we can visualize them. When we have it in the working memory of our visual field, we can index into every letter easily. I can do that task, but the models don't have access to the individual letters; they see this as three tokens. Remember, these models are trained from scratch on the internet, and they must discover how many different letters are packed into all these different tokens. The reason we even use tokens is mostly for efficiency, but I think many people are interested in deleting tokens entirely. We should really have character-level or byte-level models.

It's just that this would create very long sequences, and people don't know how to deal with that right now. So while we have the token world, any kind of spelling tasks are not actually expected to work super well. Because I know that spelling is not a strong suit due to tokenization, I can again ask it to lean on tools. I can simply say "use code," and I would expect this to work well because the task of copy-pasting "ubiquitous" into the Python interpreter is much easier. Then we're leaning on the Python interpreter to manipulate the characters of this string. So when I say "use code ubiquitous," yes, it indexes into every third character, and the actual result is "uq2s, uqts," which looks correct to me.

So again, an example of spelling-related tasks not working very well. A very famous example recently is how many "r"s there are in "strawberry," and this went viral many times. Basically, the models now get it correct; they say there are three "r"s in "strawberry." However, for a very long time, all the state-of-the-art models would insist that there are only two "r"s in "strawberry." This caused a lot of, you know, ruckus—is that a word? I think so.

Because it's just kind of like, why are the models so brilliant? They can solve math Olympiad questions, but they can't count the letters 'r' in "strawberry." The answer, again, is that I’ve kind of built up to it slowly. First, the models don't see characters; they see tokens. Second, they are not very good at counting. So here we are, combining the difficulty of recognizing characters with the difficulty of counting.

And that's why the models struggled with this. Even though I think, by now, honestly, OpenAI may have hard-coded the answer here, or I'm not sure what they did, this specific query now works. Models are not very good at spelling, and there are a bunch of other little sharp edges, and I don't want to go into all of them. I just want to show you a few examples of things to be aware of when you're using these models in practice. I don't actually want to have a comprehensive analysis here of all the ways that the models are kind of falling short. I just want to make the point that there are some jagged edges here and there.

We’ve discussed a few of these models, and some of them make sense, while others do not quite add up. You might find yourself scratching your head, even if you deeply understand how these models function. A recent example highlights this issue: the models struggle with very simple questions like this one. This phenomenon is shocking to many because these models can solve complex math problems.

They can answer PhD-level physics, chemistry, and biology questions much better than I can. However, sometimes they fall short on super simple prompts like this. So here we go: 9.11 is bigger than 9.9. It justifies it in some way, but obviously... And then at the end, it actually flips its decision later.

So I don't believe that this is very reproducible. Sometimes it flips around its answer. Sometimes it gets it right, and sometimes it gets it wrong. Let's try again.

Okay, even though it might look larger, it doesn't even correct itself in the end. If you ask many times, sometimes it gets it right too. But how is it that the model can perform so well on Olympiad-grade problems, yet fail on very simple problems like this? I think, as I mentioned, this one is a bit of a head-scratcher.

It turns out that a bunch of people studied this in depth, and I haven't actually read the paper, but what I was told by this team was that when you scrutinize the activations inside the neural network, when you look at some of the features and what features turn on or off, and what neurons turn on or off, a bunch of neurons inside the neural network light up that are usually associated with Bible verses. So I think the model is kind of reminded that these almost look like Bible verse markers. In a Bible verse setting, 9:11 would come after 9:9. Therefore, the model somehow finds it cognitively very distracting that in Bible verses, 9:11 would come after 9:9.

And so, basically, the model finds it cognitively very distracting that, in Bible verses, 9.11 would be greater. Even though it is trying to justify this and arrive at the answer through math, it still ends up with the wrong answer. So, it doesn't fully make sense and isn't completely understood; there are a few jagged issues like that. That's why you should treat this as what it is—a stochastic system that is really magical, but one that you can't fully trust. You want to use it as a tool, not something that you just let run free on a problem and then copy and paste the results. In terms of the major stages of training large language models, we observed that in the first stage, known as the pre-training stage, we are essentially training on internet documents.

And when you train a language model on internet documents, you get what's called a base model, which is basically an internet document simulator. Now, we saw that this is an interesting artifact, and it takes many months to train on thousands of computers. It's a kind of lossy compression of the internet. While it's extremely interesting, it's not directly useful, because we don't want to sample internet documents; we want to ask questions of an AI and have it respond to our inquiries. For that, we need an assistant. We observed that we can actually construct an assistant in the process of post-training, specifically during the process of supervised fine-tuning, as we call it.

In this stage, we observed that it is algorithmically identical to pre-training. Nothing is going to change; the only thing that changes is the dataset. Instead of internet documents, we now want to create and curate a well-defined dataset of conversations. Our goal is to have millions of conversations on a wide range of diverse topics between a human and an assistant.

And fundamentally, these conversations are created by humans. So, humans write the prompts, and humans write the ideal responses. They do that based on labeled documentation. Now, in the modern stack, it's not actually done fully and manually by humans; they now have a lot of help from these tools. We can use language models to assist us in creating these data sets, and we have done so extensively. But fundamentally, it all still comes from human curation in the end.

We create these conversations, which now become our dataset. We fine-tune on it or continue training on it, and we end up with an assistant. Then we shifted gears and began discussing some of the cognitive implications of what this assistant is like. For example, we observed that the assistant tends to hallucinate if no mitigations are applied. We found that hallucinations are common and subsequently explored some of the methods to mitigate those hallucinations.

And then we saw that the models are quite impressive and can do a lot of things in their heads. However, we also observed that they can lean on tools to enhance their capabilities. For example, we can utilize web search to reduce hallucinations and possibly retrieve more recent information. Additionally, we can use tools like a code interpreter, allowing the large language model to write code, execute it, and see the results. These are some of the topics we have explored so far.

Now what I'd like to do is cover the last and major stage of this pipeline, which is reinforcement learning. While reinforcement learning is often considered part of the post-training phase, it represents the final and distinct major stage in the process. It is a different approach to training language models and typically follows as the third step. Within companies like OpenAI, the structure is organized into separate teams. One team focuses on data for pre-training, while another team handles the training for pre-training. Additionally, there is a team responsible for conversation generation, along with another team dedicated to supervised fine-tuning.

And then there will be a team for reinforcement learning as well. So it's kind of like a handoff of these models. You get your base model, then you fine-tune it to be an assistant, and then you go into reinforcement learning, which we'll talk about now. That's the major flow. Let's now focus on reinforcement learning, the last major stage of training. Let me first motivate it and explain why we would want to do reinforcement learning, as well as what it looks like on a high level.

So now I'd like to try to motivate the reinforcement learning stage and what it corresponds to. That's something that you're probably familiar with, and it is basically going to school. Just like you went to school to become really good at something, we want to take large language models through school. Essentially, what we're doing is employing a few paradigms for giving them knowledge or transferring skills.

So, in particular, when we're working with textbooks in school, you'll find that there are three major pieces of information in these textbooks, categorized into three classes of information. The first thing you'll notice is a significant amount of exposition. By the way, this is a totally random book I pulled from the internet. I think it's some kind of organic chemistry text or something. I'm not sure.

But the important thing is that you'll see that most of the text, most of it, is kind of just like the meat of it; it's explicit. It's kind of like background knowledge, etc. As you are reading through the words of this exposition, you can think of that roughly as training on that data. And that's why when you're reading through this material, this background knowledge and all this contextual information is kind of equivalent to pre-training. It’s where we build a sort of knowledge base from this data and get a sense of the topic. The next major kind of information that you will see is these problems along with their worked solutions.

So, basically, a human expert, in this case the author of this book, has given us not just a problem, but has also worked through the solution. The solution is essentially equivalent to having an ideal response for an assistant. The expert is demonstrating how to solve the problem in its entirety. As we read the solution, we are essentially training on the expert data, and later we can attempt to imitate the expert. This roughly corresponds to having the supervised fine-tuning (SFT) model.

That's what it would be doing. So basically, we've already completed pre-training, and we've covered the imitation of experts and how they solve these problems. The third stage of reinforcement learning is essentially the practice problems. Sometimes, you'll see this as just a single practice problem here, but there will usually be many practice problems at the end of each chapter in any textbook. We know that practice problems are critical for learning because they encourage you to engage with the material. What are they prompting you to do?

They're encouraging you to practice independently and find ways to solve these problems on your own. In the practice problem, you receive the problem description, but you are not provided with the solution. However, you do get the final answer, usually found in the answer key of the textbook. This means you know the final answer you are trying to reach, along with the problem statement, but you lack the solution itself. You are attempting to practice finding the solution, experimenting with various approaches to see which one leads you to the final answer most effectively. In this way, you are discovering how to solve these problems.

And in the process of that, you're relying on, number one, the background information, which comes from pre-training, and number two, maybe a little bit of imitation of human experts. You can probably try similar kinds of solutions and so on. So we've done this and that, and now in this section, we're going to practice. We’re going to be given prompts.

We're going to be given solutions—sorry, the final answers—but we're not going to be given expert solutions. We have to practice and try things out, and that's what reinforcement learning is about. Okay, so let's go back to the problem we worked on previously, just so we have a concrete example to discuss as we explore the topic here. I'm here in the tokenizer because I'd also like to, well, I get a text box, which is useful.

But number two, I want to remind you again that we're always working with non-dimensional token sequences. So, I actually prefer this view because it is like the native view of the LLM, if that makes sense. This is what it actually sees: it sees token IDs, right? Okay.

So, Emily buys three apples and two oranges. Each orange costs $2. The total cost of all the fruit is $13. What is the cost of each apple? I'd like you to appreciate that there are four possible candidate solutions as an example, and they all arrive at the answer of three.

Now, what I'd like you to appreciate at this point is that if I'm the human data labeler creating a conversation to be entered into the training set, I don't actually know which of these conversations to add to the dataset. Some of these conversations set up a system of equations, some of them just discuss the problem in English, and some skip right to the solution. If you look at the solution, ChatGPT, for example, and you give it this question, it defines a system of variables and goes through the process. However, what we need to appreciate and differentiate between is that the primary purpose of a solution is to reach the correct answer. Of course, we want to arrive at the final answer: three.

The important purpose here is clear. However, there is also a secondary purpose; we are trying to make the process enjoyable for the human. We are assuming that the person wants to see the solution, including intermediate steps, and we want to present it nicely. Therefore, there are two separate objectives: first, the presentation for the human, and second, the aim of arriving at the correct answer. For now, let's focus solely on reaching the final answer.

If we only care about the final answer, then which of these is the optimal, or the best, prompt? Sorry, the best solution for the LLM to reach the right answer? What I'm trying to get at is that we don't know. As a human labeler, I would not know which one of these is best. For example, we saw earlier when we looked at the token sequences and the mental arithmetic and reasoning that for each token, we can only spend a finite amount of compute, which is not very large. You should think about it that way. Therefore, we can't actually make too big of a leap with any one token; maybe you should consider it in that context.

And so, we can't actually make too big of a leap in any one token; that’s maybe the way to think about it. As an example, in this case, what's really nice is that there are very few tokens, so it will take us a very short amount of time to get to the answer. However, when we consider the expression "13 minus 4 divided by 3," we are actually asking for a lot of computation to happen on that single individual token. This might not be a good example to give to the language model (LLM) because it incentivizes it to skip through the calculations very quickly, which can lead to mistakes in this mental arithmetic.

So maybe it would work better to spread it out more. Perhaps it would be better to set it up as an equation. Maybe it would be better to talk through it. We fundamentally don't know. And we don't know because what is easy for you or me—or as human labelers—what's easy or hard for us is different from what’s easy or hard for the LLM. Its cognition is different.

And the token sequences are kind of like different hurdles for it. Some of the token sequences here that are trivial for me might be too much of a leap for the LLM. For instance, this token would be way too hard. Conversely, many of the tokens that I'm creating here might be trivial for the LLM, and we're just wasting tokens. Why waste all these tokens when this is all trivial?

So, if the only thing we care about is reaching the final answer and we're separating the issue of presentation to the human, then we don't actually know how to annotate this example. We don't know what solution to provide to the LLM because we are not the LLM. This is particularly clear in the case of a math example, but it is actually a very pervasive issue. Our knowledge is not the LLM's knowledge; the LLM possesses a vast amount of knowledge equivalent to a PhD in math, physics, chemistry, and more. In many ways, it actually knows more than I do.

And I'm potentially not utilizing that knowledge in its problem-solving. But conversely, I might be injecting a bunch of knowledge into my solutions that the LLM doesn't have in its parameters. Then those are like sudden leaps that are very confusing to the model. Our cognitions are different, and I don't really know what to put here if all we care about is reaching the final solution and doing it economically, ideally. So, long story short, we are not in a good position to create these token sequences for the LLM.

And they're useful by imitation to initialize the system, but we really want to allow it to discover the token sequences that work for it. We need to find, and it needs to find for itself, what token sequence reliably leads to the answer given the prompt. It needs to discover that in the process of reinforcement learning and trial and error. So, let's see how this example would work in reinforcement learning. Okay, we are now back in the Hugging Face Inference Playground.

And that just allows me to very easily call different kinds of models. So, as an example, here on the top right, I chose the Gemma 2 billion parameter model. Two billion is very, very small, so this is a tiny model, but it's okay. The way that reinforcement learning will basically work is actually quite simple.

We need to try many different kinds of solutions, and we want to see which solutions work well and which do not. So we're basically going to take the prompt, run the model, and the model generates a solution. Then we're going to inspect the solution. We know that the correct answer for this one is $3. Indeed, the model gets it correct; it says it's $3.

So this is correct. That's just one attempt at the solution. Now we're going to delete this and rerun it again. Let's try a second attempt. The model solves it in a slightly different way, right?

Every single attempt will be a different generation because these models are stochastic systems. Remember that at every single token here, we have a probability distribution, and we're sampling from that distribution. So we end up kind of going down slightly different paths. This is a second solution that also ends in the correct answer. Now, we're going to delete that.

Let's go a third time. Okay, so again, we have a slightly different solution, but it also gets it correct. Now we can actually repeat this many times. In practice, you might sample thousands of independent solutions or even up to a million solutions for just a single prompt. Some of them will be correct, while others may not be very accurate. Basically, what we want to do is encourage the solutions that lead to correct answers.

Let's take a look at what that looks like. Over here, we have a cartoon diagram that illustrates it. We start with a prompt and then try many different solutions in parallel. Some of the solutions may succeed and provide the correct answer, which is indicated in green. However, other solutions may not perform well and may not reach the correct answer, also highlighted in green.

And some of the solutions might go poorly and may not reach the right answer, which is red. This problem here, unfortunately, is the best example because it's a trivial prompt, and as we saw, even a 2 billion parameter model always gets it right. So it's not the best example in that sense. But let's just exercise some imagination here and suppose that the green ones are good and the red ones are bad. We generated 15 solutions, and only four of them got the right answer.

And so now, what we want to do is encourage the kinds of solutions that lead to correct answers. Whatever token sequences occurred in these red solutions clearly indicate that something went wrong somewhere along the way, and this was not a good path to take through the solution. In contrast, the token sequences in these green solutions show that things went well in this situation. Therefore, we want to replicate more of this behavior in similar prompts. The way we encourage this kind of behavior in the future is by training on these sequences.

But these training sequences now are not coming from expert human annotators. There's no human who decided that this is the correct solution. This solution came from the model itself. So the model is practicing here. It has tried out a few solutions, four of which seem to have worked, and now the model will train on them.

And this corresponds to a student basically looking at their solutions and thinking, "Okay, well this one worked really well, so this is how I should be solving these kinds of problems." In this example, there are many different ways to tweak the methodology a little bit. However, to get the core idea across, it might be simplest to just consider the single best solution out of these four—like, say, this one. That's why it was highlighted in yellow. This is the solution that not only arrived at the correct answer but also had some other nice properties. Maybe it was the shortest one, or it appeared more appealing in some way, or there's another criterion you could think of as an example.

But we're going to decide that this is the top solution. We're going to train on it, and then the model will be slightly more likely, once you do the parameter update, to take this path in this kind of setting in the future. However, you have to remember that we're going to run many different and diverse prompts across lots of math problems, physics problems, and whatever else there might be. We may have tens of thousands of prompts in mind, each with thousands of solutions.

And so, this is all happening kind of like at the same time. As we iterate through this process, the model is discovering for itself what kinds of token sequences lead it to correct answers. It's not coming from a human annotator. The model is, in a way, playing in this playground; it knows what it's trying to achieve and is discovering sequences that work for it. These are sequences that don't require any mental leaps.

They seem to work reliably and statistically, fully utilizing the knowledge of the model as it exists. This is the process of reinforcement learning. It's basically a guess and check approach. We will guess many different types of solutions, check their effectiveness, and will focus on what worked in the future. That is reinforcement learning.

So, in the context of what came before, we see now that the SFT model, the Supervised Fine-Tuning model, is still helpful because it initializes the model into the vicinity of the correct solutions. It's like an initialization of the model in the sense that it helps the model generate solutions, understanding how to set up a system of equations or interact with a solution. This process brings the model closer to finding correct solutions. However, reinforcement learning is where everything gets dialed in. We truly discover the solutions that work for the model, identify the right answers, encourage them, and then the model improves over time. Okay, so that is the high-level process for how we train large language models.

In short, we train AIs in a manner quite similar to how we train children. The main difference is that children go through chapters of books and engage in various types of training exercises within each chapter. Instead, when we train AIs, we do it stage by stage, depending on the type of stage. First, we conduct pre-training, which is equivalent to reading all the expository material. We examine all the textbooks simultaneously, focusing on the exposition, and we aim to build a knowledge base. The next step is the SFT stage, where we look at all the fixed solutions from human experts, analyzing various worked solutions across all the textbooks. We then create an SFT model that can imitate the experts, albeit in a somewhat blind manner. It does its best to mimic statistical expert behavior based on the worked solutions.

And then finally, in the last stage, we do all the practice problems in the reinforcement learning stage. Across all the textbooks, we only do the practice problems. And that's how we get the reinforcement learning model. So, on a high level, the way we train large language models is very much equivalent to the process that we use for training children. The next point I would like to make is that, actually, these first two stages—pre-training and supervised fine-tuning—have been around for years. They are very standard, and everyone uses them; all the different large language model providers do.

It is this last stage, the RL training, that is much earlier in its process of development and is not yet standard in the field. Therefore, this stage is still quite early and nascent. The reason for this is that I actually skipped over many minor details in this process. The high-level idea is very simple: it's trial-and-error learning.

But there are a ton of details and little mathematical nuances regarding how to select the best solutions and how much to train on them. This includes considerations like the prompt distribution and how to set up the training run to ensure effectiveness. There are many intricate details and adjustments to the core idea, which is actually very straightforward. Therefore, getting these details right is not trivial. Many companies, such as OpenAI and other large language model (LLM) providers, have been experimenting internally with reinforcement learning fine-tuning for LLMs for some time, but they have not discussed it publicly; it's mostly kept internal to the company.

And so that's why the paper from DeepSeek that came out very recently was such a big deal. This paper is from a company called DeepSeekAI in China. It discussed reinforcement learning fine-tuning for large language models and highlighted its significance in enhancing reasoning capabilities within these models. We'll delve into this shortly. This paper reignited public interest in using reinforcement learning for large language models and provided many of the crucial details necessary to reproduce the results and successfully apply the techniques to larger language models.

Let me take you briefly through this DeepSeq R1 paper and what happens when you correctly apply reinforcement learning to language models, what that looks like, and what benefits it provides. The first thing I’ll scroll to is Figure 2, where we examine the improvement in how the models are solving mathematical problems. This figure presents the accuracy of solving math problems based on the AIME accuracy metric. We can also visit the webpage to view the specific types of math problems being measured. These are simple math problems. You can pause the video if you’d like.

But these are the kinds of problems that the models are essentially being asked to solve. You can see that, in the beginning, they are not performing very well, but as you update the model through thousands of steps, their accuracy continues to climb. The models are improving and solving these problems with higher accuracy as you conduct trial and error on a large dataset of such problems. The models are discovering how to solve math problems. Even more incredible, beyond the quantitative results of solving these problems with greater accuracy, is the qualitative means by which the models achieve these results.

So, when we scroll down, one of the figures that is kind of interesting is that later on in the optimization, the model seems to be using an average length per response that goes up. The model appears to be using more tokens to achieve its higher accuracy results, indicating that it is learning to create very long solutions. Why are these solutions so lengthy? We can examine them qualitatively here.

So basically, what they discover is that the model's solutions get very, very long. This happens partially because, here’s a question and here’s kind of the answer from the model. What the model learns to do—an emergent property of the optimization it undergoes—is that it discovers that this approach is effective for problem-solving. It starts to do things like this: "Wait, wait, wait, that's an aha moment I can flag here. Let's reevaluate this step by step to identify if the correct sum can be achieved." So, what is the model doing here, right?

The model is basically re-evaluating steps. It has learned that it works better for accuracy to try out lots of ideas, approach problems from different perspectives, retrace, reframe, and backtrack. It's engaging in many of the strategies that you and I use while problem-solving for mathematical questions, but it's rediscovering what happens in your mind, not what you put down in the solution. There is no human who can hard-code this kind of reasoning into the ideal assistant response. This is something that can only be discovered through the process of reinforcement learning, because you wouldn't know what to input here. This approach turns out to work well for the model, improving its accuracy in problem-solving.

So the model learns what we call these chains of thought in your head, and it's an emergent property of the optimization. That's what's bloating up the response lens, but it's also what's increasing the accuracy of problem-solving. What's incredible here is that the model is essentially discovering ways to think. It's learning what I like to call cognitive strategies—how to manipulate a problem and approach it from different perspectives, how to pull in analogies, and various other techniques. It also explores many different options over time, checks results from multiple angles, and ultimately solves problems. Remarkably, this process is discovered through real-life interactions.

So incredibly amazing to see this emerge in the optimization without having to hard code it anywhere. The only thing we've provided are the correct answers, and this result comes from trying to solve them accurately, which is truly impressive. Now, let's return to the problem we've been working on and examine what it would look like for this type of model, which we refer to as a reasoning or thinking model, to solve that problem. Okay, so recall that this is the problem we've been focusing on, and when I pasted it into ChatGPT 4.0, I'm getting this kind of response. Let's take a look at what happens when you give the same query to what's called a reasoning or thinking model.

This is a model that was trained with reinforcement learning. The model described in this paper, DeepSeek, is available on chat.deepc.com. The company that developed it is hosting it. You need to ensure that the DeepThink button is turned on to access the R1 model, as it's called. We can paste it here and run it.

And so, let's take a look at what happens now and what the output of the model is. Okay, here's what it says. This is previously what we obtained using a supervised fine-tuning (SFT) approach, which mimics an expert solution. This is what we get from the RL model.

Okay, let me try to figure this out. So, Emily buys three apples and two oranges. Each orange costs $2. The total is $13. I need to find out, blah, blah, blah.

So here, as you're reading this, you can't help but think that this model is thinking. It's definitely pursuing a solution. It determines that it must cost $3. Then it says, "Wait a second, let me check my math again to be sure." After that, it approaches the problem from a slightly different perspective.

And then it says, "Yep, all that checks out. I think that's the answer. I don't see any mistakes. Let me see if there's another way to approach the problem, maybe setting up an equation. Let's let the cost of one apple be $8, then blah, blah, blah." Yep, same answer.

So, definitely, each apple is $3. All right, I am confident that that's correct. Once it completes the thinking process, it writes up a clear solution for the human. This focuses on the correctness aspect as well as the presentation aspect, where it neatly organizes the information and boxes in the correct answer at the bottom. What's incredible about this is that we get to see the thinking process of the model.

And this is what’s coming from the reinforcement learning process. This is what’s bloating the length of the token sequences. They’re thinking and trying different approaches. This is what’s resulting in higher accuracy in problem-solving. This is where we are seeing these "aha" moments, various strategies, and ideas for ensuring that you’re getting the correct answer.

The last point I want to make is that some people are a little bit nervous about putting very sensitive data into chat.deepseek.com because this is a Chinese company. As a result, people tend to be a bit careful and cautious. DeepSeek R1 is a model that was released by this company. This is an open-source model with open weights, available for anyone to download and use. However, you will not be able to run it at full precision in its entirety.

You won't run that on a MacBook or any local device because this is a fairly large model, but many companies are hosting the full, largest model. One of those companies that I like to use is called Together.AI. When you go to Together.AI, you sign up and access the playgrounds. You can select "DeepSeek R1" in the chat, and there are many different kinds of other models that you can choose from. These are all state-of-the-art models.

So this is kind of similar to the Hugging Face Inference Playground that we've been using so far. However, Together.AI usually hosts all the state-of-the-art models. So, select DeepSeek R1. You can try to ignore a lot of these options; I think the default settings will often be okay.

And we can include this. Since the model was released by DeepSeek, what you're getting here should be essentially equivalent to what you're getting elsewhere. However, due to the randomness in the sampling, we may observe slight differences. In principle, this should be identical in terms of the model's power, and you should be able to see similar results both quantitatively and qualitatively. It's worth noting that this model is coming from an American company.

So that's DeepSeq, and that's what's called a reasoning model. Now, when I go back to chat, let me go to chat here. Okay, so the models that you're going to see in the dropdown here, some of them, like 01, 03 Mini, 03 Mini High, etc., refer to models that use advanced reasoning. What this refers to is that they were trained using reinforcement learning with techniques similar to those of DeepSeq R1, according to public statements from OpenAI employees. So these are reasoning models trained with reinforcement learning, and models like GPT-4.0 or GPT-4.0 Mini that you're getting in the free tier should be thought of as mostly SFT models, supervised fine-tuning models.

They don't actually think in the way you see in reinforcement learning (RL) models. Even though there's a little bit of reinforcement learning involved with these models—I'll go into that in a moment—they are mostly supervised format design (SFD) models. I think you should consider it that way. In the same vein as what we observed here, we can choose one of the thinking models, such as the 03 Mini High. By the way, these models might not be available to you unless you pay for a ChatGPT subscription of either $20 per month or $200 per month for some of the top models. So, we can select a thinking model and run it.

Now, what's going to happen here is that it's going to say "reasoning," and it's going to start to do things like this. What we're seeing is not exactly what is happening behind the scenes. Under the hood, the model produces these kinds of chains of thought. OpenAI chooses not to show the exact chains of thought in the web interface; instead, it presents brief summaries of those chains. I believe OpenAI does this partly because they are concerned about what is called a distillation risk.

That is, someone could come in and actually try to imitate those reasoning traces and recover a lot of the reasoning performance by just imitating the reasoning chains of thought. They kind of hide them and only show little summaries of them, so you're not getting exactly what you would get in DeepSeq with respect to reasoning itself. Then, they write up the solution. These are kind of equivalent, even though we're not seeing the full under-the-hood details.

Now, in terms of performance, these models and DeepSeq models are currently roughly on par, I would say. It's kind of hard to tell because of the evaluations. But if you're paying $200 per month to OpenAI, some of these models, I believe, still look better. However, DeepSeek R1, for now, is still a very solid choice for a thinking model that is available to you either on this website or any other website because the model has open weights that you can download. So, what is the summary so far? Well, we've talked about reinforcement learning and the fact that thinking emerges in the process of optimization when we run reinforcement learning on many math and code problems that have verifiable solutions, like an answer of three, etc. These thinking models can be accessed, for example, in DeepSeq or any inference provider like Together.ai, especially when choosing DeepSeq. Additionally, these thinking models are also available in ChatGPT under any of the 01 or 03 models.

But these GPT-4O models, etc., they're not thinking models. You should think of them as mostly SFT models. Now, if you have a prompt that requires advanced reasoning and so on, you should probably use some of the thinking models or at least try them out. However, empirically, for a lot of my uses, when you're asking a simpler question—like a knowledge-based question or something similar—this might be overkill. There's no need to think for 30 seconds about a factual question.

So for that, I will sometimes default to just GPT-4.0. Empirically, about 80-90% of my use is just GPT-4.0. When I come across a very difficult problem, like in math and coding, I will reach for the thinking models. However, I have to wait a bit longer because they are processing. You can access these on ChatGPT, on DeepSeq. Also, I wanted to point out that AISTudio.google.com, even though it looks really busy and unattractive because Google struggles with this kind of interface, raises the question: What is happening?

But if you choose the model and you select "Gemini 2.0 Flash Thinking Experimental 0.1.21," that's also an early experimental thinking model by Google. We can go here, provide it with the same problem, and click "Run." This thinking model will also produce a similar result and arrive at the correct answer. Essentially, Gemini also offers a thinking model, while Anthropic currently does not provide one. Overall, this represents the frontier development of these language models.

I think reinforcement learning is kind of like this new, exciting stage, but getting the details right is difficult. That's why all these models and thinking frameworks are currently experimental as of early 2025. This is essentially the frontier of development, pushing the performance on these very challenging problems using reasoning that emerges through these optimizations. One more connection I wanted to bring up is that the discovery that reinforcement learning is an extremely powerful way of learning is not new to the field of artificial intelligence. One place where we've already seen this demonstrated is in the game of Go. Famously, DeepMind developed the system AlphaGo, and you can watch a documentary about it, where the system learns to play the game of Go against top human players.

When we look at the paper underlying AlphaGo, we find a really interesting plot that I think is somewhat familiar to us. We're sort of rediscovering it in the broader context of arbitrary problem solving, as opposed to the specific domain of the game of Go. Essentially, what they observed—and what we will see in large language models (LLMs) as they become more mature—is illustrated by the Elo rating of players in the game of Go. Lisa Dull is represented as an extremely strong human player. In the comparison, they assess a model trained by supervised learning against a model trained by reinforcement learning. The supervised learning model imitates human expert players. By acquiring a vast amount of games played by expert players in Go and attempting to imitate them, one can improve, but there is a limit. You will never surpass the top players of the game, such as Lisa Dull, because you are merely imitating human players. If you are exclusively imitating human players, you cannot fundamentally exceed their capabilities. On the other hand, reinforcement learning is significantly more powerful. In the context of a game of Go, reinforcement learning means that the system plays moves that statistically and empirically lead to victories.

And so AlphaGo is a system that plays against itself. It uses reinforcement learning to create rollouts. The diagram here is the same, but there's no prompt. Because there's no prompt, it's just a fixed game of Go.

But it's trying out lots of solutions, experimenting with various strategies, and then reinforcing the plays that lead to a win instead of a specific answer. These successful strategies are strengthened. Thus, the system learns the sequences of actions that empirically and statistically lead to winning the game. Reinforcement learning is not constrained by human performance; it can significantly outperform even the top players, like Lisa Dole.

And so, they could have probably run this longer, but they chose to crop it at some point because this costs money. However, this is a very powerful demonstration of reinforcement learning. We are only starting to see hints of this diagram in large language models for reasoning problems. We won't get too far by only focusing on that; we need to go beyond it, set up these little game environments, and allow the system to discover unique reasoning traces or ways of solving problems that simply work well. Now, on the aspect of uniqueness, notice that when you're doing reinforcement learning, nothing prevents you from veering off the distribution of how humans are playing the game.

And so when we go back to this AlphaGo search, one of the suggested modifications is called Move 37. Move 37 in AlphaGo refers to a specific point in time where AlphaGo played a move that no human expert would play. The probability of this move being played by a human player was evaluated to be about 1 in 10,000. It's a very rare move, but in retrospect, it was a brilliant move. AlphaGo, in the process of reinforcement learning, discovered a strategy of playing that was unknown to humans.

And it was a very rare move, but in retrospect, it was brilliant. I recommend the YouTube video "Lee Sedol vs. AlphaGo Move 37 Reactions Analysis." This is kind of what it looked like when AlphaGo played this move. The value... That's a very surprising move. I thought it was a mistake.

When I see this move, anyway, so basically, people are kind of freaking out because it was a mistake. When I see this move, anyway, so basically, people are kind of freaking out because it's a move that a human would not play. AlphaGo played it because, in its training, this move seemed to be a good idea. It just happens not to be the kind of thing that humans would do. And so that is, again, the power of reinforcement learning. In principle, we can actually see the equivalence of that if we continue scaling this paradigm in language models.

And what that looks like is kind of unknown. So, what does it mean to solve problems in such a way that even humans would not be able to understand? How can you be better at reasoning or thinking than humans? Go beyond just a thinking human. Perhaps it means discovering analogies that humans would not be able to create, or maybe it involves a new thinking strategy.

It's kind of hard to think through. Maybe it's a wholly new language that actually isn't even English. Perhaps it discovers its own language that is much better at processing thoughts because the model is unconstrained to stick with English. So, maybe it takes a different language to think in or discovers its own language. In principle, the behavior of the system is much less defined. It is open to doing whatever works, and it is also open to slowly drifting from the distribution of its training data, which is English.

But all of that can only be done if we have a very large, diverse set of problems in which these strategies can be refined and perfected. A significant portion of the frontier large language model (LLM) research currently underway involves creating these kinds of prompt distributions that are extensive and varied. These serve as game environments in which the LLMs can practice their thinking. It's similar to writing practice problems; we need to create practice problems for all domains of knowledge.

And if we have practice problems—tons of them—the mouse will be able to reinforce its learning on them and create these types of diagrams, but in the domain of open thinking instead of a closed domain like the game of Go. There's one more section within reinforcement learning that I wanted to cover, and that is learning in unverifiable domains. So far, all of the problems we've looked at are in what's called verifiable domains. That is, any candidate solution can be scored very easily against a concrete answer. For example, if the answer is 3, we can very easily score these solutions against the answer of 3.

Either we require the models to box in their answers, and then we just check for equality of whatever's in the box with the answer, or we can also use what's called an LLM judge. The LLM judge evaluates a solution, obtains the answer, and scores the solution based on its consistency with the answer. LLMs are empirically good enough at their current capability to perform this task fairly reliably. Therefore, we can apply these kinds of techniques as well. In any case, we have a concrete answer, and we’re simply checking solutions against it, which we can do automatically without any human involvement.

The problem is that we can't apply the strategy in what's called unverifiable domains. Usually, these are tasks like creative writing, such as writing a joke about pelicans, composing a poem, or summarizing a paragraph. In these kinds of domains, it becomes harder to assess our different solutions to the problem. For instance, when writing a joke about pelicans, we can generate many different jokes, which is perfectly fine. For example, you can go to ChatGPT, and we can ask it to generate a joke about pelicans.

So much stuff in their beaks because they don't pelican in backpacks. What? Okay, we can try something else. Why don't pelicans ever pay for their drinks? Because they always bill it to someone else.

Ha ha. Okay, so these models are obviously not very good at humor. Actually, I think it's pretty fascinating because I believe humor is secretly very difficult and the models lack that capability. Anyway, you could imagine creating lots of jokes. The problem we are facing is: how do we score them?

Now, in principle, we could, of course, get a human to look at all these jokes—like I did just now. The problem with that is if you are doing reinforcement learning, you will be making many thousands of updates. For each update, you want to examine, say, thousands of prompts. For each prompt, you may potentially look at hundreds or thousands of different kinds of generations. Therefore, there are simply too many of these to review.

And so, in principle, you could have a human inspect all of them, score them, and decide that, okay, maybe this one is funny, and maybe this one is funny, and this one is funny, and train on them to get the model to become slightly better at jokes, at least in the context of pelicans. The problem is that it's just way too much human time. This is an unscalable strategy. We need some kind of automatic strategy for doing this. One proposed solution to this was introduced in a paper that presented what's called reinforcement learning from human feedback.

And so this was a paper from OpenAI at the time. Many of these people are now co-founders at Anthropic. This paper proposed an approach for doing reinforcement learning in unverifiable domains. Let’s take a look at how that works. This is the cartoon diagram of the core ideas involved.

So, as I mentioned, the naive approach is that if we just set an infinite amount of human time, we could run reinforcement learning (RL) in these domains just fine. For example, we can run RL as usual if I have infinite humans. I want to conduct 1,000 updates, where each update will involve 1,000 prompts. For each prompt, we’re going to have 1,000 rollouts that we’re scoring. We can run RL with this kind of setup. The problem is, in the process of doing this, I will need to ask a human to evaluate a joke a total of 1 billion times.

And so, that's a lot of people looking at really terrible jokes. We don't want to do that. Instead, we want to take the Reinforcement Learning from Human Feedback (RLHF) approach. In the RLHF approach, the core trick is the direction of feedback. We involve humans just a little bit, and the way we cheat is by training a whole separate neural network that we call a reward model. This neural network will imitate human scores.

So we're going to ask humans to score rollouts. Then, we're going to imitate human scores using a neural network. This neural network will become a kind of simulator of human preferences. Now that we have a neural network simulator, we can perform reinforcement learning against it. Instead of asking a real human, we're asking a simulated human for their score of a joke, as an example. Once we have a simulator, we're off to the races because we can query it as many times as we want, making the whole process automatic. We can now conduct reinforcement learning with respect to the simulator.

And the simulator, as you might expect, is not going to be a perfect human. However, if it is at least statistically similar to human judgment, then you might expect that it will produce some meaningful results. In practice, indeed, it does. Once we have a simulator, we can utilize reinforcement learning, and everything works great. Let me show you a cartoon diagram to illustrate what this process looks like. Although the details are not extremely important, it captures the core idea of how this works.

Here we have a cartoon diagram of a hypothetical example of what training the reward model would look like. We start with a prompt like "Write a joke about pelicans," and then we have five separate rollouts, each containing different jokes. The first thing we will do is ask a human to order these jokes from best to worst. In this case, the human concluded that this joke is the best and funniest.

Here is the reformatted text with correct capitalization and spelling: "So, the number one joke. This is the number two joke, the number three joke, four, and five. So, this is the worst joke. We're asking humans to order instead of giving scores directly because it's a bit of an easier task. It's easier for a human to provide an ordering than to give precise scores."

Now, that is now the supervision for the model. The human has ordered them, and that is kind of like their contribution to the training process. However, separately, we are going to ask a reward model about its scoring of these jokes. The reward model is a completely separate neural network, and it is likely a transformer, but it is not a language model in the sense that it generates diverse language, etc. It is simply a scoring model.

The sense that it generates diverse language, etc., is simply a scoring model. The reward model will take two inputs: the prompt, number one, and the candidate joke, number two. These are the two inputs that go into the reward model. For example, in this case, the reward model would take this prompt and this joke.

Now, the output of a reward model is a single number. This number is considered a score and can range, for example, from zero to one. A score of zero would represent the worst outcome, while a score of one would represent the best outcome. Here are some examples of what a hypothetical reward model at some stage in the training process might assign as scores to these jokes: a score of 0.1 is very low, while a score of 0.8 is quite high, and so on.

And so now, we compare the scores given by the reward model with the ordering provided by humans. There is a precise mathematical way to calculate this; basically, we set up a loss function and calculate a correspondence, then update the model based on it. However, I want to give you the intuition. For instance, regarding this second joke, the human thought it was the funniest, and the model somewhat agreed—0.8 is a relatively high score. But this score should have been even higher, right?

So, after an update, we would expect that the score should have improved and possibly increased to around 0.81 or something similar. In this case, there is a significant disagreement because a human evaluator believed this was number two, but the score is only 0.1. Therefore, this score needs to be much higher. After an update, in addition to this type of supervision, it might increase significantly, perhaps to around 0.15 or something like that.

And then here, the human thought that this was the worst joke, but the model actually gave it a fairly high score. You might expect that after the update, this would drop to maybe 3.5 or something like that. Essentially, we're doing what we did before: slightly nudging the predictions from the models using a neural network training process. We're trying to ensure that the reward model scores are consistent with human ordering. As we update the reward model using human data, it becomes a better and better simulator of the scores and orders that humans provide.

And then it becomes kind of like a simulator of human preferences, which we can then use for reinforcement learning. But critically, we're not asking humans to look at a joke one billion times. We're maybe looking at a thousand prompts and five relapses each, so perhaps 5,000 jokes in total that humans have to evaluate. They simply provide the ordering, and then we're training the model to be consistent with that ordering. I'm skipping over the mathematical details, but I just want you to understand the high-level idea: this reward model is essentially providing us with the scores, and we have a method for training it to be consistent with human orderings.

And that's how RLHF works. Okay, so that is the rough idea. We basically train simulators of humans and Reinforcement Learning (RL) with respect to those simulators. Now, I want to talk about, first, the advantages outside of reinforcement learning from human feedback. The first thing is that this allows us to run reinforcement learning, which we know is an incredibly powerful set of techniques, in arbitrary domains, including those that are unverifiable.

So, activities like summarization, poem writing, joke writing, and any other forms of creative writing really outside of math and code are interesting areas. Now, empirically, when we apply Reinforcement Learning from Human Feedback (RLHF), we observe that this method improves the model's performance. I have a strong perspective on why this happens, but I don't think it is well established why this is the case. You can see empirically that when you implement RLHF correctly, the resulting models perform slightly better. However, the reasons behind this improvement are not as clear. Here’s my best guess.

My best guess is that this is possibly mostly due to the discriminator-generator gap. What that means is that, in many cases, it is significantly easier to discriminate than to generate. In particular, an example of this occurs during supervised fine-tuning (SFT), where we ask humans to generate the ideal assistant response. As I have shown, the ideal response is very simple to write in many cases, but it may not always be the case. For example, in summarization, poem writing, or joke writing, how are you, as a human labeler, supposed to give the ideal response in these instances?

It requires creative human writing to do that. And so RLHF kind of sidesteps this because we get to ask people a significantly easier question as the data labelers. They're not asked to write poems directly; they're just given five poems from the model and asked to order them. This is a much easier task for a human labeler to perform.

And so what I think this allows you to do, basically, is that it enables a lot more accurate data because we're not asking people to perform the generation task, which can be extremely difficult. We're not asking them to do creative writing; we're just trying to get them to distinguish between creative writings and find the ones that are best. That is the signal that humans are providing—just the ordering—and that is their input into the system.

And then the system in our LHF just discovers the kinds of responses that would be graded well by humans. This step of indirection allows models to improve. That is the upside of RLHF. It enables us to run reinforcement learning, empirically results in better models, and allows people to contribute their supervision, even without having to perform extremely difficult tasks, such as writing ideal responses.

Unfortunately, Reinforcement Learning with Human Feedback (RLHF) also comes with significant downsides. The main issue is that we are essentially conducting reinforcement learning not with respect to actual human judgment, but with respect to a lossy simulation of humans. This lossy simulation could be misleading because it's just a simulation; it's merely a language model outputting scores. It may not perfectly reflect the opinions of an actual human with a functioning brain in all possible cases.

So that's number one. But there's actually something even more subtle and devious going on that really dramatically holds back our LHF as a technique that we can effectively scale to significantly smarter systems. That is that reinforcement learning is extremely good at discovering ways to game the model and the simulation. The reward model that we are constructing here provides the scores, and these models are transformers. These transformers are massive neural networks with billions of parameters.

And they imitate humans, but they do so in a simulated way. Now, the problem is that these are massive, complicated systems, right? There are a billion parameters here that are outputting a single score. It turns out that there are ways to manipulate these models. You can find types of inputs that were not part of their training set, and these inputs inexplicably receive very high scores, but in a deceptive way.

So very often, what you find if you run RLHF (Reinforcement Learning from Human Feedback) for a long time—let's say, for example, if we do 1,000 updates, which is quite a lot—you might expect that your jokes are getting better and that you're generating some real gems about pelicans. However, that's not exactly what happens. What occurs is that in the first few hundred steps, the jokes about pelicans probably improve a little bit, but then they dramatically fall off a cliff, and you start to receive extremely nonsensical results. For instance, the top joke about pelicans might end up being something like "da-da-da-da-da-da-da-da-da," which makes no sense at all. When you analyze it, you wonder why this would be considered a top joke. However, when you take that phrase and plug it into your reward model, you might expect a score of zero. Surprisingly, the reward model actually favors this as a joke.

It will tell you that the score is 1.0. This is a top joke, and this makes no sense, right? But it's because these models are just simulations of humans, and they're massive neural networks. You can find inputs at the bottom that kind of get into the part of the input space that gives you nonsensical results.

These examples are what's called adversarial examples. I'm not going to delve too much into the topic, but these are adversarial inputs to the model. They are specific little inputs that kind of slip through the cracks of the model and yield nonsensical results at the top. Now, here's what you might imagine doing: you might say, "Okay, this is obviously not a score of one."

It's obviously a low score. So let's add it to the data set and give it an ordering that is extremely bad, like a score of five. Indeed, your model will learn that it should have a very low score and will give it a score of zero. The problem is that there will always be an infinite number of nonsensical adversarial examples hiding in the model. If you iterate this process many times and keep adding nonsensical input to your reward model while assigning very low scores, you'll never win the game. You can do this for many rounds, and reinforcement learning, if run long enough, will always find a way to exploit the model.

It will discover adversarial examples. It will achieve very high scores with nonsensical results. Fundamentally, this is because our scoring function is a giant neural network, and reinforcement learning is extremely good at finding the ways to trick it. So, long story short, you should always run Reinforcement Learning from Human Feedback (RLHF) for maybe a few hundred updates. The model is improving, and then you have to crop it, and you are done.

You can't run too much against this reward model because the optimization will start to game it, and you basically crop it, call it, and ship it. You can improve the reward model, but you will eventually encounter these situations at some point. So, RLHF, basically what I usually say is that RLHF is not RL. What I mean by that is, RLHF is RL, obviously, but it's not RL in the magical sense. This is not RL that you can run indefinitely. In these kinds of problems, where you are getting a concrete correct answer, you cannot game this as easily.

You either got the correct answer or you didn't. The scoring function is much simpler; you're just looking at the boxed area and checking if the result is correct. Therefore, it's very difficult to manipulate these functions. However, gaming a reward model is possible. In these verifiable domains, you can run reinforcement learning (RL) indefinitely.

You could run for tens of thousands, hundreds of thousands of steps, and discover all kinds of really crazy strategies that we might not even think about for performing really well on all these problems. In the game of Go, there's no way to gain the winning of a game or losing a game. We have a perfect simulator. We know where all the stones are placed, and we can calculate whether someone has won or not. There's no way to game that.

And so you can do reinforcement learning (RL) indefinitely, and you can eventually beat even the least challenging scenarios. However, with models like this, which are gameable, you cannot repeat this process indefinitely. So, I kind of see reinforcement learning from human feedback (RLHF) as not real RL because the reward function is gameable. It’s more like a form of fine-tuning. It's a slight improvement, but it's not something that is fundamentally set up correctly, where you can easily insert more compute, run for longer, and achieve significantly better and more impressive results.

So it's not reinforcement learning in that sense. It's not reinforcement learning in the sense that it lacks magic. It can fine-tune your model and achieve better performance. Indeed, if we go back to ChatGPT, the GPT-4 model has undergone reinforcement learning from human feedback (RLHF) because it works well, but it's just not reinforcement learning in the same sense. RLHF is like a small fine-tune that slightly improves your model.

It's maybe like the way I would think about it. Okay, so that's most of the technical content that I wanted to cover. I took you through the three major stages and paradigms of training these models: pre-training, supervised fine-tuning, and reinforcement learning. I showed you that they loosely correspond to the process we already use for teaching children.

And so, in particular, we talked about pre-training being sort of like the basic knowledge acquisition of reading exposition, supervised fine-tuning being the process of looking at lots and lots of worked examples and imitating experts, and practice problems. The only difference is that we now have to effectively write textbooks for LLMs and AIs across all the disciplines of human knowledge, and also in all the cases where we actually would like them to work, like code and math and basically all the other disciplines. So, we're in the process of writing textbooks for them, refining all the algorithms that I've presented at a high level, and then, of course, doing a really, really good job at the execution of training these models at scale and efficiently. In particular, I didn't go into too many details, but these are extremely large and complicated distributed jobs that have to run over tens of thousands or even hundreds of thousands of GPUs. The engineering that goes into this is truly at the state of the art of what's possible with computers at that scale.

So I didn't cover that aspect too much, but this is a very serious endeavor underlying all these simple algorithms, ultimately. Now, I also talked about the theory of mind of these models. The key takeaway is that these models are really good, but they're extremely useful as tools for your work. You shouldn't trust them fully, and I showed you some examples of that. Even though we have mitigations for hallucinations, the models are not perfect and they will still hallucinate.

It's gotten better over time, and it will continue to improve, but they can hallucinate. In addition to that, I covered what I call the "Swiss cheese" model of language model capabilities that you should keep in mind. The model is incredibly good in many different disciplines, but it can fail randomly in some unique cases. For example, when asked what is bigger, 9.11 or 9.9, the model doesn't know the answer, yet it can simultaneously solve Olympiad questions.

And so this is a hole in the Swiss cheese, and there are many of them, and you don't want to trip over them. So don't treat these models as infallible. Check their work, use them as tools, use them for inspiration, use them for the first draft, but work with them as tools and be ultimately responsible for the product of your work. And that's roughly what I wanted to talk about. This is how they're trained, and this is what they are.

Let's now turn to what some of the future capabilities of these models may be, as well as where you can find them. I have a few bullet points on some of the things you can expect in the near future. The first thing you'll notice is that the models will very rapidly become multimodal. Everything I've discussed regarding technology will soon lead to language models that can not only handle text but also operate natively and easily with audio, meaning they can hear and speak, as well as images, allowing them to see and create visuals. We are already observing the beginnings of this trend, but it will all be integrated natively within the language model. This development will facilitate more natural conversations.

And roughly speaking, the reason that this is actually no different from everything we've covered above is that, as a baseline, you can tokenize audio and images and apply the exact same approaches for everything we've discussed so far. So it's not a fundamental change; we just have to add some tokens. For example, when tokenizing audio, we can look at slices of the spectrogram of the audio signal, tokenize those, and then add more tokens that represent audio, integrating them into the context windows and training on them just like we did before. The same applies to images; we can use patches and separately tokenize those patches. Then, what is an image? An image is simply a sequence of tokens.

And this actually kind of works, and there's a lot of early work in this direction. We can create streams of tokens that represent audio, images, and text, interspersing them to handle all simultaneously in a single model. That's one example of multimodality. Secondly, something that people are very interested in is that currently, most of the work involves handing individual tasks to the models on a silver platter. For example, asking them to solve a specific task for us.

And the model sort of performs this little task, but it's up to us to organize a coherent execution of tasks to perform jobs. The models are not yet capable of carrying this out in a coherent, error-correcting way over long periods of time. They are not able to fully string together tasks to accomplish these longer-running jobs. However, they are improving over time. What is likely to happen is that we will start to see what are called agents, which perform tasks over time. You supervise them, monitor their work, and they occasionally come to you to report progress and so on.

So we're going to see more long-running agents—tasks that don't just take a few seconds to respond but many tens of seconds or even minutes or hours over time. However, these models are not infallible, as we discussed earlier, so all this will require supervision. For example, in factories, people talk about the human-to-robot ratio for automation. I think we're going to see something similar in the digital space; we will be talking about human-to-agent ratios, where humans become much more like supervisors of agentic tasks in the digital domain. Next, I believe everything is going to become much more pervasive and invisible, integrated into tools and present everywhere.

And in addition, it’s somewhat akin to computer usage. Right now, these models aren't able to take actions on your behalf, but I think this is a separate bullet point. For example, ChashCapt launched the operator, which is an early instance where you can actually hand off control to the model to perform keyboard and mouse actions on your behalf. I find that very interesting. The last point I have here is just a general comment that there's still a lot of research to potentially conduct in this domain. One example of that is something related to test-time training.

So remember that everything we've done above and what we talked about has two major stages. First, there's the training stage where we tune the parameters of the model to perform the tasks well. Once we obtain the parameters, we fix them, and then we deploy the model for inference. From that point on, the model is fixed and does not change anymore.

It doesn't learn from all the activities it engages in during test time. It has a fixed number of parameters, and the only thing that changes is the tokens inside the context windows. Therefore, the only type of learning or test-time learning that the model has access to is the in-context learning, which involves a kind of dynamically adjustable context window based on what it is doing at test time. However, I think this is still different from humans, who are able to learn based on their experiences, especially during sleep. For example, while you sleep, your brain is updating your parameters or something similar, right?

So there's no equivalent of that currently in these models and tools. There are still many more unconventional ideas that need to be explored. In particular, I believe this will be necessary because the context window is a finite and precious resource. Especially as we begin to tackle very long-running multimodal tasks, which may include videos, these token windows will essentially start to grow extremely large—not just thousands or even hundreds of thousands, but significantly beyond that.

And the only kind of trick we have available to us right now is to make the context windows longer. However, I think that approach by itself will not scale to actual long-running tasks that are multimodal over time. Therefore, I believe new ideas are needed in some of these disciplines and specific cases, where these tasks will require very specific contexts. These are some examples of what you can expect coming down the pipeline. Now, let's turn to where you can keep track of this progress and stay updated with the latest developments in the field.

So, I would say the three resources that I have consistently used to stay up to date are, number one, LLM Arena. Let me show you LLM Arena. This is basically an LLM leaderboard that ranks all the top models. The ranking is based on human comparisons.

So, humans prompt these models, and they get to judge which one provides a better answer. They do not know which model is which; they are simply looking at which model gives the better response. You can calculate a ranking, and then you obtain some results. What you can see here are the different organizations, like Google or Gemini, for example, that produce these models.

When you click on any one of these, it takes you to the place where that model is hosted. Here, we see Google is currently on top, with OpenAI right behind. We also see DeepSeq in position number three. The reason this is a big deal is the last column here, which indicates the license. DeepSeq is an MIT-licensed model.

It's open weights. Anyone can use these weights. Anyone can download them. Anyone can host their own version of DeepSeq, and they can use it in whatever way they like. Therefore, it's not a proprietary model to which you don't have access.

It's basically an open weights release, and so this is kind of unprecedented that a model this strong was released with open weights. So, pretty cool. The team. Next up, we have a few more models from Google and OpenAI. And then, when you continue to scroll down, you're starting to see some other usual suspects.

So, XAI is here, Anthropoc with Sonnet at number 14, and then Meta with Llama over here. Llama, similar to DeepSeq, is an open weights model, but it's down here as opposed to up here. Now, I will say that this leaderboard was really good for a long time. I do think that in the last few months, it's become a little bit gamed, and I don't trust it as much as I used to. I think, just empirically, that a lot of people, for example, are using Sonnet from Anthropoc, and it's a really good model, but that's all the way down here at number 14. Conversely, I think not as many people are using Gemini, but it's ranking really, really high.

So I think you should use this as a first pass, but try out a few of the models for your tasks and see which one performs better. The second thing I would point out is the AI News newsletter. Although AI News is not very creatively named, it is a very good newsletter produced by Swix and friends, so thank you for maintaining it. It has been extremely helpful to me because it is very comprehensive. If you go to the archives, you will see that it is produced almost every other day, and it includes a mix of content written and curated by humans, as well as a lot that is constructed automatically with LLMs.

So you'll see that these are very comprehensive, and you're probably not missing anything major if you go through it. Of course, you're probably not going to go through it because it's so long, but I do think that these summaries all the way up top are quite good and have some human oversight. This has been very helpful to me. The last thing I would point out is just X and Twitter. A lot of AI happens on X.

And so I would just follow people who you like and trust to get all your latest and greatest on X as well. So those are the major places that have worked for me over time. Finally, a few words on where you can find the models and where you can use them. The first one I would recommend is that for any of the biggest proprietary models, you just have to go to the website of that LLM provider. For example, for OpenAI, that's chat.com, which I believe actually works now.

So that's for OpenAI. Now for Gemini, I think it's gemini.google.com or AI Studio. I believe they have two options for some reason that I don't fully understand. No one does. For the open-weight models like DeepSea, Klamma, etc., you have to go to some kind of inference provider of LLMs.

So, my favorite one is Together.ai. I showed you that when you go to the playground of Together.ai, you can choose from a variety of different models. All of these are open models of different types, and you can interact with them here as an example. Now, if you'd like to use a base model, it's less common to find base models even on these inference providers, as they are all targeting assistance and chat. I couldn't see any base models here, so for base models, I usually go to Hyperbolic because they serve my Llama 3.1 base, and I love that model. You can just interact with it here. As far as I know, this is a good place for a base model, and I wish more people hosted base models because they are useful and interesting to work with in certain cases. Finally, you can also take some of the smaller models and run them locally. For example, with DeepSeq, the biggest model, you won't be able to run it locally on your MacBook, but there are smaller versions of the DeepSeq model that are what's called distilled.

And then you can also run these models at a smaller precision. So, not at the native precision of, for example, FP8 on DeepSeq or BF16 in Llama, but much lower than that. Don't worry if you don't fully understand those details; you can run smaller versions that have been distilled and then at even lower precision. You can fit them on your computer, allowing you to run quite capable models on your laptop. My favorite place to go is LM Studio, which is basically an app you can get.

And I think it actually looks really ugly. I don't like that it shows you all these models that are basically not that useful. Everyone just wants to run DeepSeq, so I don't understand why they provide these 500 different types of models. They're really complicated to search for, and you have to choose different distillations and different precisions, which is all really confusing.

But once you actually understand how it works—and that's a whole separate video—you can load up a model. For example, I loaded up a Llama 3.2 Instruct 1 billion, and you can just talk to it. I asked for pelican jokes, and I could request another one, and it gives me another joke, etc. All of this happens locally on your computer, so we're not actually going anywhere else. This is running on the MacBook Pro. So that's very nice. You can then eject the model when you're done, which frees up the RAM. LM Studio is probably my favorite, even though I think it has a lot of UI/UX issues and is really geared towards professionals. However, if you watch some videos on YouTube, I think you can figure out how to use this interface. So those are a few words on where to find them.

Let me now loop back around to where we started. The question was: when we go to chashypte.com, enter a query, and hit "Go," what exactly is happening? What are we seeing? Who are we talking to? How does this work? I hope that this video has given you some appreciation for some of the behind-the-scenes details of how these models are trained and what is being returned.

So, in particular, we now know that your query is taken and first chopped up into tokens. We go to the token tokenizer, and this is where we place the user query in the appropriate format. Our query is inserted into what we discussed earlier: the conversation protocol format, which allows us to maintain conversation objects. This insertion results in a token sequence—specifically, a one-dimensional token sequence under the hood. ChatGPT sees this token sequence, and when we hit "go," it essentially continues appending tokens to this list.

It continues the sequence. It acts like a token autocomplete. In particular, it tokenizes into this list. It continues the sequence and provides us with this response.

So, we can basically just put it here and see the tokens that it continued with. These are the tokens that it continued with, roughly. Now the question becomes, okay, why are these the tokens that the model responded with? What are these tokens? Where are they coming from?

What are we talking to, and how do we program this system? That's where we shifted gears and discussed the underlying components. The first stage of this process, which consists of three stages, is the pre-training stage. This stage fundamentally involves knowledge acquisition from the internet into the parameters of the neural network. In this way, the neural network internalizes a significant amount of knowledge from the internet.

But where the personality really comes in is in the process of supervised fine-tuning. What happens here is that a company like OpenAI curates a large dataset of conversations—say, one million conversations across very diverse topics. These conversations are between a human and an assistant. Even though there is a lot of synthetic data generation used throughout this entire process, and a lot of large language model (LLM) help, fundamentally, this is a human data curation task with many people involved. In particular, these individuals are data labelers hired by OpenAI, who are given labeling instructions that they learn. Their task is to create ideal assistant responses for any arbitrary prompts.

So, they are teaching the neural network by example how to respond to arbitrary prompts. So, they are teaching the neural network, by example, how to respond to prompts. So, what is the way to think about what came back here? Like, what is this? Well, I think the right way to think about it is that this is the neural network's simulation of a data labeler at OpenAI.

So it's as if I gave this query to a data labeler at OpenAI, and this data labeler first reads all of the labeling instructions from OpenAI, and then spends two hours writing up the ideal assistant response to this query and giving it to me. Now we're not actually doing that, right? Because we didn't wait two hours. So what we're getting here is a neural network simulation of that process. And we have to keep in mind that these neural networks don't function like human brains do.

They are different. What's easy or hard for them is different from what's easy or hard for humans. As a result, we are essentially getting a simulation. Here, I have shown you a token stream, which fundamentally represents the neural network, with a variety of activations and neurons in between. This is a fixed mathematical expression that combines inputs from tokens with parameters of the model; they are mixed together to provide the next token in a sequence. However, this process involves a finite amount of computation for every single token. Thus, this represents a kind of lossy simulation of a human, constrained in this way. Whatever humans write, the language model imitates at the token level, utilizing this specific computation for each token in a sequence. We also observed that, as a result of these cognitive differences, the models suffer in various ways.

You have to be very careful with their use. For example, we saw that they can suffer from hallucinations. Additionally, we have the concept of the Swiss cheese model regarding LLM capabilities, where there are essentially holes in the cheese. Sometimes the models will arbitrarily do something illogical. So, even though they are capable of many impressive things, there are times when they just can't perform as expected.

So maybe you're not giving them enough tokens to think, and maybe they're just going to make stuff up because their mental arithmetic breaks down. Perhaps they are suddenly unable to count the number of letters, or maybe they can't tell you that 9.11 is smaller than 9.9, which looks kind of silly. Therefore, it's a Swiss cheese capability, and we have to be cautious about that. We have seen the reasons for this. But fundamentally, this is how we think about what comes back.

It's, again, a simulation of this neural network of a human data labeler following the labeling instructions at OpenAI. So that's what we're getting back. Now, I do think that things change a little bit when you actually go and reach for one of the thinking models, like O3 Mini High. The reason for that is that GPT-4.0 basically doesn't do reinforcement learning. It does do RLHF, but I've told you that RLHF is not RL.

That GPT-4 basically doesn't do reinforcement learning. It does use reinforcement learning from human feedback (RLHF), but I've told you that RLHF is not the same as reinforcement learning (RL). There's no magic in there; it's just a little bit of fine-tuning when you look at it. However, these advanced models do use RL.

So they go through this third stage of perfecting their thinking process and discovering new thinking strategies and solutions to problem-solving that look a little bit like your internal monologue in your head. They practice this on a large collection of practice problems that companies like OpenAI create, curate, and then make available to the elements. So when I come here and I talk to a thinking model and I input this question, what we're seeing is no longer just a straightforward simulation of a human data labeler. This is actually something new, unique, and interesting. Of course, OpenAI is not showing us the underlying reasoning and the chains of thought behind this process, but we know that such a thing exists, and this is a summary of it.

What we're getting here is actually not just an imitation of a human data labeler; it's something that is kind of new, interesting, and exciting in the sense that it represents a function of thinking that has emerged in a simulation. It's not merely imitating a human data labeler. It comes from this reinforcement learning process. Here, we are, of course, not giving it a chance to shine because this is not a mathematical or reasoning problem; it's more of a creative writing problem, roughly speaking.

And I think it's an open question as to whether the thinking strategies developed within verifiable domains transfer and are generalizable to other domains that are unverifiable, such as creative writing. The extent to which that transfer occurs is unknown in the field, I would say. So, we're not sure if we can apply reinforcement learning to everything that is verifiable and see the benefits of that in areas that are unverifiable, like this prompt. So, that's an open question. The other interesting point is that this area of reinforcement learning is still quite new, primordial, and nascent. We are just beginning to see the hints of greatness in addressing reasoning problems.

We're seeing something that is, in principle, capable of something like the equivalent of move 37, but not in the game of Go; rather, it pertains to open-domain thinking and problem-solving. In principle, this paradigm is capable of doing something truly cool, new, and exciting—something that no human has thought of before. These models can make analogies that no human has made. I think it's incredibly exciting that these models exist, but again, it's very early, and these are primordial models for now. They will mainly excel in domains that are verifiable, such as mathematics and coding, etc.

So very interesting to play with, think about, and use. And then that's roughly it. I would say those are the broad strokes of what's available right now. Overall, it is an extremely exciting time to be in the field. Personally, I use these models all the time—daily, tens or hundreds of times—because they dramatically accelerate my work.

I think a lot of people see the same thing. I believe we're going to witness a significant amount of wealth creation as a result of these models. However, be aware of some of their shortcomings. Even with reinforcement learning models, they will experience some of these issues. Use them as tools in your toolbox.

Don't trust it fully because it may randomly make mistakes. It can hallucinate or skip over some mental arithmetic, leading to incorrect answers. Sometimes it might not even count properly. Use these tools as part of your toolbox, but always check their work and your own output. They can be great for generating inspiration for first drafts and asking questions, but it’s essential to verify everything. By doing so, you will achieve great success in your work. I hope this video was useful and interesting to you. I appreciate your time, and I apologize for its length. Thank you, and I will see you later!