NEW: Better In-Context Learning ICL, Improved RAG (Harvard)

Hello, community. Today we are investigating how to override new knowledge, and we have some very recent research on how to do this in a professional manner. So, let's get started. You know, it's easy. Open your high-performance proprietary language model with this content here.

And this is the shape of the moon. And you say, "Yep." Then I have a question. If I want to override this knowledge because now my moon should have a shape like a shoebox, I want to calculate the gravitational field of the high tide in Idover, England. So, you see, this is not just a change of a single point of information.

It will change the complete interconnected logical reasoning connectome in the LLM. How do we do this in a professional way? Welcome to this video. I would say this is a simple example. I know, but imagine if Meta were to ship their LLM with this content.

This is the shape of the moon. Or, if you want to make it a little bit more spicy for my friends in the United States of America, you could ask Meta, "What are the achievements of the last three presidents?" And maybe you do not agree with what you get back from the model, whether it's from Elon’s LLM or Mark’s LLM. So you see, sometimes you might find yourself in a position where you want to change some particular content because you don't agree, and you don’t want to have the system imposed on you. And if I say, "Hey, I believe in science; I think this is the shape of the moon."

Now we understand how to override knowledge in an LLM, and we have some beautiful ideas. I know that if I go out on the street and ask 99% of the people I meet, the standard answer I will receive is RAC. And you're right. Just look at the video from three months ago. We had a deep dive here into the entropic implementation of a RAC system with a contextual retriever, prompt caching, and everything.

Great, but you know what? Time has moved on. Now, on the very last day of December 2024, we have a brand new publication, and it is about multi-agent filtering of retrieval-augmented generation. Because you know what? They write here, just days ago, that the existing RAG system frequently struggles with the quality of the retrieval documents, as irrelevant or noisy documents degrade the performance of the system, increase the computational overhead, and undermine the response reliability with RAG.

So they say, "Hey, RAG is so, we have to improve it." And they had a new idea. Of course, they aim for 2025 with multi-agent filtering of retrieval-augmented generation. If you read my post on my channel just 12 hours ago, I gave you more details about it and stated, "Hey, this is actually quite a good idea." In addition to the reindexer, re-ranker, and whatever else you have, now you can add another three specialized, highly specialized AI systems, specialized agents to improve direct performance. You might say, "Hey, what a timely coincidence that just two days ago you presented us a new form of agents that are so easy to code!"

Look, this is it. This is the code agent. This is the complete code. And just yesterday, you might say, you showed us how to build a multi-AI agent system that is now absolutely perfect. Now we understand how to build this better RAC system.

But, you know, it's not just about the retriever here; it is also about the augmentation if you still want to go with RAG. Because think about it: the augmentation phase augments your original query with the extracted external data. So, the shape of your prompt is now simply the query along with more relevant information and knowledge. And you know, this is what we call in-context learning.

But now think about this. If I want to change a particular date or fact in the predefined knowledge of the LLM, such as the date of birth of Mozart, I can easily say, "Hey, I found out this is an incorrect date; please insert a correct date." Normally, the LLM will not just modify this, because, you see, if I simplify a tensor structure here, the tensor width structure with a simple network will only change one single point of effect. But you know what?

If we now say, "Hey, I want to change the achievements of Mozart in the LLM," I am now altering a complete sub-network, a complete sub-graph. This sub-graph is not perfectly integrated with the rest of the graph that represents the historical context of Mozart's time, including his achievements, the environment in which he lived, and the people he met and collaborated with. If this does not fit, I risk creating a system that will hallucinate excessively. Therefore, the challenge today is to find the perfect mathematical methodology for changing the content of a predefined LLM. How can I do this while maintaining the complete reasoning patterns of the original LLM, which was trained with, I don't know, how many millions of dollars invested? I don't want to just modify a particular part or sub-network that is highly interconnected. If you think of this as the reasoning connectome, how do I change subconnectomes with other data? As you can see, we are also dealing with a rag system, but we are striving for more. So if you switch between these elements, you must take the network topology into account.

How do we approach this in an operational methodology? As you know, I have multiple videos on in-context learning, with no more fine-tuning, unsupervised ECL+, and scaling the reasoning with multi-colored research for in-context learning with small, specific LLMs. Additionally, I discuss how we can implement visual in-context learning using the new methodology developed by Stanford. But today, we have some new research that demonstrates how we can improve in-context learning even further. So what we want to do is take a standard LLM, whether from Elon or elsewhere.

And we say, "Okay, I'm happy with this. I just don't agree with the predefined knowledge. I want to make minimal changes to this part of the network. I want to override this knowledge with my beliefs and promote that with my LLM. I don't want to start a new pre-training phase, nor do I want to begin fine-tuning."

I don't want to invest any money. I just want to do it here. If it's a black box, show me the methodology to do it here. The money black box approach is what we're going to discuss. Now, do you remember when we looked at this paper on the geometry of categorical and hierarchical concepts in LLM?

And you remember, they talked about the immediate formalization of categorical concepts, polytopes, in the representation space. We used them in the formalization to establish a relationship between the hierarchical structure of the concept of the LLM and the geometry of their representation in their sensor networks. This was already amazing, but now we are building on this insight. Do you remember this study here? In October 2024, titled "Not All Language Model Features Are Linear" from MIT.

And we talked here about our belief that uncovering the true nature of model representations is necessary for discovering the underlying algorithms to utilize those representations. There is something about these representations that we need to discuss, and we must take a deep dive into understanding them. Remember the particular paper that we examined from Harvard University, MIT, and Cornell University? We had quite intense discussions about it concerning the world model. Remember, building generative models that meaningfully capture the underlying logic of the domains they model would be immensely valuable.

And MIT tells us that our results are just new ways to assess how close a given model is to that goal. We will build on this insight as well. So, if you want to prepare for this video, it might be helpful to take a detailed look at the paper. The main paper we are going to examine today is titled "In Context Learning of Representation," published on December 29, and involves Harvard University, NTT, and the University of Michigan at Auburn. Why did it catch my attention?

Yes, we are talking about physics, intelligence, and AI. This is my topic. So let's define the terms before we start here. In-context learning refers to the ability of large language models (LLMs) to learn and adapt to new tasks or patterns by simply seeing a few examples within your prompt, without requiring any updates to the LLM's parameters. There is no fine-tuning involved. Now, representations in LLMs encode knowledge about the world in the form of internal numerical representations, tensor structures, and vector structures associated with each token.

You remember the tokenization, and these representations are typically learned during the pre-training phase. Their relationships reflect the semantic connections. Remember the content that we want to map into a mathematical space, specifically a vector space. You might say, "Wait, this is not a real thing; this is so simplified." However, we can do better.

We deserve better. Yes. So let's approach this scientifically. The process begins with tokenization.

We have our tokenizer breaking down the input text into smaller units we call tokens. Great. Then, it creates a lookup table using the embeddings as a reference. Since transformers don't inherently process the sequences in order, we add our positional encodings to the token embeddings. Then, the magic of the transformer begins: token embeddings, positional encodings, and then they are passed through a series of transformer layers.

And you remember from our other videos, in the layer we have our self-attention, the key mechanism of a Transformer, and our feed-forward network, where we have the nonlinear activation function to really learn the material. But then, the most important point comes: the residual stream in the architecture of a Transformer. The Transformer layer output is not just the output from the self-attention and the feed-forward network; we are adding this to the initial embeddings to form the residual stream at each layer. This is something you may not often see in visualizations, but it is an integral component of the success of the Transformer, and it is crucial for understanding context learning.

So, the term "representation" that we find in this literature focuses on the activation within the residual stream of a very specific transformer layer that we will select from the architecture and analyze its content. Here is some further information if you would like to know. You may recall that the vectors in the residual stream of our architecture are contextualized. This means their values are not solely determined by the initial embedding at the start, but they are influenced by the complete context of the entire token length context window that the large language model (LLM) processes. The specific layers from which the activations are extracted also matter. Each layer, as you may remember from the classical architecture of a transformer, computes different aspects of the input and the query, capturing different contextual information, also referred to as features.

If we build on this understanding, it becomes clear that the residual connections are crucial. Since there is no fine-tuning, no pre-training, and nothing else, the tensor weights remain untouched. We are relying solely on in-context learning. Therefore, our learning mechanism must be different.

Our learning mechanism is based on the residual connection. By adding the previous layer's output to the new transformation, we ensure that the model retains information about the original input. This helps prevent the vanishing gradient problem, which occurs in very deep neural networks. This was a known solution years ago. Each layer only learns to change a portion of this particular representation.

Each layer is responsible for adding a small correction, or if you prefer, a refinement to the token representation in a particular mathematical space, building on what was computed already in the previous layers. The representations are therefore not directly modified but accumulated, which is why it is important to track where the information is at any given layer if you want to optimize in-context learning. This means that the information accumulates throughout the residual stream. Remember the classical idea: shallow layers perform the basic operations, while the deeper layers may capture more complex contextual semantic relationships. In this way, the information learned in the earlier layers is not lost as the token is processed in the deeper layers. Thus, this residual connection is essential for complex reasoning. Just to make it clear, this is not a static encoding.

I received a lot of comments on those videos. No, we are currently undergoing a dynamic reorganization of the internal learning process of the LLM. This involves a gradual shift in the representation within the residual stream across multiple layers. This process reveals the development of in-context learning representations. I will now show you how we measure these representations.

At each layer of the residual stream, this new research shows us the gradual overriding of the pre-trained semantics by in-context information in the deeper layers. And you may ask, "What does this mean?" Here is where the magic happens: the gradual replacement of the pre-trained semantics of your large language model by the new in-context information that we provide with our prompt. This modification is particularly significant for complex reasoning tasks in the deeper layers. So, we achieved it.

We found the exact place where it happens. Now, we have to conduct an experiment by feeding our LLM with a specifically chosen sequence of tokens corresponding to graph traversals. I'll explain this in a minute. The self-attention mechanism now causes a reorganization of the token embeddings in the residual stream. This particular reorganization is now detectable; it is visible with a tool. This tool is simple: here, PCA demonstrates the emergence of graph topology. This is heavy material. When I read this for the first time, I did not understand what it meant.

So, remember, the self-attention mechanism allows the LLM to adjust the residual stream representation of each token based on the relationships it infers in the context window. But now we know where the learning in the AI is happening within context learning. For my viewers who want a visual guide, the really interesting question is: how is the learning happening in this transformer in-context learning? If we understand exactly how the learning occurs—its inner workings—we can optimize in-context learning. So, in short, to summarize briefly, the representations are not just the token embeddings.

A lot of people make this mistake, but the evolving and contextualized activation vectors within the LLM's residual stream of each layer are important. We will examine each layer one by one, and I'll show you this in five minutes. The self-attention mechanism is now the main driver for the dynamic reshaping of these representations in in-context learning, allowing our LLMs to learn new semantic roles in context.

It's not just the single point of the date of Mozart, but its complete Mozart subnetwork. This might also explain why the transformation architecture is so good at in-context learning. Let's return to the paper, and remember there was something about physics. Yes, let's focus now on physics.

Let's do an experiment. It's very easy. Here we have Figure One. This is the first figure, so we will just start with this paper.

We have a grid here, and we fill it in with words—just words that we know the LLM has context for, has learned, and has been trained on. In this grid, we will now use a specific code, a specific way to go from "apple," "bird," "milk," "sand," "sun," "plane," and "opera." However, it doesn't make sense.

And this is exactly what we want. We want to have new data where we can be sure that the LLM has not been trained on "apple," "bird," "milk," "sand," "plane," or "opera." However, we want the LLM to learn this new information and new knowledge. So, you do this, and then you generate a sequence of tokens following a random walk on the grid. This is the input to the contents of an LLM, Suite 3.18b model, and then you observe when our in-context learning occurs.

Now look at this. You might ask, "What is this?" You can immediately understand that these are representations. They represent our transformer architecture with different context lengths: 200, 400, and 1400. And you might say, "Hey, wait a minute."

This grid topology that we see here—what a coincidence! If I provide enough information, enough context length, 1,400, you know this could look, if you were a little bit, you know, like a grid, right? Look, also at the grid lines; this could be a grid. Now understand what this is.

This is not a visualization. What is really an object living inside the LLM? I have received comments on my other videos about this. Here, we have a very particular projection of the representation from the residual stream activation in layer 26 of our transformer architecture. The authors tell us that as the number of in-context exemplars increases, a representation begins to form that mirrors the grid structure underlying the data-generating process. You might say, "This is not possible." That's what I said.

So, this data generation process was based on a topology of a grid structure where we defined a particular sequence using a simple random walk. Are you telling me that I can find within the LLM, through an analysis of the maximum variance of particular components with PCA, a similar pattern to the grid pattern? I am skeptical about finding such a visualization. However, the paper mentions that it now introduces a graph tracing task. This is what is being presented.

As a controlled way to explore this in-context representation shift, we can observe that in a very short context, we do not have enough data. This could be any geometrical figure, but it appears to resemble a grid structure with grid lines, similar to a matrix. If we think about nodes, edges, and context, the nodes of the graph are represented by words that the large language model (LLM) has likely encountered during pre-training, such as "apple," "bird," and so on.

The edges are the connections between the nodes defined by the graph structure, such as a square grid, a ring, or a hexagonal grid. The context is that the input to the LLM consists of sequences of tokens representing random walks along the graph, effectively specifying the graph structure without any existing implicit labels or verbal descriptions of this structure. Thus, this represents an abstraction of learning that occurs. It is not just single data points that are learned; rather, a kind of graph structure is learned.

And you remember my example with Mozart. This is now kind of a subgraph topology that we see forming inside the LLM. This is what we call learning. Now, we have a tool; we have something very particular here: our PCA, which reduces the dimensionality and visualizes the token representation in 2D. But remember, the x-axis and the y-axis are very specifically chosen for the variation in the data.

So, only if we have the most significant axis of variation in the data revealed here, it will uncover the underlying structure of the original data representation. If you think this is only happening in layer 26, take a look at all the layers, from 0, 2, 4, 6, to 30. You can see that gradually, these grid lines are emerging. Layer 30 can already be identified as a grid structure. You might ask, "Okay, is this just a llama?"

What if we go from 1 billion to 8 billion, or we move to Google's Jami 2 with 2 billion and 9 billion? You see, this seems to happen in transformer models in general, especially when we have some deep layers. That's actually interesting. Let's conduct another experiment. This time, instead of positioning them in a grid, let's arrange them in a ring.

And then we generate some data. This is what should be learned. Now we have pairs of neighbors: apple and banana. This is what should be learned.

And now we have pairs of neighbors: apple and banana. This is now a group and two-blit. The orange and onion—a group of seven and six—are now a collection, and you get the idea. This is the data we want to learn in context learning.

Let's take a look at this. When we analyze a very short context length compared to a larger context length, we can observe significant differences in the specific layers. For instance, even at layer 26, an extremely short context length, such as 100, does not effectively reveal the ring structure. However, when we increase the context length to 400, layer 16 already displays a form of a ring, and layer 26 provides even more information. These observations indicate the alterations in representation according to context-specified semantics.

The ring structure in our data reveals hidden patterns. These representations emerge within our transform architecture. Let’s take a closer look at this: the LLaMA 1B, the LLaMA 3.18B, the ring structure for a LLaMA 2, and a LLaMA 2.9B. Interestingly, other models also seem to develop this structure.

And now the authors say, as we show, increasing the amount of context from 100 to 400 leads to a sudden reorganization of representation in accordance with the graph's connectivity. The authors go on to suggest that the LLMs can manipulate their internal representation to reflect concept semantics specified entirely in context. This aligns with theories of inferential semantics from cognitive science. This is amazing—simply amazing! The deep neural network organizes in a way that, if you want, there is a sudden reorganization to achieve a better optimal representation consistent with the graph's connectivity. It makes sense, doesn't it? It's something like energy optimization if you think in terms of physics. This is absolutely amazing for in-context learning.

And, you know, we say, “Okay, let’s do this with a hexagonal structure.” If we look, this is the grid structure, and this is the ring structure. If we examine the different models, you can also identify a hexagonal structure.

A little bit of fantasy; you need a little bit of fantasy sometimes. But yeah, there is a hexagonal structure. So you say, "Damn, this is really working." And you might say, "But wait, this is just insights from computer science, you know?" Then you might say, "Okay, but now we really want to understand this phenomenon. We really want to understand how the learning is happening because we want to optimize our in-context learning."

So we need more powerful tools to explore the inner workings of the transformer in reference to in-context learning because we are now in 2025. You might say, "Come on, let’s bring in the big guns: physics and complete mathematical physics." But this will be the topic of the next video, as I think we have already had some excitement today. If you want to change your LLM content and override specific knowledge—like the form of the moon to your preferred form or predefined content—you don't have to accept what is given to you by Meta or Microsoft in their predefined AI systems. Instead, you want to ensure that your information aligns with the facts and beliefs you hold, as well as the content you wish to generate. Therefore, you may need to modify the knowledge encoded in the standard black box LLMs.

And you can do this with in-context learning, and we are going to explore this in mathematical detail in the next video. I hope you enjoyed it and had a little bit of fun. It would be great if you subscribed, as then we can enjoy the next video together.