Goodbye RAG - Smarter CAG w/ KV Cache Optimization

Hello, community! It is the end of 2024, and we will be removing RAC from our systems because we have a new system called CAG. Let's take a look. You may remember Turbo RAC from October 2024; it was a novel RAC system that utilized pre-computing and the storage of key-value caches for documents for the first time. They told us, "Hey, we are 10 times faster than any classical RAC system." Now, in December 2024, just four days before Christmas, we have a new publication: "Don't do RAC anymore, when Cache-Augmented Generation (CAG) is all you need for your knowledge tasks."

By our research colleagues from Taipei, Taiwan. So beautiful! Let's have a look at what this is all about. First, there's a message that goes here. Our findings suggest that as long as all documents fit within the extended context length, our traditional RAC system can be replaced by systems like O1 or Gemini. And you might say, "Unbelievable! How dare they claim this?"

But you know what? It is so easy because even Gemini 1.5 Pro, the old Gemini, already had a token count of 2 million tokens in context length. So easy. And you know, the other argument is, "Yeah, but we have a quadratic complexity with the transformer architecture for long sequences." Eight months ago, I showed you the solution: ring attention for real long context lengths.

Here is the text reformatted with correct capitalization and spelling: Or, if you want this video, also from eight months ago, Infinia Attention here, we already asked, "Hey, is this substituting our RAG system?" So we have no external data anymore. Can we integrate this and also show the other alternatives? But there's a second message in this beautiful new paper from our colleagues in Taipei. They say that with the LLMs featuring significantly extended context windows, and maybe your developer is already working with a 4 or 6 million token context length, this paper proposes an alternative to RAG: the Cache Augmented Generation (CAG), which completely bypasses the real-time retrieval of our classical RAG system.

And the idea is simple: we just preload all the relevant external resources into our model, and then we cache its runtime parameters. So, we have an original internal representation of the data from our self-attention mechanism in the decoder structure of a transformer. But I know what you're going to say. You might say, "Hey, wait a minute," and there are two arguments I can't anticipate. You might say, "But hey, Rack has access to an extremely large vector store, you know?"

This is, of course, about commercial stores, and you have to pay them. There is an established industry that we face, and this industry makes a case for commercial solutions, trying to convince you that vector stores are all you need. However, as you, my viewers, noticed in my last video—where I showcased some amazing insights—we don't necessarily need RAC. We can incorporate another component, ECL, here in INCA. You pointed out that nobody would put personal medical or financial data in a vector store anymore, especially where we have vector embeddings of our personal data. Nobody would do this.

And there are 10 or 20 responses from you, my viewers, who said, "Hey, nobody would do this in a rag anymore, right?" And you know, this is beautiful because now, suddenly, what used to be our old classical rag has transformed. We now have, if you want, a vector store, and we also have the idea that all the private, detailed personal data now goes into a secure database. It is no longer like it was at the beginning of this year, where everything had to be indexed, everything had to be embedded, and everything had to be a vector. Now that we break away from this idea, we can move forward.

So, you told me, and my viewers also told me, that you provided the primary knowledge on how to solve a new task in a vector store and in vector embedding. We build these mathematical spaces, and we are experts because we can create our own domain-specific vector mathematical spaces. Additionally, we have this detailed and secure database. If you think about it, we have a kind of separation of the know-how required to perform a new task. This is an instruction-based methodology derived from the top-down individual financial data of Mr. Green and Mrs. Blue. So, you see, we now have a new idea on how to handle this.

And this is great because if we now deny the vector store access to the private personal data, we won't need them embedded in a vector representation. This is a huge step forward because you know what's happening now. I showed you this in my last video when we went beyond RAC. The pirates of RAC. RAC is not secure. RAC introduces challenges such as retrieval latency, potential errors in document selection, increased system complexity, and it is not secure.

And if you can get rid of all of that, that would be an interesting step, especially at the end of this year. So, you might say, "Hey, wait a minute, is this kind of an aggregated view already something? Is this similar to the abstraction that Meta introduced when it moved from the large language model with next-token prediction to the large concept model?" Is this concept somehow related? Yeah, a little bit, but I want to show you here in more detail so that you better understand that we are working on a more general approach than Meta's concept. Yeah, and the second argument, of course, is, "Hey, but Reg is established."

And a lot of you write to me, "Hey, this is an established method. You show us new methods and new technologies." I understand that you might see 10 of us presenting new methods and technologies. You say you've seen 10 YouTubers, or even 20 YouTube channels, promoting commercial vector stores. All of those YouTubers tell you how great these stores are in their advertisements, where the vector stores pay them for the promotion, emphasizing how wonderful this registration system is. And yes, I understand that.

But, you know, if you were into science and if you had an open mind, you might ask, "Hey, what the hell is this CAG in detail?" So let's start here with a new meta-final that will replace your old RAC system. It is a new framework that leverages the extended context capabilities of the long-context LLMs. Remember, Gemini already has 2 million. Maybe you can work with stronger models to enable a retrieval-free knowledge integration of external knowledge.

A retrieval-free system. So in our RAG, we don't have a retriever anymore. We don't have vector embeddings anymore. We don't need them anymore. And you want what comes out of what is already implemented in TurboRAG.

Turborag is taking a step further by demonstrating that we no longer need RAG. The development of Turborag indicates that they have already implemented the next generation of knowledge integration. This was achieved by pre-computing the key value tensor structures and caching them. As a result, they were able to address the computational challenges and inefficiencies that were inherent in real-time retrieval within traditional RAG systems. Essentially, from within RAG, it dismantles RAG itself, allowing a new phoenix to rise from the ashes of the RAG system. So if you love RAG, this is the new CAG.

I hope I motivate you with this video. Okay, back to science. And you say, "Hey, wait a minute, key value? I know key value!" You know? These are present in the transformer architecture. If we look at our transformer block or a multi-head self-attention mechanism, we see it in detail. We calculate the tensor structures, we have the learned weight parameters, and we compute the specific attention for the attention heads using the softmax function. Then, we have our feedforward network.

Are those key values exactly what we calculate here? Yes, you are absolutely spot on. We are really deep into the self-attention mechanism of the transformer layers in the multi-heads, and this is what we're going to use for the next step. So, the key-value cache in the transformer model—remember, the transformer is a decode-only model. We have autoregressive generation, powerful mechanisms; yes, you know this.

But key values need to be recomputed classically for each token in autoregressive generation. For every next token, all the key values must be recomputed for every other token. Years ago, researchers discovered that by caching these key values in tensor structures, they could optimize key value storage. By storing the previously computed key values and reusing them for subsequent token generation, redundant computations could be reduced, significantly improving inference efficiency. Thus, key value caching is a helpful alternative, but we have not utilized it in the way that we will explore in this video. Classically, I would have explained the reasoning behind this in the context of transformers.

Why do we cache key-value pairs to accelerate the inference of our LLMs? If you look closely, you'll see that at each decoding step, the auto-generative model performs self-attention over the entire sequence of tokens to generate the next token \( x \) at time \( t + 1 \). This process requires the computation of the key and value matrices for all previously processed tokens since we need to consider the complete left side up to time \( t \). Notably, when generating the token for time \( t \), the LLM has already computed the key and value for all tokens in the interval \( t - 1 \). The key-value cache optimizes this process by storing the previously computed key-value tensor structures (matrices) for \( t - 1 \) and reusing them for the follow-up calculation. As a result, only the computation of key-values for the next token \( x \) at time \( t \) is required. This, of course, significantly improves the computational efficiency of our matrix multiplication and tensor operations. If you would like to see this in detail, here is the cached key-value matrix that we use.

This CAG methodology is available here; if you want, you can find the GitHub repository. It was published just days ago and updated 17 hours ago. However, nobody seems to be noticing it because no one is informing you about it. Yes, I understand.

If you're a YouTuber and you have a lot of revenue from doing commercial ads for Rex, System, and Vector stores, I understand that there may be some conflict of interest for you. However, in this repository, you have everything you need to try it out yourself. So, perhaps you are interested and will give it a try. Here's a very short summary.

So, what is this cash-augmented generation? Why don't we have a retriever anymore? Why don't we build extensive vectors here? Why can we go with a cached internal representation of the transformer tensor product itself? Because it is so much faster, so much better, and so much more performant.

Here’s the reformatted text with correct capitalization and spelling: "So, we have a novel approach to enhance the performance of large language models (LLMs). Unlike the traditional, old-fashioned retrieval-augmented generation, we preload all the relevant documents into the model during a pre-processing phase. While we build up the vector spaces with the retrieval, what we do now is, let’s say you are a company and you have your handbook of company processes. And let’s say, I don't know, 2,000 pages or whatever. Two million tokens is about 20 books."

So I think that even with two million, the classic model is free, and Gemini is free to use here. You already have a context length of two million tokens available. So, if you want to upload all the quality management information from your company or anything else, it is already sufficient. You can preprocess this in a phase, preload it, and your LLM will ingest all this external information and store it in an internal representation for future tensor calculations in the self-attention mechanism itself. We don't have to construct a workaround over a vector space or a vector store.

We directly stay within the operational mathematical framework of the LLM itself. We do not seek other representations. We work with this new self-attention from the external data and store it externally to the working director of the LLM. We then wait until a new query for this particular domain comes in, at which point we upload this preprocessed system.

So, how does it work? Step one: document. Gather all relevant documents, including your quality management documentation, process documentation, and any materials related to specific domains, such as mathematics or theoretical physics. Whatever you have, upload it. These documents are domain-specific and will be incorporated here.

Beautiful. Then you encode it here in this pre-calculation. So the documents are fed into the LLM. Nothing else happened—no training, nothing. Just feed it in.

You now have the key-value cache from the LLM encoder. You capture this tensor structure because it contains the essential self-attention information regarding the semantic content of all these documents in a format that you can feed into your LLM. Once you upload the next set of data, you will perform a cache reset. So, whatever you store now—the key-value cache of physics, the key-value cache of mathematical concepts, the key-value cache of operational information, or the key-value cache of quality management—you get the idea. Whenever a new query comes into the LLM, it knows that it doesn't have to rely solely on its original parametric knowledge encoded during pre-training, fine-tuning, and alignment.

I have an external pre-calculated key-value cache where all the quality management operations for this particular company are detailed. I have a complete process layout. Now, I need to upload this knowledge. I need to upload this external key-value cache into my operational memory, and I can immediately answer the query, with no registration involved at all. Isn't this nice? Because in the old-fashioned registration, you know what we had to do?

We had the vector embedding and retrieved the relevant documents in real time. The query is in a high-dimensional vector space, and then we had to reduce the vector space to find similar vectors to the query vector, using the encoding of the query vector. We calculated our cosine similarities, and then we had a retriever followed by a reranker. The first reranker processed 200 elements, while the next one only handled 100 elements.

My goodness, the system became increasingly complex, and you paid more for commercial solutions. This introduced more latency and potential errors, leading to a system that was hallucinating significantly. Now, this new CAG claims to have advantages because everything is preloaded into your standard system, eliminating the need to fine-tune it.

You just upload it in advance. You don't have vector embeddings, and you are much more efficient because everything is already waiting in the inference phase. If you have the inference, all this pre-computed key-value cache memory is stored, which contains the LLM's internal representation of all the relevant external knowledge related to the complete process management of your company, the complete quality management of your company, or whatever else is relevant, is now loaded into the LLM's working memory, given that you have a specific query. So, we have an inference computation where we upload some additional pre-computed key-value cache already in the internal representation of the LLM into the working memory. It is such a beautiful methodology, with a lot of ideas on how to further optimize this, offering advantages such as reduced latency, error mitigation, and simplified architecture. Now, I know that the central argument for this new cache augmenter generation is that, with the increased context window capabilities of the LLMs—2 million plus—preloading the relevant knowledge directly into the model context and caching its runtime parameters, the internal LLM runtime parameters provide a more efficient and robust solution than any classical rack. And now I know what you're going to say, because it's exactly what I thought.

Hey, if I work on an experimental 8-million-token context length LLM, is the maximum content length really a limiting factor for this methodology? As it turns out, it is not. Now, I'm trying to do a little teaser to show you what is happening, and I've never seen this on any other YouTube channel. I don't know why.

But look, just in the week of Christmas, what is happening. Here we have a new publication in our archive. On December 27, just yesterday, a survey on large language model acceleration based on key-value cash management was published. Or take a look at this research from Tsinghua University and Fudan University. They investigate a multi-matrix factorization attention mechanism in relation to cash management.

Here's a reformatted version with correct capitalization and spelling: "Or look at this from Tsinghua University and Fudan University. They examine a multi-matrix factorization attention mechanism, where they optimize these under stringent key value cache constraints. Or consider this one: here we have a pre-attention key value cache paper from December 24, 2024, involving the University of Maryland, Yale University, and others."

So, you see, look at this. For the day before Christmas, December 23, 2024, Google DeepMind is focusing on cash augmentation with a very specific offline coprocessor that operates solely on the model's key-value cache. This coprocessor enhances the cache by providing a set of lettered embeddings for improved decoding. They demonstrate that, with a frozen large language model (LLM), they can be augmented using an offline coprocessor that works on the model's key-value cache, resulting in further optimization of the LLM's performance—especially for reasoning-intensive tasks, allowing it to think more effectively. It's definitely a hot topic.

Look at this. This is Purdue University, December 18. We're already looking here for key value cache tensors with a 4-bit quantization. The researchers are all trying to reduce this to make it compatible with a consumer GPU system. Well, look at this.

This is from December 19. We have here a task-aware adaptive key-value cache compression method for the long context LLM. I just hope that I show you, because you don't see something on YouTube or don't see something on Medium, or you don't have a Reddit sub; whatever it is, it is there. This is happening. This is what is coming.

Look at this here: December 18, optimizing key-value cache compression in long context generation with a beautiful idea. Maybe I will do a separate video on this. Those are the papers I am currently reading, and I think they indicate a path to the future. But if you think this key-value cache is only relevant this week around Christmas with this publication, you're mistaken. Let's have an overview of the methodologies.

We can optimize this on different levels. We have token-level optimization, which focuses on fine-grained control of the key-value cache. We have model-level optimization, also centered around fine-grained control of the key-value cache. Additionally, we have model-level optimization through architectural innovations, specifically using attention grouping in non-transformer designs. Finally, we have system-level optimization, particularly regarding resource utilization and, specifically, memory optimization.

And these are really hard topics. The advanced memory management: how do you handle this with these LLMs so that you are not restricted to 2 million tokens, 8 million tokens, or whatever? Imagine a system where we would have almost unlimited memory management. So let's take a look at this. We will start with token-level optimization for key-value cache management.

From this literature and the overview of the survey, it is clear that these papers focus on optimizing the token-level performance of the key-value cache. They discuss selection, budget allocation, and the merging of different key-value caches. Additionally, if you consider chunking different topics and domains across multiple key-value caches and implement quantization, you will achieve low-rank decomposition. Yes, Laura, you have a unique perspective on this—static processes are essential.

These are all research papers that examine exactly this. We can explore key-value cache management at the model level. Consider token-level optimization, model-level optimization, and system-level optimization, as well as how data sets and benchmarks are interrelated. Alternatively, let’s take a look at system-level optimization for key-value cache management and pure memory management. This is truly something for specialists and experts.

The scheduling task, the hardware, and the design for which GPU you are using are all important considerations. What are the possibilities of the new GPUs? How can you integrate hardware and software optimization for this key-value cache management? For this advanced reasoning, where you do not need a RAC system anymore, the research is successful. The findings regarding key-value cache management and advanced reasoning are simply amazing. Flash attention to all the other things involved. There are topics I don't even know about.

I have never read about this, but I think it's amazing what is currently happening in cash optimization for our LLMs during these years. So, this is it. I hope this serves as a teaser for you to understand. Perhaps in 2024, we can all come together and say, "Okay, we don't need a rag system anymore," and further evolve.

And now we have new technological possibilities. We have new ideas for optimization. We don't need to build vector spaces. We don't need to create retrievers that operate on a vector embedding structure. We can have this in the internal tensor representation of the self-attention of our particular transformer architecture, whether we use a decoder-only model or another type.

And we have here a pre-computed cache, augmented generation of external knowledge with the inclusion of all the possibilities that the key value cache optimization at each and every level provides for our new AI systems in 2025. I hope there were some new ideas. I hope you are a little bit outside of your comfort zone. I definitely enjoyed making this video. Maybe you want to subscribe, or perhaps you'd like to see my next video, because I can tell you this was not the last video. We're going to talk about better reasoning for our LLMs, better performance for our LLMs, and what new technologies you can apply, hopefully real soon, for your particular AI applications.

It would be great to see you in my next video.