Chollet -- "o-models FAR beyond classical DL"

I've never been in the purely symbolic camp. If you go back to my earliest writing about program synthesis, I was saying we need deep learning-guided program synthesis. We need a merger of intuition and pattern recognition together with discrete step-by-step reasoning and search into one single data structure. I have said very repeatedly, for the past eight years or so, that human cognition is a mixture of intuition and reasoning, and that you're not going to get very far with only one of them. You need the continuous kind of abstraction that's provided by vector spaces and deep learning models in general, as well as the more discrete symbolic kind of abstraction provided by graphs and discrete search.

So why do your detractors see you as a symbolist when you're clearly not? Well, I'm not sure. I've been into deep learning for a very long time—since basically 2013. I started evangelizing deep learning very heavily around 2014. Back then, the field was relatively small. You know, especially given the chaos, I think I've done quite a bit to popularize deep learning and make it accessible to as many people as possible.

I've always been a deep learning guy, right? When I started thinking about the limitations of deep learning, I was not considering replacing it with something completely different. Instead, I was thinking of augmenting deep learning. I envisioned enhancing deep learning with symbolic elements.

What is your definition of reasoning? I don't really have a single definition of reasoning. I think it's a pretty loaded term, and it can mean many different things. However, there are at least two ways in which I see the term being used, and they are actually quite different. For instance, if you're simply memorizing a program and then applying that program, you could say that's a form of reasoning.

Let's say, for example, in school, you're learning the algorithm for multiplying numbers. When you're learning that algorithm and then you apply it during a test, that is reasoning. I believe it is indeed a form of reasoning.

And it's the kind of reasoning that large language models (LLMs) and deep learning models, in particular, are very good at. You're memorizing a pattern, and that's the time you're fetching a pattern and reapplying it. However, another form of reasoning occurs when you're faced with something you've never seen before, and you have to recombine and recompose the cognitive building blocks you have access to—your knowledge, and so on—into a brand new model, doing so on the fly. That is also reasoning, but it's a very, very different kind of reasoning and internalizes very different kinds of capabilities.

So I think, you know, the important question about deep learning models and large language models (LLMs) is not, "Can they reason?" There's always some sense in which they are doing reasoning. The more important question is, "Can they adapt to novelty?" Because there are many different systems that could just, you know, memorize programs provided by humans and then repeat them. What's more interesting is, "Can they come up with their own programs, their own abstractions on the fly?" Broadly speaking, I think programming from input-output pairs will be a widespread capability in the future, and that will be accessible to anyone because you won't need to write any code, right?

You're just specifying what you want the program to do. Then the computer will handle it, and if there's any ambiguity in your instructions—and there will always be ambiguity, especially if the instructions are provided by a non-technical user—you don't have to worry. The computer will ask you to clarify. It will say, "Okay, so I created basically the most plausible program based on what you told me, but there are some ambiguities here and there."

So, what about this input? Currently, I have this output. Does that look right? Change it. As you change it, you know, iteratively, you are creating this correct program in collaboration with the computer. Is there just a massive new type of architecture we need to build for this?

Yeah, I think we're going to need a completely new type of architecture to implement lifelong distributed learning, where you have many instances of the same AI solving different problems for different people in parallel and looking for commonalities between the problems and the solutions. Anytime they find sufficient commonality, they abstract these commonalities into a new building block, which goes back into the system and makes it more capable and intelligent. I think I've got it now. So, what you're building is a globally distributed architecture based on finding good solutions. Well, I can't really tell you exactly what we're building, but it's going to be cool.

Yeah, it sounds pretty cool. What's your theory on how O1 works? Well, you know, we can only speculate. I'm not sure how it really works, but what seems to be happening is that it is running a search process in the space of possible chains of thought, trying to evaluate which branches in the tree work better, potentially backtracking and editing if the current branch is not working out. It ends up with this very long, unsophisticated, and plausibly near-optimal chain of doubt, which basically represents a natural language program describing what the model should be doing. In the process of creating this program, the model is adapting to novelty. So, I think something like O1 is a genuine breakthrough in terms of the generalization power that you can achieve with these systems.

We are far beyond the classical deep learning model. Music, music, music, music, music, music, music, music—the end is sponsored by CentML, which is the compute platform specifically optimized for AI workloads. You might have seen the interview we did last month with Gennady, their CEO and co-founder. He spoke about some of the optimizations on CentML that they have implemented, making it dramatically faster than the competition. Anyway, if you're interested in experimenting with CentML, I will be doing a live session with one of their engineers in the coming weeks.

So feel free to jump on that live session, and you can ask their team any questions. 2 for AI Labs is a very exciting new research lab that has just started in Zurich. They are looking for amazing ML engineers to join their team. It's a small team, and it’s an exciting opportunity.

They are doing things like test time computing and inference time scaling. They want to get models to reason, to achieve that System 2 thinking—an amazing capability we've been discussing for years on MLST. If that sounds like you, go to TwoForLabs.ai. I mean, I love the show in general. I think of it as the Netflix of machine learning.

Well, you're going to learn about Arc 2. You'll discover what we've learned from running the Arc Price Competition in 2024. You may also pick up on some of my grant ideas regarding abstraction generation and how to implement AGI. Additionally, you'll find out a little bit about what François is doing next.

A little bit, a little bit. Wonderful. François Chollet, it's an honor to have you back on the show. Thank you so much. Thanks for having me. So, we're now at the end of the RKGI Prize, and you just released a technical report about it.

But can you reflect on the competition? Right. I think we learned a lot, and overall, it has been a big success. In 2024, we've seen a huge shift in the narrative around AI. Previously, the mainstream narrative was that we could just train a larger model on more data—100 times larger models with 100 times more data—and get something that's basically a general intelligence.

And while more recently, in the past year or so, there has been a realization that we were going to need something akin to System 2 reasoning, it became clear that this is not something that will simply emerge from pre-training larger models on larger datasets. It requires some form of integration into the system. Of course, you can use test-time search, program synthesis, and similar techniques to achieve this. I think ArchPrize has been a significant part of that narrative shift, and it's one of the reasons why ArchPrize has gained popularity, with many teams entering, lots of discussions surrounding it, and using it as a reference to assess whether we might have achieved DGI or not. By the way, this is not what ARC is intended to be; it has not been an indicator of whether we have achieved DGI.

We just intend it to be a tool, a research tool that encourages you to think about the right problems in order to focus on the right directions. The reason why it has been successful is that there was a latent demand for something like this. Many people had the intuition that plain LLMs (Large Language Models) weren't going to suffice and that we needed something more—either as a replacement for LLMs or some kind of superstructure around them that would implement a system as well. This intuition was floating around, and I think many people latched onto the Arc Prize as a concrete sign that their intuition was correct.

There were two flavors of the Arc Prize: a compute-restricted one and the main one. Can you reflect on the difference between the two in terms of the entries? Sure. The main track for the competition was for submissions that were self-contained.

And the reason why is primarily that we need to keep the private test set fully private, right? So we cannot send it to a third-party server via an API. In this context, people are submitting notebooks that will need to run on a virtual machine in less than 12 hours, and the VM has just a P100 GPU. This is equivalent to a total of roughly $10 worth of compute per submission. Additionally, we had the public leaderboard, which was targeted at frontier models.

And it was evaluated on a different set of tasks because, of course, we cannot evaluate on the private test set that we just leaked to a third-party server. So instead, it's evaluated on an entirely new set of tasks, which we should refer to as the semi-private test set. It's semi-private because it's not published anywhere, so it's not public. However, it's also not entirely private since we are actually sending the task data via API to our company and our philanthropic partners. Each submission on that leaderboard can use up to $10,000 in API credits.

So that's 1,000 times more in terms of compute worth than the private leaderboard. In the BERTAS basis section, it's a little bit less because we're evaluating more tasks. We're assessing the 400 tasks from the public evaluation and then the 100 from the semi-private evaluation, which is really what we're focusing on. Therefore, on a per-task basis, that's about 200 times more compute. What was really remarkable, and frankly quite shocking, is that the scores you end up seeing on the public leaderboard track closely with what you're seeing on the private leaderboard.

Like in both cases, we’re at about 55%. This demonstrates that it’s not just about allocating more compute resources to the benchmark. Compute is really a multiplier for ideas. Of course, if you have infinite compute, you can solve the benchmark in a very inefficient way, such as through brute force search, for instance. However, having better ideas significantly enhances the utility of your compute, right?

And so, this is why we end up with solutions consuming $10 worth of compute that are achieving only 55%. Meanwhile, there are solutions consuming $10,000 worth of compute that are doing the exact same thing; they are just not nearly as compute-efficient. So, which was the most successful method that worked across the board? There were already two categories of approaches that worked well.

So, one approach is deep learning-guided program synthesis, which is really my favorite personally and something I've been advocating for many years. Most people nowadays are engaging in deep learning-guided program synthesis using large language models (LLMs). They are utilizing LLMs to generate code and to iteratively debug code. Some individuals are attempting to use deep learning-guided programs with building blocks from a domain-specific language (DSL). I believe this is a very under-explored approach, but it should be an effective one.

The other category of approach has been test-time training, where you're going to use a large language model (LLM) to directly try to predict the solution given a task description. You look at a set of demonstration pairs and then at an input grid, directly attempting to generate the output grid. This process is called transduction, as opposed to induction. In induction, like program synthesis, you're trying to write down the program that will map the input grids to the output grids. In transduction, you're simply trying to predict the output grid directly. Of course, if you try to do this with LLMs and stay within the classical deep learning paradigm—where you have a large model that is pre-trained on tons of data, and then at inference time it is static, effectively just running the fourth class of the model—you cannot adapt to any meaningful amount of novelty. You are limited to memorizing patterns and test-time fetching and reapplying the patterns that you've memorized.

And to go beyond that, people have started using test-time training, which is the idea that you begin by pre-training a base model that understands arc and arc tasks. At inference time, for each new task you encounter, you will try to fine-tune the base model on the demonstration pairs to essentially recombine the knowledge contained within the latent space into a new model that is adapted to the specific task at hand. If you don't do this, if you forgo test-time adaptation, then LLM-based models and transformer models tend to be stuck below 10% accuracy, roughly speaking. However, when you incorporate test-time training, you unlock a dramatically higher level of generalization, allowing you to reach well into the 50%, 55%, and potentially even 60% range soon. The big question remains whether this aligns with the spirit of the challenge or not.

So I've done some interviews on this transduction with active fine-tuning, where you take the test instances, generate a dataset, and perform some augmentation. You fine-tune the model, and it performs really well. As a methodology to broadly generalize to many tasks, it is effective. However, the problem is that it still requires human supervision. You've emphasized from the very beginning, in your measure of intelligence, that we need developer-aware generalization. This means that we can't have a human supervisor specializing in every downstream task. We need to create a system that can generalize to tasks the developer wasn't aware of.

So by that metric, do you feel that it's not in the spirit of ARK? No, I think it's a completely legitimate way to approach the challenge. I also believe it represents a very significant breakthrough in generalization power and in the ability of these models to adapt to something they've not encountered before. Additionally, I don't think the supervision we are discussing is truly provided by humans; rather, you're using the demonstration pairs to fine-tune the model.

This is actually fairly autonomous. Of course, this needs to have been programmed by a human aware of the task format. However, the same would be true for a program induction-type approach. So I think it's very much in the spirit of the challenge. Furthermore, I believe it does demonstrate a legitimate breakthrough in generalization. Regarding the legitimacy aspect, it stands to reason that we engage in some form of active inference.

Of course. So we're always adapting to novelty, building new models, and so on and so forth. So what is the difference between transductive active fine-tuning and what we do? Well, I'm not sure what we do exactly. Of course, we are doing active inference. What does that mean exactly?

So, what algorithms and data structures are we leveraging? We don't know. Therefore, I can't really tell you what the difference is. However, it does seem to me—I will say one thing—when you're doing test-time training with an LLM, you are allowing a gradient descent process to perform the knowledge recombination.

To adapt to novelty, it is necessary to take the knowledge that you have and recombine it, and there are multiple different approaches you could use to achieve that. You could conduct a program search, where the element that is adapted to the new task is a program, and you are building this program through a search process. Additionally, something like what the GPT-3 model from OpenAI is doing is similar, where you are effectively searching in the space of chains of thought and writing down this chain of thought, which essentially serves as a natural language program for the model to execute. This search is conducted through a sort of zero-shot style search process, similar to a tree search process. So that's one perspective.

You can also just use discrete program search to write down the program. The program is the artifact that models the task at hand. Alternatively, you can try to modify the weights of the model and the representations of the model to create a new model that is adapted to the task. In this case, the artifact that is adapted to the task is the model itself. This is what test time training does, and it accomplishes this via gradient descent.

And, well, my take is that I don't think humans adapt to novelty by recombining what they know via gradient descent specifically. I think the level at which we represent knowledge, especially in the context of solving arc puzzles, is much more abstract and symbolic in nature. The way we combine it is much closer to function composition than to what you can achieve with gradient descent. In general, I don't think it's a good idea to try to use gradient descent as a replacement for a programming process. So, I'm actually more of a fan of what a system like O1 is doing than trying to perform the test and training.

We'll get to O1 in a second, but there are folks who are really bullish on this transduction thing, and I think you and I agree that the reason it doesn't work in principle is that language models are just finite state automata. They don't have this compositional generalization, even in principle. We know there's evidence that transformers on their own can't perform basic tasks like copying and counting, among other things. However, some people are optimistic because they believe we could improve the architecture so that it could handle those types of tasks. In that case, a transductive approach might work in the future. I mean, would you rule that out? No, I think it's entirely plausible. As you point out, even with lots of data, there are many algorithmic tasks that you cannot train a transformer to do, right?

Or even if you can, it will learn a solution that does not generalize very well. It will work on inputs that are pretty close to what it has been trained on. If you try an input that is very far away from the training distribution, it will just fail. And while some people think we can move past these limitations by making architectural tweaks, well, you know what?

They are right. It is always possible to take a deep learning model, modify the architecture to incorporate strong structural priors about the algorithmic problem you're trying to solve, and then use gradient descent to find a solution that generalizes. However, this process relies on a human engineer to first understand the task at hand and translate that understanding—essentially a symbolic understanding—into a better architecture, one that is isomorphic in important ways to the causal structure of the problem. Consequently, if you want to autonomously adapt to novelty, you cannot solely depend on a human engineer to intervene and modify the architecture.

The process has to be fully autonomous. And so the question is: can you create an architecture search or architecture generation machine that will take a problem, identify the key elements of the problem structure that you need to incorporate into your architecture, and then generate the architecture? If you can do that, then sure, maybe you can leverage that to achieve strong generalization. However, I think that this problem is at least as difficult as program search in the general case. It's also pretty overfit, as it turns out.

So, you know, on the public leaderboard, we are evaluating on the public evaluation set, but we are also evaluating on this semi-private evaluation set. The reason we're doing that is to test for overfitting. Some solutions might be overfitting to the public data, and it was actually the case for their solution. They scored something like 10% lower on the semi-private set.

And other solutions, particularly those based on program synthesis, do not exhibit that drop at all. They are actually scoring the exact same on both sets. This suggests that whatever they did was, in some important way, overfit to the data that they had. Can we touch on that as well? You mentioned that when you take an ensemble of all the original Kaggle 2020 competition results, it comes to about 49%.

That's right. Tell me more. In the very first ArcGGI competition on Kaggle back in 2020, the highest score by a single submission was only 20%. That was the winner, Icecuber, who was using basic brute-force program enumeration. However, if you looked at all the submissions in the competition and assembled them together, you would see a high score of 49%, which would have been considered state-of-the-art until very recently.

And that was, again, about four years ago. What that tells you is that there's roughly half of the private test set that is easily brute-forceable because every single entry in the competition back in 2020 was utilizing some kind of brute-force program for enumeration. That is not easy and certainly not the type of solution we are looking for. The fact that employing this kind of approach at scale could yield a score of 49% is a strong indication that the benchmark is flawed. Today, if you look at the 2024 competition, the state of the art for any single submission is around 55%.

So, you could say, okay, we are very far from solving the benchmark since the bar is 85%. We also know that humans can solve very closely to 100%. If I showed you the private test set, you would probably achieve around 97%, 98%, or 99%, for example. If you take an ensemble of everything that was submitted in the competition in 2024, you would reach a high score of 81%, right? That's pretty close to what we are looking for.

And I don't think that what was tried in the competition this year is really close to AGI in the meaningful sense. The ensemble of everything is still not close to AGI. It just shows that scale, that brute force compute scale, will eventually crack the challenge. What that indicates is that the benchmark is flawed and is now close to saturation, which means we need something else. That's why we are working on Ark2.

What that indicates is that the benchmark is flawed and is now close to saturation, suggesting that we need something else. That’s why we’re working on ARC 2. ARC 2 is not exactly a novel idea; it is not a reaction to the results we obtained in 2024. I first publicly announced ARC 2 in early 2022, which was some time ago. Back then, it was in response to the 2020 competition results, where I became aware that the benchmark had flaws and was not diverse enough in terms of task diversity.

Not every task was entirely unique in the orders or the amount for denancy. It might not have been challenging enough. So, I wanted to create version 2. Back in 2022, in partnership with Lab42 in Davos, we started crowdsourcing a variety of new tasks. Since then, we've continued to develop many more new tasks.

We have started filtering the data, trying to analyze which problems were difficult for humans and which were difficult for AI. We have collected a lot of human testing data as well. For this, we hired people to actually attempt to solve these problems. This provides us with valuable information about the number of attempts made for different tasks and how many times each task was solved by various individuals.

And you can turn this information into a kind of human-facing difficulty rating. Then you can try to cross-correlate it with what AI can do. We are going to be releasing Arc 2 early next year. It's going to address all of the flaws of Arc One, so it's going to have slightly more data.

It's going to be leveraging three sets: the public evaluation, the semi-private evaluation, and the private evaluation. One problem with the first few competitions on Kaggle is that we've been reusing the same private test sets across every competition. Any time anyone made a submission, they could immediately see their score on the private evaluation. This can actually lead to some amount of information leakage about the private test set over time.

And, in fact, there are very well-documented techniques about how you can actually start to reconstruct, given enough submissions, the contents of the priority value. So, we want to avoid this in the competition next year. The way we are doing this is that we will be evaluating using a semi-private evaluation during the competition. When you submit something, you will receive your semi-private evaluation score. Only at the very end of the competition, to create the final leaderboard, will we actually run the submissions on the fully private test set.

And another nice thing about doing that is that it enables you to make direct apples-to-apples comparisons between the Kaggle leaderboard, the Cargo leaderboard, and the public leaderboard with all the frontier models. Of course, the amount of compute may not be the same, but the scores will still allow for apples-to-apples comparisons. You'll be able to say, "Okay, so keep an eye on this test set; they're scoring as well as model 01, or whatever." For instance, you know that Kaggle 2021 has an aggregate score of 49%. First of all, which models were more enforceable?

Because I suppose one way of looking at it is certainly with the mosaic patterns that, in the solution space, there's an exponential number of combinations. But, of course, in the compositional space, it might be brute-forceable. However, we're also starting to see with the induction and transduction methods that they can solve different types of problems. Additionally, when you consider human abilities, humans can solve different types of problems as well, and there are all these overlapping Venn diagrams. How do you think about that?

Yes, one of the most interesting findings of the 2024 competition is that program induction and transduction using deep planning models, typically in "nalamm," will lead to solving substantially different sets of tasks. This was surprising to me, but in retrospect, it makes sense, especially when you start analyzing the tasks and understanding what makes them different. Some tasks are very perceptual in nature; they are essentially pattern cognition problems. This is the type of task that is effectively solved using transduction methods.

And there are other tasks that are effectively pattern cognition problems. This is the kind of challenge that you solve well with transduction methods. There are also other tasks that are much more algorithmic in nature, much more discrete. For these, you cannot provide an easy solution based on pattern recognition, but it is very straightforward to simply write down an algorithm to produce a solution. Conversely, if you look at puzzles that are very perceptual, it can be quite challenging to write solution programs for them, right?

Because the program would have to formalize a lot of perceptual concepts that make intuitive sense to us, but are actually very difficult to express in program form, right? It's kind of like trying to recognize something outside an arc. Let's say, for instance, trying to recognize the letter "A." If you're just looking at letters, you can immediately see it. If you have pre-training knowledge—visual knowledge of the shape of letters—this becomes a trivial problem.

You just recognize layers; it's just what you do with spatial recognition. But if you try to write down an algorithm that would take a handwritten letter and tell you if it's an "A," for instance, it's actually a tremendously difficult problem. A couple of thoughts on that: First of all, regarding the Kevin Ellis paper on induction and transduction, it seems that for political reasons, they wanted to find an explicit inductive function. In their ensemble method, they first searched for inductive functions by "greenblatting" it, and then they failed over to transduction. It occurs to me that just because we can't write a Python program to recognize a digit, it doesn't mean that such a Python program couldn't exist.

And shouldn't we be thinking about making models generate such a program? Maybe the reason they can't is because nothing like that is in distribution. I think the reason why it's difficult is that we are talking about an input space that is structurally continuous, and where decision boundaries are misleading, essentially. So we're dealing with a problem that is fundamentally a pattern recognition problem. Neural networks are intrinsically a good data structure to approach this type of issue.

And discrete symbolic programs are not. So I really think it depends on what problem you're looking at. There are problems where vector spaces are the right data structure and problems where symbolic discrete programs are the right data structure. An interesting thought experiment is that there must exist a Python program that does what an MNIST model does. Sure. And what is the simplest possible representation of that program?

Would it still be ridiculously complicated? I think the simplest representation of that program will look a lot like what the candidate is doing, to be honest. Right? Yeah, I think that's the clincher, isn't it?

So there's no way of decomposing it into a much simpler version. Yeah, I think it really depends on the nature of the problem. For some problems, program synthesis is just a bad idea, right? Perceptual problems are certainly in this category.

And the other way around, for some problems, trying to use a pattern cognition machine is just a bad idea—specifically, problems that are algorithmic in nature. How do you think we could effectively combine induction and transduction methods? Well, the way Kevin and his team are doing it in their paper is basically to start with induction and then fall back on transduction when it doesn't work.

And I think that's a very smart strategy because induction is actually formally verifiable. You can try to run your candidate program on the demonstration pairs that you have access to and see, well, first of all, does it run? And second, does it get you the right result? If it does, you can have a fairly high degree of confidence that it is going to generalize, right? Whereas, you know, when you're doing transduction, it's more like you're guessing where the answer might be, and you don't really have any way to verify that this is the right guess.

One thing you can do is simply increase the sample size. I can make many independent guesses, and then you can look at which answers come up the most often. However, you are essentially making the assumption that wrong guesses are all going to be wrong for different reasons, so you will end up with different incorrect answers. On the other hand, correct guesses are all going to be correct for the right reasons.

So the correct answer will show up more often. But really, you have no way to ensure 100% that your guesses are correct. So it's much smarter to start with induction because then you can have a high degree of confidence that the solution you have is the right solution, and then fall back when it just doesn't work. Essentially, induction is the method you should prefer, and you should only use transduction if it isn't working. Should we think of them as being completely different?

So hypothetically, if you used a shared model for doing induction and transduction, do you think there could be some crossover between them? Absolutely. In fact, this is something that some people in the competition have tried. They are using the so-called OmniARC approach, where the team uses the same model to solve a range of different ARC-related tasks, including writing down programs, interpreting programs, performing transduction, generating more inputs, and so on. All these different tasks are tackled with one single model, and that does lead to better representations for the concepts found in ARC.

Can you give me some more intuition on that? In a sense, you can get the network to consider the symbolic version at the same time as the solution. The basic intuition is that if you look at the same problem from different angles, you are more likely to understand the true nature of the problem. This is especially true if your data structure of choice is a neural network, as neural networks have a tendency to latch onto noisy statistical regularities. If you're only targeting one problem and using just one input modality, you are much more likely to overfit on the noise within that problem.

But if you're forcing the same representations to work across many different views of the problem, it acts as a mechanism to improve understanding. First of all, you gain better information about the problem, right? This is because there is some knowledge transfer and information exchange between tasks, such as trying to predict the output grade and generating more input grades, for instance. Additionally, it serves as a regularization mechanism, where the noise learned from one modality is countered by what you're learning from another modality. Another approach, from someone named Clément Bonnet, involves better searching in the latent space. This doesn't necessarily mean brute-force searching, but actually taking the time to explore the latent space thoroughly before making a prediction. What do you think about that?

Yeah, this is actually personally one of my favorite papers that was submitted in the competition. It presents a very original idea: learning a latent space of programs. Then, at test time, adaptation to the problem at hand is necessary. You need to recombine your knowledge. While some people use test-time training, others utilize search.

And with Clement Bonnet and his colleague, Macfarlane, they are doing something very original. They are learning this latent space of programs, and at this time, they are performing gradient descent in the latent program space to navigate around and find the point in the space that best matches the task. I think this is a great idea. There are many ways, I believe, to enhance the concept, but it's a very, very original take on test-time adaptation that is neither search nor fine-tuning, so I like it a lot.

Would you still call that process thinking? Is thinking a System 2 process, or would you consider that thinking as well? It is a form of test-time search; however, it's not a discrete search—it's based on gradient descent. So, sure, I don't see why you couldn't implement some form of System 2 processing with that. Yeah, I wonder whether that breaks the analogy with human thinking, particularly regarding perceptual deliberation.

It's quite an interesting category. It involves deliberation in latent space, yes. So, I think one way to improve the process is that you could also try to decode your latent programs back into a symbolic discrete form. Then, you can start doing a local discrete search around the decoded programs. The benefit of that approach is that you would have the ability to actually run the programs and verify whether they work.

And as long as you stay in latent space, even if you're performing this gradient descent-guided search within latent space to find the best possible point that represents a target program, you are very much limited to guessing. You have no way to assert that what the latent space is telling you matches the reality on the ground. Therefore, the ability to decode back into real program space and run these programs would be a very good addition to the system. Are there any potential issues with that approach? I suppose it helps if the latent structure is quite homogeneous and the modes are easy to find.

Could it be improved? What do you mean by that exactly? He performs gradient search over the latent space to find some optimal position, and then he conducts inference from there. However, wouldn't that work better if the space were relatively convex, and not so well if it were very heterogeneous? Of course, to perform gradient descent effectively, you need a relatively smooth surface. I think that's why they are using a Variational Autoencoder (VAE).

And I'm not sure if they've tried directly learning a program embedding space where one point represents one program. As it turns out, this is not what they are doing. They are using a variational autoencoder (VA). The reason for this is that when you're using a variational autoencoder, you are learning much more structure and creating smoother latent spaces. I think this is key to making test-time grading work decently. So, considering other unexplored avenues of ARC, maybe a better way to ask this is: If you spent a year working on ARC, what would you do?

I would be working on deep learning-guided program synthesis, and I think the way people are currently approaching it is incorrect. Many are leveraging large language models (LLMs), which makes sense because they are powerful tools that contain a wealth of useful knowledge applicable to various problems. We've invested billions of dollars into creating these tools, so not utilizing them would feel like missing out on a great deal of power.

But I think it is not the right approach to treat program synthesis as just token-by-token code generation. The right way to think about a program is as a graph of operators, and program synthesis is basically a tree search process. I believe you're better off trying to use deep planning models, and in particular, large language models (LLMs), to guide that tree search process. This is not something that many people are focusing on today, but I think it would be closer to the right approach.

And another thing that people are not doing today, but should be, is that if you look at the way humans solve arc puzzles, there aren't many different solution programs in their minds. They're only trying a few. I think humans have the capability to first describe a model in their minds and use this model, essentially describing a grid, for instance, in terms of the objects it contains, their properties, and their relationships with other objects, with a particular focus on causal relationships. You can use these descriptive models to constrain the search space when you are finally looking at input-to-output programs, and that's the reason why we only need to consider a handful of programs before finding the correct one. So in that sense, it might be possible to do enough modeling of the task to almost make search irrelevant—to almost entirely remove the need for a neutral search. It's so interesting what you just said.

We should meditate on that just for a second. First of all, many people I've interviewed this week, especially in the neural evolution space, under Jeff Clune for example, are latching onto large language models (LLMs). They argue that we need to have a measure of interestingness or novelty. LLMs, because they are trained on a vast amount of data, capture our instincts and intuition really well. They are a great way of generating programs. However, you mentioned that it's not such a good idea to simply generate the program.

The next step of evolution is guiding the search. I think we're starting to see this enlightenment in the use of large language models (LLMs) in many commercial software applications. For example, the original implementation of an LLM in an app meant simply having a chatbot integrated into the software. Now, with tools like Cursor, they're exposing a low-level API and utilizing tool use, where the LLM is actively guiding the low-level API interactions. You are advocating for a similar evolution here, where the LLM actively guides the discrete search process rather than just generating code.

That's right. The idea is that by creating your program through this iterative discrete search process, you actually have the ability to make targeted modifications to your program graph, which would be significantly harder to achieve if you only treated the program as a sequence of tokens. Additionally, you are changing the nature of the space in which you are making addition decisions. Graphs are the natural data structure for representing programs; programs are not merely sequences of tokens.

Another thing we're seeing is an intermediate solution. Kevin Ellis did something called remixing, where you have 100 handcrafted solutions. For every single arc task, you have a generator that can create new instances, allowing for the generation of sprites, layouts, and so on. They also implemented an expansion using retrieval-augmented generation, which demonstrated an implicit form of library learning by mixing all the solutions together. Is that an intermediate solution to what you're talking about? Not quite.

I think that's a separate avenue. The idea is that, in order to get large language models (LLMs) to perform well on the dataset, you need to expose them to as much and as dense a sampling as possible of ARC space. Of course, there aren't many ARC tasks available, so new ones have to be created. One easy way to make new tasks is to leverage an LLM to extract the programmatic concepts found in, for instance, the training set and then remix them into new tasks. However, at the end of the day, this might lead you to see the other feeling.

And this is exactly what you're seeing with your solution. There's been an interesting evolution where, even in DreamCoder, Kevin had an explicit concept of awake and asleep, dreaming. In his newer work on learning by example pairs, he has kind of made that dream sleep process implicit. It feels like we might achieve some of the same outcomes by developing a proxy or implicit version of it. Yes, definitely.

Yeah. Are there any other avenues for Arc that you're interested in? By the way, I think you spoke with George from Symbolica, and he had this kind of program verification approach that he mentioned discussing with you. Yes. What we discussed is that what you described to me is basically the idea of using a symbolic process to transform a problem definition or task into a deep learning architecture, and then training that deep learning architecture.

I think that's a very, very original approach. I don't think there's anyone else, to the best of my knowledge, working on something similar. So I'm very curious about what he’s going to be doing with it. It sounds fascinating. Yeah, absolutely.

Do you think, when discussing benchmarks in general, we need to incorporate the compute budget into the benchmark? Yes, absolutely. I believe this will be a very pressing need in the future, particularly next year for the publicly developed architecture. It is always possible, with test-time compute—like test-time search and test-time training—to achieve higher performance at the cost of additional compute. Typically, there is a logarithmic relationship between compute and performance. This means that if you want an apples-to-apples comparison between two systems, you need to consider the compute budget.

You can only compare systems that are using the same amount of compute, right? For instance, if you look at the O1 model from OpenAI, you cannot really attribute a fixed score to it on ArcGIS unless you're also limiting yourself to a certain amount of compute. It's always possible to logarithmically improve your performance by just adding more computers to the problem. Of course, this is true for O1, but even before that, it was also true for brute-force program search systems. Assuming that you have the right DSL, extremely crude and basic brute-force program enumeration can solve ARC at a human level.

It would just take hundreds of millions of dollars of compute to crack the entire dataset that way. It's an extremely inefficient and, to be honest, a rather foolish idea. But in theory, it's possible, right? Philosophically speaking, do you think there's always a commensurate relationship between the amount of compute that is required to complete a task? What I mean is that when we use language and cognitive tools, we might not realize we're utilizing compute, but the universe presumably expended a significant amount of compute for those things to emerge. So, in some sense, is it really possible to compress the amount of compute that we use?

I think so. I believe humans are tremendously complex and compute efficiently. You can see this in the fact that, let's say you're solving tasks; for instance, you can solve the entire private test set and only expend a few calories, basically. You might say, "Okay, but it's because, you know, we're just using extremely little energy per operation that our brain does." However, that's actually not true at all.

If you're comparing transistors and neurons, for instance, you find that neurons are tremendously more energy-efficient than transistors. It just so happens that we are managing to solve extraordinarily hard problems using a comparatively small amount of neural operations. We are also incredibly energy-efficient compared to current AI. Once we achieve a similar level of financial efficiency, we expect to see significant advancements. What's your opinion on using a programming language like Python or Turing completely versus using a domain-specific language (DSL) for these approaches?

Well, I think using a DSL, like for ARC, for instance, is fundamentally limiting. No matter what you do, no matter what base language you're using, you should be able to learn the functions that you're applying from the data that you have. In fact, you should be able to do this as a lifelong process. Every time you encounter a new task and you're solving it, you should be able to derive useful abstractions in the process—abstractions that may relate to problems you've encountered in the past.

And so you're going to want to turn that into reusable functions, reusable building blocks, and store them so that the next time you come across a similar problem, you can reapply the same building blocks and save compute, solving an equally difficult problem in fewer steps. No matter what you do, you want to learn the language that you're going to be using. Of course, that could mean learning the DSL; it could also mean using something like Python, but within it, increasingly writing higher-order functions, classes, and other reusable building blocks to enrich your language. I wondered if you're softening your position at all. So you tweeted, "It's highly plausible that fuzzy pattern matching, when iteratively applied and sufficiently iterated many times, can asymptotically turn into reasoning."

And it's even possible that humans basically do it in that way, but it doesn't matter. It's the optimal way to do reasoning. Is that a shift in your position? Compared to what position? I don't think it's a shift. I suppose it's saying that, first of all, it's possible that we think in this way.

I think we do, yes. Oh, interesting, because I would have thought that, due to the Spellkey view on things, you would have believed that we engage in high-level reasoning and do not perform fuzzy matching. I think the cognitive units in the human brain actually involve fuzzy pattern cognition. That's the core element of what you do. When you're engaged in something more akin to reasoning or planning—essentially System 2 processing, which is a slow, logical, step-by-step approach—what you're really doing is iteratively applying your intuition, but in a structured form.

And by the way, this is exactly what deep learning-guided program synthesis is about, which is the approach I've been advocating for since 2017. What are you doing when you're engaging in deep learning-guided program synthesis? You are building a program—essentially a graph of operators—step by step. At each step, when you're deciding what to edit in your graph, what to add, and what branching decision to make, you're applying your intuition and leveraging a guess provided by a deep learning model. Thus, you're iteratively guessing to create this highly structured, symbolic, discrete artifact known as a program.

And when you're running this program, that system too, I think this is basically the way we do systems as humans. We are iteratively guessing and applying fuzzy pattern cognition to construct an artifact that is, in fact, symbolic in nature. For instance, let's say you're playing chess. When you're calculating in your mind, you're unfolding some moves step by step, but you're only going to be doing it for a few of the moves that are on the board. How do you select which moves to consider? You're applying pattern recognition. Then, when you're simulating one move into the future, you're not going to be simulating the entire board.

You're going to be focusing on some areas. So again, that's pattern recognition. And sometimes, by the way, sort of pattern recognition is basically a guess at heart. So it might be wrong, right, in some way. While sometimes in chess you're calculating, some of your intuition about future states of the board may be incorrect. Then you play the move and realize, "Oh, oops."

Right? I think this is basically how humans implement this concept. This is not a shift in my position; I've been contemplating these ideas for quite a while. In fact, this is the foundation of my current favorite theory for interpreting consciousness.

It's the idea that, in order for something like system two to arise from iterated fuzzy pattern cognition, the iteration sequence needs to be highly self-consistent. Everything you add must be double-checked to ensure that it matches what came before it. If you're just pattern matching with no guardrails whatsoever, you're essentially hallucinating; you're dreaming. This is exactly what happens when you're in a dream state, by the way; you're just repeatedly intuiting what comes next with no regard whatsoever for consistency or self-consistency with the past. I believe this is why any sort of deliberate, logical processing in the brain needs to involve awareness and consciousness. Consciousness serves as a kind of self-consistency check.

It's the process that compels the next iteration of your intuition, of this pattern cognition process, to be consistent with everything that has come before it, right? The only way to achieve this consistency is through these back-and-forth loops that bring the past into the present and your prediction of the future into the present. You have this nexus point in the present, this thing you're focusing on, and that nexus is essentially your consciousness. Consciousness is the process that transforms an iterated pattern of cognition into something that resembles reasoning, something that is truly self-consistent.

Yeah, so I think the issue was that you haven't changed your position. It's just a case of people understanding what your position is. I think people dichotomize the system, you know, like symbolic people who think it's all in this discrete world. Then there's the alternative, which is the connectionist approach, where it's all fuzzy matching. I've never been in the purely symbolic camp.

If you go back to my earliest writing, I mentioned that we need program synthesis. I emphasized the necessity for deep learning-guided program synthesis. I proposed a merger of intuition, pattern recognition, and cognition with discrete step-by-step reasoning into one single data structure. I have repeatedly stated over the past eight years that human cognition is truly a mixture of intuition and reasoning, and that relying on only one of these will not get you very far. You need the continuous kind of abstraction provided by vector spaces and deep learning models, as well as the more discrete symbolic kind of abstraction offered by graphs and discrete search. Why do your detractors see you as a symbolist when you are clearly not?

Well, I'm not sure. I've been into deep learning, you know, for a very long time, since basically 2013. I started evangelizing deep learning very heavily around 2014. Back then, the field was pretty small. I've done quite a bit to popularize deep learning and make it accessible to as many people as possible.

I've always been a deep learning guy, right? And when I started thinking about the limitations of deep learning, I was not thinking in terms of replacing it with something completely different. I was thinking of augmenting deep learning with symbolic elements. So, you commented in this tweet that you were just discussing consciousness, and you suggested that all System 2 processing involves consciousness. Could you explain more about what you mean by that? You suggested that all System 2 processing involves consciousness. Yes. Can you elaborate on that? Any sort of explicit step-by-step reasoning needs to involve awareness.

And in reverse, if there's any cognitive process that you're running unconsciously, it will not have this strong self-consistency guarantee. It will be more like a dream, a hallucination. It's basically the idea that if you're just iteratively guessing, unless you have this strong self-consistency guarantee, you will end up drifting; you will end up diverging. Consciousness is the self-consistency guardrail, basically. And this is why you cannot have a system without consciousness. What is your definition of reasoning?

I don't really have a single definition of reasoning. I think it's a pretty loaded term, and it can mean many different things. However, there are at least two ways in which I see the term being used, and they are actually quite different. For instance, if you're simply memorizing a program and then applying that program, you could say that's a form of reasoning. Let's say, in school, you're learning the algorithm for multiplying numbers, for instance.

While you're learning that algorithm, when you have a test, you're actually applying the algorithm. Is that reasoning? I think yes, that's one form of reasoning. And it's the kind of reasoning that LLMs and deep-generating models, in particular, are very good at. You're memorizing a pattern; this time, you're fetching a pattern and reapplying it.

But another form of reasoning is when you're faced with something you've never seen before, and you have to recompose and recombine the cognitive building blocks you have access to—your knowledge and so on—into a brand new model, doing so on the fly. That is also reasoning, but it's a very different kind of reasoning that internalizes very different kinds of capabilities. The important question about fine-tuning models and large language models (LLMs), in particular, is not whether they can reason—since there’s always some sense in which they are engaging in reasoning. The more important question is whether they can adapt to novelty. Many different systems could just memorize programs provided by humans and then replicate them, but adaptation is key.

What's more interesting is, can they come up with their own programs and their own abstractions on the fly? To start creating abstractions, first of all, you need to be solving novel problems. Solving a novel problem means that you're starting from some base of knowledge, some building blocks. Then, when faced with a new task, you are recombining these building blocks into a model of the task, and you're applying this model, and it works. In the process, as you solve many problems, you will start noticing that some patterns of recombination of the building blocks you had occurred often, right?

And when you start noticing this, it means you can tag them and abstract them, refactoring them into a more reusable form. Then, this reusable form can be added back to the set of building blocks that you have access to, right? So the next time you encounter a similar problem, you'll be able to solve it in fewer steps, expending less energy because you have access to this higher-level abstraction that fits the problem. Is there a way to measure the strength of reasoning? Again, you would need to start by defining precisely what you mean by reasoning.

I think you can define, for instance, generalization power precisely, which is basically the amount of novelty that you can adapt to. It's quite interesting because I suppose you define it in terms of performance—how good my model is. Another way of describing it is to imagine reasoning purely as traversing the deductive closure, which means composing together knowledge we already have in new configurations. Then we make that leap in solution space because we found a new model that works really, really well. Is there an intrinsic way of measuring the type of model rather than its generalization power? No, I think you really have to observe what the model does. You cannot just inspect the model and determine how strong it is at reasoning. So there's no intrinsic form of reasoning that can be classified as "good"?

Given two models of a problem, for instance, which model is better? Can you just look at them and tell which one is better? I think it's very much goal-dependent, right? You cannot reevaluate a model like a simulation of a thing, for instance, if you don't have something that you want to do with it.

But if you do have a goal, then you just look at what the causal factors are that are required to achieve that goal. The best model will probably be the simplest model that retains these causal factors. Francois, what are you doing next? Well, I just left Google a few weeks ago, so I'm starting a new company, a new venture. I can't share much for the time being, but we're going to be tackling program synthesis, particularly the planning-guided program synthesis.

And we're currently building the team. Amazing! Are you looking for people to join? Yes. Tell me more.

What are you allowed to tell us? Well, what I can share is that I've been discussing some of these questions, like the best ways to approach HUS. Specifically, I've been exploring topics such as deep learning, guided program synthesis, and RKGI methods. I've also been contemplating what the next benchmark after RKGI might be. I've been thinking about these questions on the side while working full-time at Google, where my main role was developing Keras. Now, I feel that the time has come to focus full-time on these research questions.

So, make the research not a side project, but the main focus. Is the emphasis on creating new benchmarks or beating the benchmarks? I think it's both. I'm a strong believer in the idea that you need to co-evolve the solution alongside the problem. That was actually the motivation for creating ArcGri in the first place: to have the right challenge that forces you to focus on the right questions and the main bottlenecks to achieving strong generalization with AI. Therefore, I don't think RKGI is the last benchmark.

And of course, there's going to be version 2 of RKGI, but that will not be the last benchmark. I think we will always need new benchmarks, exploring new things that are challenging for AI but easy for humans. Is it cheating in any way if you work on your own benchmark? I don't think so. The benchmark is meant to be a tool for research.

Again, something like RKGI, for instance, is not really meant as a tool for research. Similarly, something like ArcGIS is not meant to serve as a binary indicator telling you whether we have HGI or not. It's really intended as a research tool. It challenges you to work on the right questions, directing your attention to the right problems and helping you make progress. So you could say that ArcGIS is basically a compass toward the GI. It's not like a test for a GI.

And, you know, there's a spectrum of possible solutions to Arc. For example, because you know what's in the private test set, you could just put the answers in directly. Of course, I'm not going to be answering half-price in any way. I mean, obviously not. I'm running half-price anyway, so why would I answer it?

But on that spectrum of generalization, there's the moonshot, which is aiming for extreme generalization, or there's one notch below that. What are you aiming for? I would like to build AGI. I would like to create something with human-level capability. And what would that mean?

What kind of things do you think you could achieve with that kind of AGI? Well, the most important thing, the obvious thing, is solving programming, right? If you solve AGI, then you can just describe what you want to a computer, and the computer will build it for you. If it's really AGI, then it will scale to the same level of complexity, the same level of code-based complexity that you can achieve with a human software engineer. And, you know, it’s probably not going to stop there either.

How long do you think it will be until we understand the role of humans in software engineering as we start to get there? Well, we'll see. I think we're going to start collaborating to create entirely new tools and interfaces to work with this technology when it's ready. However, we are still pretty far from that. We're talking about something that doesn't quite exist yet.

I don't think frontier models, not even O1, are quite at that level. Right now, programmers are very technical. Do you think that programming might be democratized in the future? I think so, yes. I believe that in the future, anyone should be able to develop their own automation processes based on their domain expert knowledge of the problems they’re facing.

Everybody should be able to program—not really in the sense of writing down code, but in describing to the computer what they want to automate and how they want to automate it, and the computer will always do it. A big aspect of software is tackling complexity at different scales. It's certainly what you've been doing for your entire career. Do you think we'll always have this problem where we'll be on the boundary of this incredible complexity? What do you mean by that?

Even if we democratize one or two steps up the hierarchy, wouldn't we still just build software that is really complicated? Yeah, quite possibly. But the idea is that we'll be able to offload that complexity to an external complexity-processing AI. So we will transition into a future where we no longer understand the code that is being run in any way. Absolutely.

You know, I think to a large extent, this is already true. If you look at any sizable codebase, there is no single software engineer who actually understands it completely. We're always limited by a fragmented understanding of what we are doing, which is fine as long as we have a good grasp of the high-level goals and constraints of the system. So where should the source of agency be? Because, you know, you're describing a "blind man and the elephant" type challenge, where loads of developers have their own perspective on a very small part of the system.

But when we have the AGI version, how could that change? Broadly speaking, I think programming from input-output pairs will be a widespread programming paradigm in the future, and that will be accessible to anyone because you don't need to write any code. You're just specifying what you want the program to do, and then the computer programs itself. If there's any ambiguity in what you meant—and there will always be ambiguity, especially if the instructions are provided by a non-technical user—it's important to address that.

Well, you don't have to worry about it because the computer will ask you to clarify. Okay, so I created basically the most plausible program given what you told me, but there's some ambiguity here and there. So, what about this input? Currently, I have this output. Does that look right?

You want to change it. And as you change it, you know, iteratively, you are creating this correct program in collaboration with the computer. So these future systems will have program synthesis as a core component, an explicit component. But how will humans interface with it? Are we still going to describe things using natural language, gestures, images, and similar methods?

It could be natural language, or it could simply involve drawing interface elements on your screen. You could always attempt to generate a high-level representation of the program that is being created, one that a non-technical user can visualize and understand. For instance, I could display a type of data flow graph to gather input from the user about it. Very cool!

One of the characteristics of large language models (LLMs) at hyperscale is that, in a sense, it’s not that difficult because they operate like a content delivery network (CDN); they simply copy all the weights and distribute them across various locations. You’re describing something very sophisticated that might be akin to a globally distributed database where the skill programs move around among different nodes, and so on. Is there a massive new type of architecture we need to build for this? Yes, I think we’re going to need a completely new type of architecture to implement lifelong distributed learning. In this setup, you have many instances of the same AI solving different problems for various people in parallel, searching for commonalities between the problems and between the solutions. Whenever they find sufficient commonality, they can abstract these into a new building block, which is integrated back into the system, making it more capable and intelligent. I think I’ve got it now. So, what you’re building is a globally distributed architecture based on the premise that we find a good solution. Well, I can’t really tell you exactly what we’re building, but it’s going to be cool. Yeah, that sounds pretty cool!

How do you think that organizations like OpenAI are going to start incorporating not only test-time inference, but also some of your ideas realistically into their systems? How might frontier models incorporate program synthesis, for instance? Well, I think something like O1 is already doing precisely that. When you look at what O1 is probably doing, it's writing its own natural language program, describing what it's supposed to be doing. It is executing this program itself, and the way it's writing this program involves a very sophisticated search process.

So this is already program synthesis in the natural language space. There are other ways you could leverage program synthesis. You could do program synthesis in latent space, potentially, similar to what Clermont-Buddy and friends are doing. You could also simply generate actual programs. Why do we use natural language?

Sometimes, you know, maybe you might want to use an actual programming language. I think we are definitely seeing a shift towards leveraging more and more test-time compute, and that's going to accelerate. It's a fundamental trend. However, on the transductive active fine-tuning, that's a little bit more architecturally difficult, isn't it?

Because my model is always being fine-tuned, I can imagine they might build something a bit like Docker, where there's a base layer and then my fine-tuning layer, along with another fine-tuning layer. It's very fragmented. The difficulty in applying test-time training to actual frontier models is not so much the infrastructure. It is definitely true that the current serving infrastructure is absolutely not set up for performing per-task fine-tuning.

But you could re-engineer it for this purpose. The main bottleneck is actually the task format. You can only do test time training if you have pretty clear inputs and targets. You basically need input-output pairs, and for most prompts, you don't have that.

You have that for ARC, obviously, but for most problems, you don't have that. Very cool. What's your theory on how O1 works? Well, you know, we can only speculate. I'm not sure how it really works, but what seems to be happening is that it is running a search process in the space of passable chains of thought.

Trying to evaluate which branches in the tree work better, potentially backtracking and editing if the current branch is not working out. It ends up with this very long and sophisticated, plausibly near-optimal chain of doubt, which essentially represents a natural language program describing what the model should be doing. In the process of creating this program, the model is adapting to novelty. Therefore, I believe something like O1 is a genuine breakthrough in terms of the generalization power that can be achieved with these systems. We are far beyond the classical deep learning paradigm.

One school of thought, which I think you agree with, is that there's some kind of active controller during inference time. It is actually performing multiple trajectories in an isolated way. Yes, it is conducting multiple trajectory searches. Some people believe that there's process supervision and other mechanisms at training time, but at inference time, it's all just one forward pass.

No, that is certainly not plausible because of the amount of compute being spent at test time. It is very clearly doing search at test time. Very interesting. I think it is trained at training time to reproduce the best available changes for the current problem, right? Kind of like, you know, AlphaZero-style training, basically.

But it's also conducting a search at this time in the chain of thought space. This is kind of obvious, and the telling sign is just the compute expanding, like the amount of tokens and the latency. Are there any other telltale signs that this kind of thing is happening? For example, it might have explored a particular area, and then after the consolidation, that is now gone. So when you talk to the model, it feels almost like it's forgotten part of its thinking.

Yeah, honestly, this is a little bit too specific. You know, I don't have any insider info about what O1 is doing, so I can only speculate. Okay. So people like Noam Brown are really bullish on this new scaling law for test time. And certainly, I love O1 Pro.

I think it's really, really good. It's qualitatively a big improvement. Um, what do you think there? Sure. So the test time scaling law is basically this observation that if you expand more compute, if you search further, you see a corresponding improvement in accuracy.

And that relationship is logarithmic, by the way. So accuracy improves logarithmically with compute. While this is not really new, anytime you conduct test-time search, you will observe this relationship. For instance, if you're performing brute-force program search, you'll find that the ability to solve a problem improves logarithmically with the amount of compute. With more compute, you can search further into the space of possible programs, and logarithmically, you discover more solutions. Therefore, whenever you undertake any kind of test-time search, you will see this relationship.

What is your current go-to model, and what do you use it for? For the most part, I’m using Gemini Advanced, and I've actually just started using the new Gemini Flash, the latest one. Me too. So I'm paying for Gemini Advanced. I'm also using Cloud 3.5.

I think it's very good for programming. So these are the two I'm using. What's your programming workflow with LLMs? I don't use LLMs all that much when I'm programming. But typically, if I'm currently facing a problem that I feel might be a good fit for LLMs, I will just open my browser and prompt the LLM, asking it for a function that does X, Y, and Z.

And while it usually doesn't work on the first try, after a little bit of debugging and nudging, I think it's a big time saver. What kind of failure modes do you encounter when you're programming? With LLMs, the failure modes can vary based on the model that you're using, right? I think, in general, GPT-3.5, close to 3.5, is the best one. However, sometimes you might find code that exists for absolutely no reason, such as unused variables or assumptions being made by the code that are not aligned with the data that comes in.

So it's pretty clear that there is no... The code is generated in terms of statistical likelihood. There's no effort to actually make it consistent, correct, or to execute it beforehand, and so on. So I think there's a lot of room for improvement there. You could imagine LLM-based software developer assistants in the future that actually do all these things as you're prompting them. They could write the code and then try to debug it themselves before actually showing it to you. What’s your opinion on LLMHC systems, which are currently older in age?

Right. So, you know, agents have been of an older age for quite a while now. People started talking about agents being the future almost two years ago, about a year and a half-ish. And so far, agents have not really taken off. The fundamental problem here is that LLMs are not quite reliable.

You know, if you look at one-fourth of an LLM, that's basically... You can think of the LLM as a guessing machine, right? The guesses it makes are much better than random, obviously. They're very useful guesses, and they are directionally accurate, but they have some probability of being wrong. When you look at an agentic workflow, you are chaining many of these guesses together.

And so the likelihood that you will end up not where you would like to be increases dramatically as you chain more guesses like this. This is a significant issue when agents are simply not reliable; they lack a sufficient level of autonomy. People say that with better models, this will be resolved. I believe it's an empirical question.

I'm waiting to see when agentic workflows actually start working. I don't think we're there today. I've softened my position a tiny bit on this. I agree that, because of the ambiguity problem, they are misdirected. And when you chain this, they're very misdirected. There is something to be said for just having more computation at hand.

I interviewed the team behind the AI scientist paper. Certainly, if you use Claude Sonnet 3.5 and ask it to generate an entire paper, it will produce banality beyond belief. It will just be a sketch. What they did was decompose the task into many agent workers, which, using a Google Maps analogy, zoomed in repeatedly, many times over. This approach produced dramatically better results.

Yeah, this makes sense. This basically tracks the idea we were talking about earlier of System Two being something like iterated System One with strong guardrails. The guardrails are very important. In this case, the superstructure is provided by the human programmer. The human programmer is breaking down the problem into the right sub-problems and orchestrating the whole thing in the right way.

And then each sub-problem can actually be solved by producing good enough gas. Amazing. When is Arc V2 coming out? Probably early next year. We are currently finalizing human testing.

So, as I mentioned, every puzzle is going to be solved by a group of humans. We know it's solvable, and we have some data that allows us to gauge how difficult it is for the average person. The goal is to create three sets that are difficulty calibrated. If you receive a score on the public evaluation, you won't be overfitted to that dataset. You can be very confident that you'll obtain a similar score on the other two sets. When you tested with humans, you discussed this in your work on intelligence, noting that one school of thought posits the existence of a g factor in intelligence.

And another school of thought is that it's very specialized. Did your experiments reveal that a group of humans generally performed quite well across all the tasks, or did you see significant specializations? No, it's absolutely the case that there are people who are just more intelligent, and they are better at solving our tasks. You do see that in the human testing data. What about on the long tail?

Do you see specialization in the types of tasks, or is it fairly flat? I think it's pretty flat. I mean, either you're good at it or you're not. Interesting. François Chollet, it's been an honor to have you on the show.

Thank you so much. Thanks for having me. It's been great. Thank you.