The AI emperor has no DAUs why most devs still don't use code AI: Quinn Slack

I was at Bleacher Report, and I think I learned that I’m better at coding than at sports. So, I believe I’m in the right role here. I want to talk to you about why most developers still do not use code AI. It might be a mind-boggling fact to everyone in this room, but it’s true, and we’ll have a good discussion about it.

I'm a coder. I code all the time. I've been building Sourcegraph, which is code search and code AI for developers, since 2013. I merged four pull requests earlier today, but I want to talk mostly about the product and what to build, as well as how to get it into the hands of users. Because if you don't do that right, then all the code that you write is for nothing.

I want to make three points. First, most developers don't use code AI. I want to emphasize this to make you reconsider. I aim to provoke thought in this talk.

And then I want to share some of the mistakes we made and some tips from our experience of doing everything wrong and then trying to figure out how to do it right at Sourcegraph as we've been building coding. All right, first, just by a show of hands, who in the last week has used any kind of code AI tool while coding? All right, man, I wish the whole world were like this room, but we are in the heart of it in San Francisco, and the whole world is very different.

Most developers do not use CodeAI. I'm going to provide some numbers here. I'm privy to a lot of private information and insights that I've encountered, and all of this aligns with what I have been hearing. So, first number: Oh, man, this is a great statistic!

Ninety-two percent of developers use code AI tools at work. Wow, that means these tools must have tens of millions of users. This was a study from GitHub involving 500 people. However, it turns out this is only in the U.S., and they had a very broad definition of what it means to use a code AI tool.

So, here's a case where the hype kind of outstrips the reality—a common pattern. This is an infographic from GitHub. Seventy percent of developers see a benefit to using AI coding tools at work. Surely, if there's a benefit, they must all be using these tools, right? I mean, if only it worked that way.

All right, here are some numbers for broader use in the universe. There are 1.3 million paid subscribers to GitHub Copilot, according to what Microsoft reported in January 2024. 1.3 million is a lot, but if you're just paying for something, are you actually using it?

And also, 1.3 million is not a lot compared to all the people in the world who write code. There is actually a way to get an even finer point on that number. GitHub released a study stating that if you look at the fine print, in this time range this year, 935,000 developers received a suggestion—not everyone accepted it. There will be some who saw that little ghost text in their editor but never actually accepted it. Additionally, this figure represents yearly active users.

I don't know of any companies that cite yearly active users. Surely, the monthly active user and daily active user figures are much lower than that. So, these numbers are significant; you could fill a city with these people. However, it's not that many when you compare it to the 26 million professional developers in the world, and I think this is an undercount. I will say that the question remains: how do you arrive at this number?

I Googled a bunch of different statistics, and they all were around this. So, you know, that's the vibe citation there. There are a lot more people who touch code in some way or another. Again, it's really hard to get a precise number on this, but I'd say probably 100 to 150 million people in the world touch code in some way.

And that includes students and other people, and so on. So, you know, 935K—best case—what is that in MAU? Compared to the number of people that interact with code, this is a scary thing. My best case estimate is that around 5% of professional developers use Code AI.

These are people who are paid a lot of money to write code, and they do it all day, every single day. All of us here in the room know that there's this amazing new alien technology that has been introduced on Earth, which changes how we code, and only 5% are using it. That's crazy. It means that in a room of 20 people, 19 are not using this. It absolutely boggles my mind, and that's a huge problem.

And it's an even bigger problem because, if you look at what they're actually using, the vast majority of code AI use is just autocomplete—it's the ghost text. I mean, yeah, that's good, but anyone who has used anything beyond that knows that AI can do so much more in coding. It can write entire files, answer questions, fix bugs, and all of that stuff. So, this usage is absolutely tiny.

And we need to realize that we here in this room, we are all the freaks. For us to be successful, it requires us to change a lot of minds out there in the world. So how do we change those minds? Well, you know, we've been building Kodi. Kodi is a code AI tool.

It does all of these things, and so we encounter many reasons why people are hesitant at first. I've taken the reasons from our own internal tools and ranked them. Why don't most developers use code AI? The first reason is simply, "Nah." They don't have a good reason. There are some people in the world, believe it or not, who have not used ChatGPT and have not heard about it.

Even developers. There are a lot of developers who are kind of grumpy, and they say, "Well, it's not perfect. I tried it, and it gave me a wrong answer." Yeah, no kidding. But there are many other people who have figured out how to get the right answer from it.

A lot of people say, "I don't need it. It didn't help me that much." You know, again, you have to figure out how to make it useful for you. Many companies haven't adopted it; it's too expensive.

One thing we’re seeing a lot less of now is security, privacy, and legal concerns. I think it’s a situation where enough people are using it that if it’s illegal to use code AI, then we’re all in trouble, creating a scenario of mutually assured destruction. However, there are many reasons why people still haven't adopted it widely, and this should genuinely concern us. Many companies are relying on numerous individuals across various enterprises to use code AI.

There are many people's retirement accounts, options trading accounts, and other financial assets that rely on technology. Consider how money is generated. For example, the money that gets deposited into your bank account from your paycheck – where do those dollars originate? It takes quite a long route. Just to explore the market a bit, we have these foundational model companies like OpenAI and Anthropic at the top.

And then you have the AI infrastructure companies. These companies, if they are providing services and performing inference, will send some of the revenue back to the foundation model company. In some cases, these are the same companies. What drives the usage of this technology? There is some experimental usage involved.

People are being looky-loos, going out and trying it out. That could lead to some experimental revenue. However, all of this can happen without any real, actual usage. But it gives the perception of, "Holy cow, this stuff is hot. This stuff is working."

And then you start to see some real usage. Some people using it, maybe paying twenty dollars a month on their credit cards. You get some developers using it at work. And then the holy grail: the way that software makes money is from enterprises, where there's recurring revenue from real usage. That first-year contract doesn't matter.

It has to be JPMorgan paying you a million dollars the first year, and then they renew for two million dollars the next year. All of the money in our paychecks ultimately comes from that. If you're working at an AI infrastructure company or if you've raised money to sell an AI product, it's ultimately because someone out there believes that five years from now, JPMorgan will be paying you two million dollars a year. So that's the only thing that matters; everything else is downstream of that.

And it turns out there's just not that much of that going on. Additionally, this is an even more lopsided pyramid than I've depicted here. We have NVIDIA and chip makers at the top, with so much investment riding on it. So, you know, this is where we ultimately need to get to: we need this person to be using our software.

And maybe it's not the coolest thing out there, but that is the reality of how technology makes money. I want to put some numbers to this. I estimate, based on my opinion, internal information, and some of the information that GitHub has shared at this conference, that the total recurring revenue from Code AI usage is around $300 million ARR. Now, look, that's a lot of money. If you had a company that was doing that, it could go public on its own.

So that's good. But how much of that money actually goes to these other companies? Not that much. From our own data at Sourcegraph, where we spend a lot of money on AI inference, we spend less than 10% of our revenue. So if you take that number, and by the way, we're not even doing it in an optimized fashion.

If you take that 300 million number, 10% of it is $30 million going back to these foundation model companies and AI infrastructure companies. It's a long way from where we are today in terms of revenue to the kind of revenue that would justify these massive multi-billion dollar valuations. Another way to look at it is Salesforce. Salesforce's annual revenue is $36 billion. All of this hype and usage of code AI amounts to a tiny, tiny fraction—1/120th of Salesforce's revenue.

So, we have a long way to go. Usage needs to grow a lot. Or, I could be wrong, and maybe the doomers are right, possibly not for the reasons they thought. Maybe we should never have done this, and this is all just going to be one massive hype bubble that will pop, leaving us all miserable.

But I think, because I use code AI every day, and you do too, this stuff is real. It's just that it's really early, and usage needs to grow a lot. So if you're building a product, keep that in mind. This is what I’m betting on. Let's talk about some of the lessons from our experience building Cody at Sourcegraph.

I wish I had this mindset going into it. Just a quick background to establish why you would even listen to me. Maybe I should have done this in the first slide, but hey, here we are. Sourcegraph has been around, as Brittany said, for 10 years. We started out with code search, and then we found out that if we had this tool that all the developers used, containing all the code in a company, it turned out to be amazing context to build a code AI on top of.

And so we took that and built Kodi. It's got really great autocomplete and inline edits, generating unit tests, and chat. The chat feature is where we really differentiate ourselves, because that's where you can make the best use of context. You can figure out why something is broken or how to change it.

Or where should I start on this? That's where we excel. We've got a lot of really big customers. We have four of the top five customers among the five biggest banks in the country.

We have most of the FANG or MAMAA companies, along with many other great customers, including some that are presenting today. We are the number two code AI company in revenue, second only to GitHub Copilot. However, because we're number two and because we're a startup and we're scrappy, we try harder. Alright, so what have we learned? One thing is that hype fools everyone.

Hype fools us. Hype fools all of you. It fools your customers. The second thing is that AI code completion, the feature that auto-completes the rest of the line or the next few lines, is like the freakish kind of capability that comes along once in a hundred years. We have kind of gotten spoiled.

It is so perfect. So I’ll get to that. But first, just, you know, regarding the hype, how do you avoid being fooled by it? I’ll share a few tips that have worked for us. The first, and this is the most important thing: if a product requires that you use it every single day, it is not going to make it.

There is zero hope. If you're building a product with a team and the people involved don't use it every day, your customers won't use it either. However, they might say it's really awesome, and that's the dangerous thing about this hype. Customers don’t always know what they want. For about a year, every customer said, "I want fine-tuned models."

They said that all the time. Our salespeople would say, "Hey, we need to build fine-tuned models." Everyone was talking about it. "Get a copilot" has it listed as a bullet point. It's italicized and says "coming soon" alongside some vague comments.

Really, what they want is for it to work well. They may use some specific terms, but don't pay too much attention to that; what they say is likely influenced by ideas they've picked up at conferences like this. Ultimately, they want the product to function effectively. If you can't clearly explain why it works and why it's great without using the term "AI," then you might struggle to succeed.

This is a really existential question. We create a product that enhances developers' productivity by 20% to 30%. If it's so incredibly effective, why are we selling a product? It's akin to those who have stock picks and sell newsletters. Why don't we acquire a software outsourcing firm to prove our product's effectiveness and monetize it by capturing that value directly? That's an excellent question, and I wish we had taken that step a year ago. We're currently considering doing that now.

And if you don't have the confidence to go and do that, to put a few million bucks behind it, if you've raised a bunch of money, then I'd say your product's probably not ready yet. Finally, I think we've all seen that this is actually not even one of the worst examples. This is a tweet that's generated a lot of activity, right? Just keep in mind that it does not translate into daily active users, so don't feel bad about yourself all the time. I wish that Elon would add something like this that would actually show you the daily active users of the products that are getting all this hype.

But until he does that, you can imagine it. All right, I talked about autocomplete being this incredibly useful feature. Here's how we think about features at Sourcegraph: it's a kind of four-box model. In the top right, we see an AI feature that doesn’t take a lot of time to evaluate. It’s easy to use and is utilized frequently. AI autocomplete is great because it captures every keystroke, allowing you to glance at it in just a few seconds—or even milliseconds. In about 100 milliseconds, you can determine if it’s correct.

So that's a really good property of a feature. It just so happened that the first feature of Code AI was positioned right at the top right. It's pretty amazing, but most other Code AIs do not have this level of product-market fit. We need to recognize that while we can work on improving it, Chat is not used nearly as often. Additionally, it's harder to vet those long response edits. I know many people enjoy making inline edits, like using Meta + K, Option + K, and tools like Cody and Cursor. That's great, but we need to be prepared for what actually works to be a very different form from what exists today. There are also other aspects to consider, like agentic features. I mean, it’s obviously the future, but it's just not there yet.

Who here has used a CodeAI agent to merge a pull request in the last week? All right, cool. Thank you for helping us push the world forward, but it's not there yet. Regarding other features, you really need to focus on the top right and drive any feature you create to that area; otherwise, it simply won't succeed.

So what are we doing at Sourcegraph to address this? Well, we are searching for the next great code AI modality. What's the next autocomplete? There are some ideas, like the next edit suggestion or something similar to Copilot++.

I'm skeptical of that. Chat. Steve Yegge, who is at Sourcegraph, just wrote this great blog post about chat-oriented programming. It turns out there are a lot of developers who have a chat session running in tools like Codi or even ChatGPT all the time while they're coding—it's just one ongoing conversation.

It's a totally new way of coding. What's interesting is that it's practiced by both Gen Z and disgruntled programmers in their 50s. I love that it has usage in both generations. There are new ways that people can use code AI that we haven't even considered yet, and it's still in the early stages. We always have to remind ourselves to build the manual and explicit components first.

We got spoiled by autocomplete. It's automatic, and it triggers with every keystroke. No, first you've got to make something work in manual and explicit mode. If you have chat...

You've got to ensure that people manually mention the context they want before automatically inserting it. If you have an agent, integrate it into the editor so that if there's an error, the developer can easily correct it right within the editor, rather than having to navigate to code spaces or some other completely different UI. Make the process manual and explicit initially, making it really easy for developers to fix issues. Once that's established, you can introduce the magic. Chances are, you'll already be busy enough with tasks to create a great product, and you may not have the opportunity to add the magic for some time.

When you're pitching, always explicitly de-hype and set expectations low. No matter how much you de-hype it, the person receiving the pitch will hype it up in their minds. Therefore, you need to remove the hype as aggressively as possible. Here's an example from Sam Altman.

This was a tweet, I don't know, like a year ago when ChatGPT had probably tens of millions of daily active users. So he's right. We know he's right, and if he's saying this, and his product has way more validation than your product, well, you probably should be saying it ten times more and ten times more intensely. Also, it's kind of cool marketing, and it makes you seem cool. Yeah, you've got to become... Here, it's covered up.

You've got to become a developer advocate yourself or kill the product. The last thing to note is that, as an ecosystem, all these foundation model companies, AI infrastructure companies, and AI applications—like Anthropic, OpenAI, Mixtrel, and Fireworks—will either thrive or fail together. We won’t get paid if we're not in business, making a significant amount of money from our coding efforts. So, we all need to work together and recognize that we stand to benefit immensely from our technologies being utilized. It's essential to tone down the hype and focus on building great products that developers use regularly. This starts with you if you're the one building the product.

Thank you! If you would like to reach out to me, here is my contact information. Happy coding!