When Software Costs Approach Zero: Opportunities and Implications

Right, good morning everybody, or good afternoon, or good evening, from wherever you're dialing in. Thanks for being with us today. This should prove to be quite an exciting and different type of talk compared to the usual Cedar Tech Talk series. The two speakers today, Ivan and Aaron, are data scientists out there in the wild, practicing their trade. They have come here today to share with us some insights that they've gleaned so far on the road to setting up their own business.

So I'll leave it over to Ivan and Aaron now, and I'm looking forward to hearing what you guys have to share with us today. Thank you. Yeah, thanks, Adrian. Hello everyone, we've got a full crowd today.

Sure! Here's the text with correct capitalization and spelling: "Yeah, so the thing we want to talk about today is what happens when the cost of software tends towards zero. What are the implications of that? Aaron and I used to work

And obviously, starting a Web3 company, we didn't have any background in Web3, so we relied heavily on LLMs to help us write a lot of the code. I don't know, Aaron, if you want to do a quick introduction of yourself? Yeah, sure. So, hey guys, my name is Aaron.

I have a background in computer science and software development, but I recently pivoted to data science. Yeah, cool. So today, we want to talk a little bit more broadly than these talks usually discuss. Most talks will focus on how you can use LLMs in a product, like a chatbot or doing natural language processing. We're going to speak more broadly and discuss using LLMs to write code for any kind of software. What are the opportunities and the impact of software costs tending to zero? To help us think through those implications, we have two frameworks. The first is cost and reward, and the second is risk and verifiability. So regarding cost, what we want to demonstrate to you later is that the cost, or rather the time, of developing software has dramatically reduced. Obviously, if the time decreases, then the cost of development decreases as well.

So I was speaking recently to another startup. They actually have a proper development team of ten people, and they've measured a percentage speed-up in coding. They've also replaced one of the two human reviewers who typically review any code that goes into production with a large language model (LLM). So, there are already real savings being realized in the wild. One of the implications we want to emphasize is that if the cost of software is going down, then each of these dots represents a software task or project. Some of these tasks, which were previously unprofitable, have now shifted to become profitable. This means we have a lower threshold for profitability. Yeah, sorry, no worries. So now we have a lower threshold for profitability.

So, previously, on this graph, we have the software projects ordered from large to small. On the right, we have Google and Meta, the very largest companies, which make tons of money and are very profitable; however, there are only a handful of them. As we move down this long tail of potential that you can build, we see enterprise SaaS.

That's roughly where the threshold for viability is at the moment. However, our contention is that as you reduce software costs and development costs, you can explore a long tail of potential software that can be created, extending down towards niche software and individual projects. We have two examples: one is slightly tongue-in-cheek, which is analyzing the faces on a Zoom call to measure attention. Something more serious is the idea of medical devices for niche conditions. This is obviously high risk, and it may be cost-prohibitive to develop software for a rare condition.

And so, LLMs can help bring that cost down and make those medical devices more viable. I'm going to do a quick demo just to show people that, in fact, it is much quicker. So I'll walk you through the prompt first. I'm going to use just one prompt. Everyone's coming up to the front now.

So I'll demonstrate. I'll walk you through how I usually write a prompt. We ask for something very specific: we want a visualization written in React that relates to microscopy images of blood cells.

We give it some context. I always like to provide context so that it can use its common sense and has some background knowledge to fill in the gaps of the prompt I’ve given. When I specify what I want in the code, I’m quite specific and detailed. I have a list of requirements, and I mention things like wanting controls or sliders for the number of cells, the clustering, and the blur.

I talk about how I want it to look, things like this. So I'm very, very specific. I put that prompt into Claude, and it gives me a response. It provides me with some text to explain what it has done.

And then, if we open up the code, you can see that in about 30 seconds, it writes this React code. If we take a look at what it generates, it provides us with this little piece of software. The context of this, and why it's useful to me or the clients I was talking with, doesn't really matter, but it just demonstrates that you can create something really quickly in no time at all. So that's the demo I wanted to share.

And hopefully, that gives a little bit of insight into using LLMs, especially if you’ve never prompted them or if you’re struggling. What I would say to you is that we’ve all had those silly ideas for projects, you know, thinking, "Wouldn't it be great if this thing existed?" or "I could build such and such a thing, but it's going to take too much time," or "I don’t have the software expertise." You may be a domain expert in something else, like archaeology or something else entirely, but if you want to build something, now there’s a chance to go from that idea into a proof of concept. You can actually bring it to life in the real world. So, I would urge you, if you’ve never written software before, give it a go! Most of the speed of development is within your reach now. I think this is where I hand over to Aaron, so I’m going to stop sharing.

Being recorded. Okay, stop share. And yeah, Aaron, it's over to you. I'm going to mute myself, actually. Can you see my screen?

Yeah, we can. Okay, perfect. So, no worries! Hey guys, I'm here to talk a bit more about exactly how we've used AI for our software development. Before I do that, I want to give you a bit more background on what we do. Cash is essentially about bringing price bonds or premium bonds to a broader market using crypto. At the end of the day, all that matters is that it's a web app—a web app that you can use on your phone. We have a conventional system built using a typical stack: React for the front end, SQL databases for data storage, and a couple of Python servers.

Now, obviously, being a startup, our main criterion is speed. It’s all about fast iterations, making mistakes quickly, learning from the past, making quick pivots, and ultimately reducing our time to market. All of this has been made possible thanks to the cloud and the speed it provides us. We were able to achieve this by using the cloud as a pair programming partner. Let me give you a bit more insight into how we utilize the cloud.

So it's not as if we built the entire app using the cloud. We used the cloud with different levels of involvement during various parts of the app. To break that down, we can look at the app from three perspectives. The first level is the user interface (UI), where there are plenty of components that were essentially built completely in the cloud. If you think about these UI components, they are relatively low-risk components that you can easily verify.

It's easy for someone to let you get the output and ensure that it works the way it's supposed to. There's no real reason for us to delve into the details of all the React code and understand exactly what's happening here. As I mentioned, these are low-risk elements. Some of the screenshots you see on the right are simply screenshots of the app. Now, let's be honest here: we know we are not going to win any design awards for any of these app layouts and designs. But at the end of the day, what matters is the speed at which we were able to create them, especially considering the fact that neither of us had any experience building UI applications, particularly with React, which has a very steep learning curve.

So, at the first level, it's mainly the UI. Like I mentioned, that's where most of the involvement was because we had some entire components that were pretty much built by the cloud. I'll give you a swift demo of that in the next section. The second level in which we used the cloud was for the app logic. When I say the app logic, I mean the way all of the different components in the UI interact with each other, as well as what happens when you click a button and things like that.

For example, when you click the sign-in button, there's obviously a lot of activity that needs to occur. Authentication, some writes into the database, and similar tasks need to take place. Additionally, there are reads from the database, and so on. Now, this is obviously not code that you can just let the AI run wild with.

You need to understand what’s happening in this code. But, as I mentioned, because it is React and there is a steep learning curve, we still used the cloud. However, there was always a human in the loop to ensure that everything was functioning as it should be. We needed to make sure that everything integrated well and that handling various cases was also managed with the cloud. Now, coming to the third section, this is where it becomes much more high-risk. This section involves the servers and the core logic of our app. Obviously, this is code for which we need to have complete ownership and understanding, as these are highly sensitive Python scripts that interact with the database. Especially since it's a financial app, we are dealing with financial data.

So it's super high risk. For this section, we developed the scripts and servers ourselves, but then we extensively used the cloud for certain tasks, such as code reviews, debugging, and testing as well. Testing is something that we've become good at. Now, obviously, documentation, code review, and refactoring are all fairly obvious low-risk use cases. One thing that I found the cloud to be extremely useful for was brainstorming the architecture.

So when I say architecture, what I mean is, okay, where should certain functions reside? Should they sit within the apps or the servers? I also consider what kind of architecture I should use for my servers. I came up with different architectures, but then the cloud was extremely beneficial. We bounced ideas off each other, leading to different iterations of the architecture. I would point out various issues and receive feedback in return.

So, for brainstorming, it's actually an extremely good tool. I will provide an example for a UI component. Okay, you should be able to see my blog chat window here. Yes, it looks good.

Sure! Here is the revised version with correct capitalization and spelling: So, this is going to be a pretty different prompt compared to what Ivan did. I even had a pretty good, well-engineered prompt that was super descriptive. I approached this from the perspective of, let's say, a complete novice who has no experience working with React. So, all I've done is essentially give Cloud a screenshot of a component that I'm trying to build. This could be a Figma wireframe that someone else has given you, or it could be something that you just sketched in PowerPoint.

It doesn't really matter, but what's interesting and important here is that I asked Claude to create a React component. I requested it to mimic everything in the screenshot, matching the UI as closely as possible, as well as the functionalities. I also specified a price for Ethereum versus Euro, as Ethereum is not really mentioned anywhere in the screenshot.

And obviously, in order for me to test this locally, I asked Claude to give me all the steps to set up a basic React project. Right from the start, Claude provided me with a working chart where everything functions properly. However, it's clear that the data is not real; it's mock data, but we'll address that shortly. What's more interesting is that Claude provided me with all the basic steps needed to set up the project, so I have everything configured now.

It's just a couple of five or six steps, and then it gives me the entire code that I need to create this new server. I set that up here in my IDE, and after running it, I see that, okay, it's using mock data; it's not actually using accurate historic price data. So then I follow up with one more basic query, saying, "It looks fine so far, but can you use any API or something to actually utilize historic price data instead of mock data?"

And then that's it. That's pretty much all. However, the cloud doesn't have the capability to run APIs or call APIs within its IDE. But basically, if you run this code locally, we will show you the output that I'm able to get from the cloud. Yes, that should be there. The time selectors work, and the prices are actually accurate.

Everything is accurate; the change is accurate, and everything is fine. This is actually interesting because the prompt that I provided was pretty bare-bones. I never even asked it to implement the time selector. Those are all things that it assumed from the prompt. This is a great example of how Cloud can be used to get up and running quickly.

And personally, in our case, we found something extremely useful. As I mentioned, React has a really high learning curve, and cloud technology helped us learn the language quickly by building confidence at a fast iterative pace. Now, I want to cover a high-level overview of the impact of AI. There are some big numbers on the screen here. Based on surveys conducted by GitHub over the past couple of years, they report that almost 97% of U.S. developers have used AI at least once within their work. We are not talking about startups or anything; these are employees from companies with more than a thousand employees. So, these are fairly large-scale enterprises and not small to medium-sized enterprises (SMEs) or startups.

Now, okay, having that level of penetration is one thing, but how do we know that it is actually making a positive impact? This is actually a really interesting study that GitHub conducted in partnership with a university in 2023. They took 95 developers and essentially performed an A/B test, splitting them into two groups and assigning them the same task of creating an HTTP server in JavaScript. One group was allowed to use AI tools, mainly their Copilot AI assistant, while the other group could use Google Stack, workflow, etc., but no AI tools. The completion rates were satisfactory, with 78% for the group that used AI and 70% for the other group. However, what is more interesting is the actual time to completion, conditioned on the completion rate.

So this kind of speaks for itself: that's a 55% reduction in time to complete this task, which is huge, and there's essentially no denying that. There are also more studies linked in the appendix and recent surveys that show developers use it extensively for test case generation and to understand existing codebases, especially to learn new languages, similar to the demo that I showed. Like I mentioned, we are coming from a startup point of view, so it's fairly easy for us to work with AI tools because we have a small codebase, and we don't really face many of the issues that larger enterprises might encounter. However, I want to cover some common challenges faced by bigger companies, but we are not going into the details of anything here. One of the main roadblocks is obviously integrating large legacy code.

And this is a roadblock for a huge number of companies. There are a couple of options available. One that is particularly interesting and worth looking at is an SDK by Microsoft called the Semantic Kernel. It's an open-source tool and is very lightweight.

It essentially acts as a middleware that sits between the prompts entered by your developer and the existing large codebase. This is super interesting because, as I mentioned, it's open source. It's something that most people can easily look at. The second thing that's a significant issue in terms of implementing AI into your software lifecycle is that if you have huge codebases that are monolithic or were never designed to be used in the modern era, many companies in the U.S. still have large monolithic architectures that are just not AI-friendly. Changing these architectures into something more modern is something that a lot of companies are doing right now. Watson X by IBM is a tool that's being used to convert many of the global systems utilized by banking into modern Java systems. These tools, by themselves, use AI, but the outcome they bring will also help developers use AI within these companies.

The other main issue that poses a roadblock for most enterprises, especially within the EU, is obviously privacy. Keeping the data in-house is a complicated endeavor. Federated learning, which many of you are likely familiar with, involves training the model in-house and then sending the model updates or weights over the network. This concept is currently being researched in the context of large language models (LLMs). There have been a couple of research papers linked in the appendix where researchers have utilized federated learning.

And then you don't really need huge machines; moderate to medium machines can train smaller versions of the LLM like Nokia and then update it centrally. There's also a different architecture by NVIDIA that uses smaller precursor models to train additional injections that go into the prompt. This way, you can even get away with much weaker machine systems so far. But, like I mentioned, we're not going to go into the details of any of this. Just to say that there are obviously plenty of challenges that enterprises face, but there are solutions for them out there. Now, you don't have to take our word for it to understand the impact of this. Similar to the GitHub study, it's fairly easy for anyone within the software industry to test this. You can easily design an experiment, taking something that's super low risk and easily verifiable. Maybe don't go for the code generation examples like the UI demo; instead, go for something like documentation, code refactoring, or test case generation. Ideally, it has to be something that's low risk and well-defined. You can test this out within your company or enterprise using a team of senior, experienced developers, but just make sure that you have the outcomes clearly specified, as well as all the metrics for success.

And then you can see whether using AI actually gives an improvement. But it's a test worth doing because, yeah, right now it's no longer a matter of if, it's a matter of when. So it's definitely a test worth doing to see the impact that it can bring. I believe that's the end of that. I will hand it back over to Ivan.

Let me go. Yeah, cheers. Let me swap back to my screen. Share, share this, share. Okay, I know what, show up screen—nope.

"Uh, why can't I see it then? Speaker? No. Oh no! Can I not, uh, share the screen? Share, but why can't I minimize this?"

But then it's just, can I stop? Yeah, sorry. Ah, yeah, okay, great stuff. I'm getting rid of that thing. Cool.

Okay, now we can see. Alright, Grant, we're back to it. Sorry, Zoom was giving me issues. Alright, yeah. Now, let's get back to some more high-level topics.

I know Aaron is obviously the more technical of the two of us, so he has provided the more in-depth advice. We want to consider the implications for the job market, or at least what our jobs might end up being. There are two really interesting data sets—one from the EU and another from the US. These data sets take all the jobs in an economy and split those jobs into their component tasks. We have two examples here: one is a model, and one of the tasks they have to perform includes posing for a camera, maintaining a portfolio, and maintaining their appearance. In our case, we analyze data and adapt those methods to different projects, among other things.

But then, obviously, we said here that we can be replaced by AI, but fashion models can't. I should also mention that each of those tasks for all of the jobs in the economy is ranked by frequency and importance. For example, posing for a camera is something that models probably do quite frequently, and it is also quite important to their job. In the literature, researchers have analyzed how exposed each of those tasks is to a large language model (LLM)—essentially, how easy it is to automate that task with an LLM. Some research by a couple of people from OpenAI has shown that jobs like mathematicians and software developers, similar to what data scientists do, are among the most exposed roles in the economy now that LLMs are prevalent. However, there is a silver lining! Thankfully, people like tax accountants and solicitors are also highly exposed to automation. Hopefully, we'll see a democratization of those white-collar services becoming available to the rest of us. It's somewhat analogous to when a brokerage account used to cost a lot of money to make a trade.

Now it's much, much cheaper, or basically free. So, should we, as data scientists, be worried? I know I'm talking to a somewhat niche audience. Therefore, I want to provide an example of how our job could change. I aim to recreate this analysis from a couple of people here at Cedar.

So, they looked at the Airbnb prices and evaluated the factors that affect property prices in Dublin. Now, it's demo time, and I'll quickly show you what the paper looks like. They have a map of the Airbnbs in Dublin and a force path explaining an example where the room type and the reviews per month are identified as significant factors affecting price. There are also some scatter plots that, while interesting, I won't delve into since the specifics of the analysis aren't crucial. What I want to demonstrate is that I can redo much of the coding work. Claude can essentially handle the coding tasks for me as a data scientist. I've provided a prompt, similar to before, but this time I've included the PDF of the paper.

I've provided a truncated version of the data set that they use, so it has that context. I'll briefly walk you through the prompts so you can see what I did. I specified that I want to write in a Jupyter Notebook this time, so I indicated the programming language I'm using. I ensured that it will handle errors correctly and told it that I want to use a random forest instead of the XGBoost that was mentioned in the paper. There are also specific visualizations that I want at the end. I requested some box plots and a map plot, so it knows exactly what I’m looking for by the end.

And if I can go forward, what we get then is that it gives me some text so that I can read and understand what it's doing, and it writes this code in about a minute. It processes the data, prepares the new features, trains the model, and then creates the plots. So we just copy and paste this into it; I’ve literally just copy-pasted that exact same code into here, run it, and it takes whatever time it takes. Then it plots out the bee swarm plot so we can see, okay, this is the count of beds, bedrooms, and distance to Trinity—those are the most important features. We’ve got similar plots to what we had in the paper, and we also have a really nice interactive map plot where we can click into the prices of each of the properties if we want. Great, so that’s the last thing—demo. If I go back to the slideshow, that’s just a demo of how quickly we can code something up using the cloud. But I think data scientists aren’t doomed; we’re not going to lose our jobs. I actually think the job is going to change because if we’re not spending five days coding and it only takes five minutes, probably what we’re going to do, since I’m piggybacking off that paper, is focus on all the hard work that went into that paper—finding the data set, understanding what it means, and determining what interesting questions to ask.

Those are all the things that cloud is not great at, and those are the things that we can do. We can understand what’s interesting to tell people. So our job is likely going to shift to something more strategic and high-level. We might also be able to take on more projects. Instead of having no jobs, hopefully we'll have even more work to do.

That's the optimistic take on it. There is a pattern here that I think, if we look back over time, can be observed. On the far left of the screen, we have a navigator. To plot his journey, he had to use logarithm tables—looking them up to figure out where he was going. In the middle, we have an accountant who used physical spreadsheets. If they wanted to make a change, they had to erase it and redo all the tallies. Then came Excel to take over from that.

You make a change, and it just feeds through really quickly. And then finally, you've got someone working on an IDE here, and we think that's going to shift over to working with an LLM instead. In each of these cases, we're shifting away from doing donkey work, like handling the boring judge work, and we're able to build a machine to take that over first. Then we can focus on the interesting problems, such as the financial projections that are important for the company. Yeah, and then the next graph I want to demonstrate—okay, so probably a lot of people in this audience are already aware of the progress that has been made with large language models. I took that first prompt that I gave to Claude for those self-fluorescents, and GPT-4, which was released last year in March, just gave me some Python code instead and said it couldn't be done. Then, with version 4.0 in the middle, it almost got there, but the clustering doesn't work.

I'm sure that with a bit of iteration, you could get it to work. But Cloud here, 01, you know, they get it right the first time, no problem. So we've already stepped up the capability quite a lot. Actually, I want to add to this. We've been using the LLMs to write code for us for the last 12 months, and we've seen this expansion in capability. It used to be that you'd be working on a couple of lines of code.

You'd paste that into GPT. You'd say, "Okay, this is the story. This is what I'm trying to do. What's going wrong in my code?" It was a very specific problem case.

Then, as it has progressed, it has become capable of writing entire functions for you. Now, we are at a stage where you can have multiple files with functions that interact with each other. You can pass all of that information into it, and when it makes the required change in one function that impacts another, it understands that and makes the appropriate changes. Its capabilities in understanding have already expanded significantly. If it continues to progress to the point where it can understand complex codebases, will we end up with software developers being entirely replaced, or synthetic CTOs, for instance? It's hard to say, but who knows?

Sure! Here's the text reformatted with correct capitalization and spelling: --- Yeah, okay, so the last section of this talk is about how we've been, as I said, using other lenses to generate or do work for us, either code generation or text generation. What we found is, or at least for this talk, we were able to distill our thinking into this framework of high risk, low risk, and whether the output is easy to verify or not. So, basically, risk is just about whether there’s a chance of something bad happening, and verifiability is how easy it is to determine whether the output is correct or wrong. I'll give some examples to help illustrate this framework. One example we discussed earlier, which Aaron presented, was transitioning from a PowerPoint mockup. I created this mockup using shapes and text boxes, and I pasted it into the cloud. I told her that I wanted it to be built in React, and this is the actual UI on the right. It’s very easy to tell that this is the UI I want, and it’s also pretty low risk because no one is going to die. --- Let me know if you need any further assistance!

So it's like an ideal case for the use of large language models (LLMs). Other examples, as Aaron mentioned, include code refactoring, which is fairly straightforward. You can perform parallel validation by comparing your old code with the new code that the LLM has generated to ensure they produce the same output.

Yes, great. Because you're not going to push it into production without conducting those tests, that’s the ideal scenario. When moving towards something that’s high risk, you can still validate aspects of the software. For example, with medical software—where the risk is high and someone's life may be at stake—it might not have been feasible to build software for niche diseases in the past. However, you can now create comprehensive tests, implement error handling, and maintain documentation; all of this is much easier to achieve now. It is also quite verifiable.

So we think there's actually a large opportunity here for this kind of thing. The next thing is that we use large language models quite a lot for text generation, such as for our social media content and blogs. For us, it's okay because we have a small team, and basically, it's one person handling the text. However, in a large organization, there are no objective criteria for deciding, "Okay, is this good text?"

Is this following exactly the style and tone of what we want for our blog posts? It’s pretty tricky to measure or evaluate the output in this case. You can use engagement metrics and A/B tests, that kind of thing. But again, that feedback is going to be quite noisy and delayed. It’s not as simple as, you know, does my code run exactly the way I expect it to. Another example, which I think is worth mentioning, is that if you're summarizing documents, you might say, "Okay, I want my summaries to be both concise and complete." That makes sense, but how do you quantify both concision and completeness?

Like someone has to turn that into a number. However you're quantifying that, that's a design choice. What assumptions are you making? What simplifications are you implementing? This design choice will affect what you output later on.

And so, whoever is making that decision for you is essentially deciding what your outputs will be later down the road. Therefore, it is important to be very careful around the evaluation of text. The final example here is one that affected us significantly. In anything related to healthcare, the stakes are obviously high, and if it's text, it can be difficult to evaluate; however, some of the code can also be challenging to assess. We developed smart contracts to handle funds – at least, that was the intention. We had built these smart contracts to accept assets, distribute them, and perform all the required logic. While we were confident that the logic was correct and functioning, we couldn't ascertain if there were vulnerabilities, such as whether the system could be hacked.

We couldn't be sure of that. In that case, it's actually difficult to verify. I mean, you can use it as a helper tool, but you can never be 100% sure. So, really, we think that in these kinds of cases, where it's high risk and hard to evaluate, you'll always need some kind of human in the loop. It's going to be very difficult to scale any solutions because you will always need a human working on it. So, yes, I think that's the final example. If anyone in the audience wants to connect with us, please reach out. There's the email address; you can find us on LinkedIn. Now, I'll open it up to questions if anyone has any. Alright, thanks, Aaron. I’m going to leave that on the screen for just a bit longer. I will do the convention here to stop recording before we do the Q&A. So, I'm going to stop recording.