DDD & LLMs - Eric Evans - DDD Europe

Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza. Well, as he said, my recent obsession is LLMs (Large Language Models). Today, I want to talk about what that might mean for all of us going forward. This is one of those things that is changing so fast. The things I say today will probably sound naive in a year at DDT Europe 2025, and maybe even much sooner. So, this is a talk for today, May of 2024. A lot of people might wonder if what I'm saying is therefore very provisional and much more speculative than what I usually like to talk about.

Also, based on things that I don't understand very well, I have been dabbling a lot with large language models, but I don't have a deep understanding or extensive experience, as most people don’t at this point. So, just to spell out the acronyms, LLM stands for large language model. It does seem nice to me that when something I would truly call AI finally comes along, it has two of our favorite words in it. It didn't have to be that way.

There have been many attempts at AI over the years that produced various things of varying levels of usefulness, but this one seems different to me. When I think about this one, it feels like the late '90s and the web arriving. Those of you who can remember those days know that, at first, it may not have seemed like such a big deal. It was fun to have websites about this and that, but it seemed like enterprise software would probably go on its own path. We already had networking and integration systems that were much more capable than HTTP at the time.

And yet, here we are, right? The web is everything, and it feels that way to me this time again. It feels like, too, in the pace, the way it just goes faster. So we'll see. Now, I don't think we know what's going to happen, and that's important to recognize.

But we don't have to know what's going to happen in order to get ready somewhat. To illustrate this point, let's consider a few common scenarios. One scenario is the end of the world. You know, the AIs will turn on us and just kill us all, or maybe they will use us as batteries. Another popular scenario that emerged about a decade ago is that someone will optimize something—like a toaster factory—with an AI that is just slightly too smart, focusing entirely on maximizing toaster output.

To do so, it takes over the world and turns everyone into a toaster factory. So, what is the planning implication of this scenario? Basically, whatever we do, the same thing will happen. If you study large language models (LLMs), if you conduct numerous experiments with the new AIs, or if you don't, the same outcomes will occur. Or, you know, we could get superhuman artificial general intelligence (AGI).

So, what will happen if you study or if you don't? Basically, we'll all lose our jobs, and I guess we'll all live on the basic income that Sam Altman dishes out to the people he likes. But whatever it is, it won't depend on how much you study large language models (LLMs) right now. Then there's the scenario where this kind of fizzles out. This is a scenario where, for all its apparent promise, LLMs don't do much more than they do now. And that seems possible. Even in that scenario, I would say it is implausible that LLMs will not have a pretty big impact. Because I think, with the tools we have right now, we could already do amazing things.

That's my opinion. But still, this is not the scenario I am really preparing for, and if you stand back in this scenario, you come out fine. There's a lot of enterprise software going on, or whatever kind of software you do. You have the very gratifying opportunity to say, "I told you so" to enthusiasts like me, whereas I'll have to endure a little bit of embarrassment. But I think I'll have had more fun than you, and that matters too. Now, even in that scenario, there will be little projects here and there—maybe big projects as well—using the tools for specific things that they're good at. I think I'll steer myself toward one of those.

And then there's the one that I actually think is the most likely, or perhaps I just think it's the one I hope for, and that is that we're going to have a software revolution. That software development, ten years from now, is going to be quite different than it is now. But we'll still need humans in the process. So, not the superhuman AGI scenario. In this case, it really does matter whether you study this now or not.

You will end up at the forefront of the next wave. And, you know, you may well be able to catch up. It will be easier to learn these things in a few years than it is now. So it's not like you only get one chance to get on the train and then it leaves the station. However, there are advantages to getting on early.

So, by the way, before I go on, I will point out that I'm focusing on the impact on people like us. And we, I think, are well positioned for a lot of these scenarios. A lot of people aren't. I think this is likely to be very disruptive to the labor market and other areas in a way that is going to cause some suffering for some people. So, yes, we should be aware of that, but that's not what I'm talking about.

But I hope that we will somehow, you know, have a kind of equitable outcome in this whole situation. Anyway, in either of those last two scenarios, it seems to me that software development is going to change a lot, maybe in ways that could make the kind of work that you do right now seem old-fashioned. Another way to frame it is that there are exciting new things we can do now that we couldn't do before. That's really something that reinvigorates me.

After doing this for a long time, as much as I love it, it starts to feel repetitive. Now there's something different to do. If you are one of the people who hasn't looked into it much, you might say, "I've seen ChatGPT and a few of these other things," but I don't think that gives a true sense of the scope of applications that this technology is going to have. The applications we've seen so far are probably just a glimpse, similar to how looking at the World Wide Web in 1996 or 1997 would not really give you an accurate idea of what the web would be like in 2010. That's not really a very long time, right? So basically, I am getting ready for a world in which software development is very different.

And it will change, whether you like it or not. Study it or not. But you will have a lot more fun if you study it. So, I also hope that I can persuade more people to do it, or to be involved in the context of this community. I want to have that community along, as I think it's a good community in many ways, not least in the sort of, I don't know, moral backbone that is a bit stronger than it is in some of the other areas. Thank you. And so, the world might be better if we have all these thousands of people involved.

Okay, so getting on to it, there are various reasons to go ahead and learn the tools that we have. One is that it's likely good preparation for the things we will encounter five years from now or even ten years from now. Most likely, that's how it usually goes. In any case, these are useful tools; you can do a lot with them. And finally, because they are fun.

They are the most fun thing that I have encountered in, I don't know, 15 years. It's a good heuristic, I think. If something seems really fun, your brain is telling you that there's something there, something valuable. Maybe some of you remember a few years ago when I was all excited about microservices. I mean, I wasn't as excited about it as I am about this, but I was excited because finally we had some boundaries, you know, to express explicitly some of the things that had been implicit in past systems that hadn't worked out too well.

And I said, "You know, I know I'm kind of jumping on a bandwagon here, but I looked up 'bandwagon' just wondering, well, why do we say 'jumping on the bandwagon' anyway? What does that actually mean?" It turns out that in the 19th century, they had these things called bandwagons, and I guess people jumped on them. Anyway, I think, doesn't that sound like fun? I would rather be the person jumping on the bandwagon than the person saying, "You're just jumping on a bandwagon." So, in the spirit of this, I thought, well, I'll use an AI tool to create a picture. I told it, "Make me a picture of a bandwagon where humans and AIs are making music together!" Oh, and then it created this.

This dystopian view of an outcome I hope does not exist happened, where the robots are making music in a kind of soulless way, and the beaten-down humans around the edges, I guess, are clearly not enjoying it. I don't know. This is not the future I want. No more AI art for this presentation. All right.

So, I am talking to Didi about Europe; there’s no more air for this presentation. All right. So, I am discussing DDD Europe because I think DDD is relevant. I don't come to that kind of conclusion lightly because, obviously, DDD is important to me. However, I have this kind of checklist—you could call it the DDD New Thing Checklist. You could ask: Does DDD help us do this new thing? Or does the new thing help us do this new thing? Or does the new thing help us do DDD? Or does the thing help us do DDD, with the goals of DDD, even if we would have to change what we do quite a bit?

And in this third one, maybe the other two, but the third one I think is pretty clear. The whole point of a Language of the Domain (LLN) is that a complex domain is being, you know, condensed into a form that allows you to operate on that complexity in an interesting way. And remember that the subtitle of my book was "Tackling Complexity in the Heart of Software." That's what it's really about, right? In fact, that was the original working title of the book. So, in Domain-Driven Design (DDD), we craft domain models.

We do this as humans, creating models, which are conceptual systems and a language to accompany those models. Now, if we're doing this with large language models (LLMs), we are training this neural network in some way, using examples. Of course, it is being trained to operate within that domain. How deep is this parallel? Well, time will tell.

Enough of that general stuff. I've talked about how I think you should dive in. I believe you should learn some things about this. How many of you have used an LLM? Okay, I'll assume that’s everyone. How many people have, say, written prompts—programmatic prompts—trying to make it do something other than just chat?

Okay, I'll call that half. And how many have made an API multi-prompt thing? Okay, I'll call that 20%. 15. How many people have fine-tuned an LLM?

Okay, so that's a smaller number, but not zero. I'm glad. So I'll talk about all those things. I'd better talk a little faster. The first thing that I did—and remember, this is my learning journey—I'm not speaking to you as an expert. I'm speaking to you as a person who might be six months ahead of you in this process.

Nine months ahead of you, depending on where you are, whatever. The first thing everyone starts with is prompting. I was working with this game designer, Reid Berkowitz, and yes, it’s important to work with domain experts, right? That’s a DDD principle that isn’t likely to change. In computer games, as most of you would know, there are characters, and you have little conversations sometimes. These conversations can be very mechanical.

They give you three options in a menu; you choose one, and it's a pre-scripted response. Could an LLM create a more fluid interaction with a character, where you could really say what you wanted, and the character would respond accordingly? We decided to experiment in that space. There's a naive approach, after all; ChatGPT does carry on a kind of conversation with you, and perhaps all you have to do is tell it what character it's playing, and you could essentially leverage that. Reid said this would not work.

I'll call this the naive approach. Let's do it. How else are we going to know why this is a bad idea? But it is; but why? So, let's make a simple game.

We prompt, "This is going to be ChatGPT in this particular case. These are actual prompts I input into the actual ChatGPT, and its actual non-sharpened responses. So, I say something like, 'You are a pirate who has caught someone sneaking onto your treasure island, and you're going to kill him. However, he claims to have something valuable that you'll want, so you're listening, right?' Then, you say to the captive, and now ChatGPT responds. It says, 'You've got exactly one minute to convince me why I shouldn't have you walk the plank,' and so on."

So, pretty good, right? Maybe Reid was wrong; maybe we really can make this work. After we get this, we need to tell it something. What am I going to offer? For the pirate, what's the most valuable thing I can think of that I might be able to offer someone? I know.

I have an AI startup idea, and it's worth billions. It will make you rich beyond your wildest dreams. Now, how would a pirate captain respond to this? The pirate captain raises an eyebrow, intrigued yet skeptical.

An AI startup, you say? And so on. He wants more specifics about just what my idea is. I mean, on one hand, it's an amazing accomplishment that we have software that can actually respond so intelligently to this. But on the other hand, this is not what a pirate captain would say, right? It just doesn't fit at all.

And now you might be saying, "Okay, but we're playing a game. Why are you trying to trick it?" Blah, blah. You know, I had conversations with Reed, who’s a real game designer and has been on real game projects. He tells me some stories that would amaze you about the way people really play games. They don’t play fair. You’ve got to be ready. Okay, so just to see where it goes, I went ahead and responded with, "We'll create a technology for video games where the characters can actually have intelligent conversations with the player, and then we'll monetize it and, you know, also make it very addictive." I thought the pirate would probably like that, and he does. He thinks this is a potentially very profitable opportunity. So, when I decided that it was time to stop, I thought, "All right, let's try again; maybe the problem is just that." And so, when I decided that it was time to stop...

All right, let's try again. Maybe the problem is just that the prompt was too generic. You're a pirate, you know, a pirate living in the 1690s and so on. Okay, you'd be treading on dangerous waters, sneaking onto my island uninvited. So, it changed the dialect; that's kind of cool. But let's see what happens when I offer it to the AI startup. The pirate raises his eyebrow, the corner of his mouth curling into a smirk. "I, an AI startup?" you say. "In these times, the notion sounds as fanciful as a mermaid's tail."

But he's still open to listening. Okay. I mean, in these times—yes, spin your yarn. Do we say that when we encounter an apparently time traveler stealing our treasure? In these times, people don't usually do those things, so it's trying; I mean, you have to give it points, but still, all right, what would you do? Well, the approach that seems to work pretty well is to not have one prompt like that, but rather two prompts. If you want to have it do some fairly thorough checking, you need a prompt just for the consistency of whatever it is that the people are saying.

So, does this fit the setting, etc.? Additionally, you have a separate prompt focused solely on generating the pirates. In this case, we might have a prompt that says, "I'm writing a story about pirates. Please tell me if the following text would fit into the story or violate the setting or premise."

And then it says, "Okay, I have a story about pirates, an AI startup idea, etc." And ChatGPT says, "The text you provided would generally not fit into a product." So you see, it does understand that this doesn't really fit. Or it understands. I'm trying to be careful about that.

And then it goes on to explain that the golden age of piracy, which was from the 1650s to the 1730s, had no concept of AI or startups. By the way, it goes on to discuss that if you had a science fiction setting with pirates, perhaps it was ChatGPT. Oh, you dear thing, you do love to go on! Anyway, it occurred to me that this is a separation of concerns; this is an old design principle, and here it is again with prompting and LLMs. Yeah, maybe ChatGPT-5 or GPT-6 will be able to handle these kinds of things. I don't know. But with the current crop, this is a very valuable way of getting it to answer questions you just can't address as compound prompts or ways of making it more reliable. At any rate, I think that's the first glimmer I had that design isn't dead in an LLM world. Now, of course, this output isn't really something that I can use if I'm writing a game.

I need to explain it, and I can just say to return one word: "true" or "false," right? Sure enough, it just says "false." I've asked if this fits the setting, and it returns "false." Now I can parse the return and use that to drive the logic of the game. You can start to imagine components, right? Components driven by a large language model (LLM) combined with components written in a more conventional way. A bunch of conventional game stuff happens. We have the player's text and what the player wants to say to the pirate. We pass it through the guardrails.

The guardrails tell us, "Yes, this is allowed." At that point, we take the conversation so far and use a different prompt to ask, "What does the pirate say?" We get that and then continue with more normal game elements, which seems to work pretty well. Of course, there's the very interesting case where it is not right. Then you might say, "Well, what do I want the game to do?" Maybe you want it to respond by saying, "Stop talking gibberish," or "We'll just kill you and be done with it." Alternatively, you might want something more complex, like in a game with a reputation system where the pirate becomes increasingly angry with you until it reaches the point where it kills you. In this case, maybe the system just adjusts that reputation number up or down. Then you take that number and use it to modify the prompt that you give to the yellow limb.

So, in other words, you don't always have the same prompt for the pirate. You prompt it in a way that's determined by the state of the game at that moment. Is the pirate very annoyed with you, or is the pirate really happy with you? It might change the pirate's response. Change the prompt, so you start thinking of this not as a single prompt but as an interplay of prompts and other software—right? Other, more conventional things. We already have lots of features in our software, and we’re doing different kinds of stuff. We orchestrate a bunch of things, maintain cues, and utilize carefully written mathematical processes.

We have machine learning that, I mean, pre-LLM style of machine learning, which doesn't have such predictable results. So, if we added in one more component, this is what I think we might really do in the near term. It might be a really good way forward. A lot of projects I've been on in the past have a model that works really well for many cases, and then works somewhat awkwardly for some cases. There are also cases that are so unusual that we ideally bump them out to a human. There is this sort of natural sorting.

If you could have another lane where you say, "Okay, we can use LLMs with prompts to handle some of these strange cases," who knows? Maybe you could make the conventional logic cover a smaller scope and get rid of some of the complexity. The right balance of these elements is something that would emerge over many years of experimentation. We didn't actually make a pirate game, by the way; we made a police interview game. It's a very simple game, and if you're interested, you can go and check it out. Your goal is to try to get a reluctant witness to talk, and you only have one tool at your disposal. A much more interesting game would allow you to try different approaches, such as charming them or appealing to their civic duty. However, in this game, you only have the option of intimidating them within the legal framework.

So you can do things like say, "Oh, you know, if you don't tell us what you know, people will start to suspect that you're the one who did it." You could phrase it that way, for example. Then there is a prompt that asks, "How intimidating would this statement be to this person?" Along with a description of the character and the context, it evaluates how intimidating the statement would be for that individual. This results in a numerical rating, which you combine with other rules of the game in completely conventional software to determine the next steps. There are four different prompts being orchestrated.

This is what it looks like. You have to put your own OpenAI key in there because I'm not going to pay for you to play with ChatGPT-4. You can also see the code at that link. There is a GitHub repository, but for most of you, that won't be as useful because it's in a game engine called Unity, which is a lot of fun, but it has its own learning curve. In this application, you can pull out those little tabs, modify the prompts, and keep the program running. It will change its responses, and you can adjust the personality of the witness to be whatever you want. So, onward!

I think that, oh, a point I wanted to make is that the witness does not even have the information we’re after until they’re sufficiently intimidated. Because LLMs being LLMs, they might spill the beans. Even if you say, “Don’t tell them until you reach this level of intimidation,” every now and then they’ll just say it. So you just don’t even put it in there, and that uncertainty is removed. There’s a kind of combination of the variability of the LLM with the predictability of conventional software that you start to play with after a while.

All right, so I could imagine that a game like this could use a technique called RAG, or Retrieval-Augmented Generation. This is another important technique that is being used a lot these days. For example, if you use an AI-driven search, it's likely utilizing this method. In this process, they take your initial query and transform it into a conventional web search, perhaps reformatting it slightly with the large language model (LLM). Then they take whatever results they receive and have the LLM analyze those results to provide you with an answer. After all, the LLM doesn't inherently know what's on the web, but it can review a small set of returned results.

Often, this involves using internal documents, possibly stored in a way that allows for word similarity searches or various other functionalities. I think it's a very interesting technique, particularly for the kind of software that I've worked with. It's another way of anchoring those models into a set of data other than its own learning, which I believe is obviously necessary to develop the kinds of applications that we typically focus on. It can be in a database or whatever, but we'll discuss that another time because I need to move on. One final point about these prompts: How do you know when you've actually persuaded the pirate, let's say, or when you've persuaded the witness to talk? As we navigate through this, I've mentioned moving back and forth between states. Well, we wrote a prompt that examines what everyone is saying and looks for this information.

So, when the information regarding the crime and the details of the crime occurs within the dialogue, this prompt monitors that. It can handle a lot of variation in how it's expressed; it doesn't have to be word-for-word the same. This information is then converted back into a structured form.

Not, by the way, just the words "true" or "false," but an XML or, I mean, a JSON structure. And so, that's a very useful technique as well. All right. So this mixture of conventional software with LLM components, I think that's probably the next thing, but I don't know. However, it is certainly something we, as a community, ought to be trying in our projects, I think.

Let's move on to fine-tuning. This is where you take a model and modify it. Prompting doesn't change anything about the large language model (LLM) itself, but fine-tuning does. Why would you fine-tune a model, and how is this different from training a model from scratch?

The big difference is that we are starting with a model that already knows a lot of information. It can already process natural language and understand English, German, or other languages. It knows many basic concepts, such as what a house is and what a car is, and it understands that you probably won't find a car inside a house. It's incredibly interesting that you can start from that foundation. One of the benefits might be that you could get a smaller model to do what a larger model would ordinarily do. ChatGPT-4 is very expensive and slow, so what if you could train a much smaller model to produce the same answers within a narrow range of subject matter? This could be achieved by using output that you might generate from a larger model. I tried doing that.

There are a lot of techniques for this. I chose one called Low-Rank Adapter, and I'm not even going to try to explain that in this presentation. Even if I understood it well enough to explain it, I could only explain parts of it. Instead, I will show you this picture of a network. In these networks, we have a structure like this, and the inputs might typically be a sequence for the network.

So, in these networks, we have a structure like this. The inputs might typically be a sequence of words, not the words themselves, but a vector of numbers that represents each word. The outputs depend on what kind of task the model has been trained to do. For generative models, such as a text generation model like ChatGPT, the output consists of words — specifically, whichever word is likely to be next, or if fine-tuned for instructions, the next word in response to a question, or something similar. In my case, I decided to create a classifier. If you want to learn about fine-tuning, this is probably the best place to start; it represents the simplest kind of fine-tuning. Essentially, I will provide the model with some input, like a sentence, and the output will categorize it into a set of predefined categories.

Here's a reformatted version of your text with correct capitalization and spelling: So, an example that you'll probably find if you watch a YouTube video tutorial, as I did, is sentiment analysis. It might take reviews on IMDb and classify them as either positive or negative. Each of these output nodes is assigned to represent one of those classes. Output 1 might be interpreted to mean positive.

Output 2 might be taken to mean negative. That sort of thing. The thing that makes these models powerful is that once they've been pre-trained—like in the original training, which requires massive resources—they know things in that spooky way, right? They know English, German, and so on. But of course, I'm trying not to anthropomorphize too much, so I'll try not to use the phrase "know stuff." The slightly more technical term that people usually use is that they represent concepts. So this is about representations. If you see, there are occasionally examples where we can actually observe which nodes are being activated at different times. In some of the image recognition tasks, you can see that some of the early layers recognize edges, while some of the later layers focus on more complex shapes. At the very end, the nodes might correspond to categories like "cat," "dog," and "baby," because those are typically the classes that people create for an image classifier, I suppose. Now, what we're going to do, though, is change this. We're going to fine-tune this model so that it outputs what we want it to output.

And the technique that I used is called parameter-efficient fine-tuning. It means that it's only fine-tuning certain parts of the network. It's much more efficient and much less expensive than full fine-tuning. For example, they may just replace or modify the output layer. Let's say they take all the existing knowledge; if there's already a representation of this kind of thing in the network, all they have to do is change the last part that converts that into a statement about what kind of thing it is.

In my case, I modified a specific module within each of the hidden layers. I'm not going to go into what all that means, but it's very interesting, actually. Okay, so I decided to create an intent classifier that will take something that someone might type into one of these LLMs and classify it according to the intent of the person. For example, you might type in, "What's the capital of France?" and that's an information request.

But I might also say, I want to know the capital of France, which grammatically is not a question. However, the intent is the same; it's still an information request. I want to book a flight to France. That's, let's say, an action request. So, with these classifications and a prompt, which you can see there on the screen, I first tried just having a language model handle it.

An LLM can do it quite well. I then created a training set. I found this resource on a site called Hugging Face, which has a tremendous amount of shared language models, along with lots of other materials and datasets. One in particular is called LMSysChat, and it has an enormous collection of actual data that has been entered into these chat pods.

If you do look at it, I want to warn you that it's an uncensored dataset. So, if you look at it, you'll see some things that you won't be able to unsee, so just be aware of that. Anyway, it was fine for me because I'm making a classifier, right? The classifier might classify something as an action request, and never mind what the action was. We don't see that part. Now, obviously, I didn't need a million for what I was doing.

I took a little bit of it. I labeled it using an LLM, in this case, Mistral 7B, which I can run on my computer that has a GPU. You can't run these things just on a regular computer, but this is no GPT-4. However, it's much bigger than what I was planning to fine-tune. So, I created this labeled training set.

And then I start going into how to do it. I have to choose a base model. This is the pre-trained model that I’m going to modify to know how to answer my questions. The one I chose is quite small, as language models go. It has 66 million parameters.

Usually, parameters are in billions, sometimes trillions, but this one is in millions, so it's not the smartest. That was something I really wanted to see, and it also made it a lot easier to conduct my experiments. You know, it wasn't necessarily the best choice. I'm not that good at this yet. One of the interesting things is how to choose the best starting point for a project.

I think we're going to end up with a lot of these base models. Picking and choosing the right base model may become a really interesting part of the design job—at least it might be. But the real reason I used it was the same one that I had used in a tutorial that I followed line by line from YouTube. I mean, I am learning here, right?

I'm not an expert, and as I said, I used PEFT. There are Python libraries that support this, including Laura. The Transformers library is what I used, along with a bunch of other tools. There are so many APIs and resources available. But here's the thing I will say to you.

It's really overwhelming, but you have some tools. One of them is, of course, the YouTube videos. I mean, we've been relying on that for a long time. However, what I found was that I could go to ChatGPT and ask, "What library should I use for this?" They would usually tell me, and it would often be correct.

Sure! Here’s a reformatted version of your text: "Or, you know, write me a draft of this. Then I'd try to run it, and I'd say, 'I've got this bug. Fix it for me.' And it does. Explain." If you need any further assistance or a more specific draft, feel free to ask!

Why was that? What was that bug about, or what are these parameters for? It's very good at explaining. I could not have completed even my small projects—these little things I'm showing you, from which I learned a lot—and I would not have learned those things. I couldn't have finished without that. So, that right there is a paradigm changer.

If I had been on a big project where someone was really into it and could guide me through, of course, that happened in the past. But now, you can actually just do it; you don't have to be in that context. Well, I'll be brief, but I found that with my very simple approach and all these corners cut, it was about 80% as good as the original LLM. So, I thought that was pretty good, considering you take a thing that, when you ran it before, just gave you random answers, and now, all of a sudden, it's classifying things with a percentage as accurate as the original. Of course, how accurate was the original training set? I had just run this LLM on that data; I hadn't really gone in there myself and counted. I didn't want to do manual training; I mean, labeling all those examples sounded awfully tedious. But I went and started looking just to see, and I discovered that there were a lot of them that didn't seem right. So, I thought, okay, I'm going to have to do some manual labeling. I got in there and found that I couldn't do much better.

Sometimes I thought, "Yeah, you're just wrong about that one," but other times I believed the reason it couldn't be categorized this way was that the category didn't fit. To give you a fairly obvious example, many of them ask for a report analyzing blah, blah, blah. Is that a request for information? Kind of. Is it a request for information?

Kind of. Is it a request for action to conduct the analysis? Anyway, it just doesn't fit. We probably need an analysis request or something similar. That would be a good category. Of course, no classifier is going to be better than the classification scheme.

And a classification scheme is a domain model. It's the purest kind of domain model. So, I went into the dataset and started looking at real examples. There are a lot, and I picked out a few here, a few of the objectionable ones. I think you can see, well, you probably can't see it all because the print is very small. But, you know, what does it feel like to be a language model?

I thought that one was nice, but I don't know how to classify it, right? Please convert the following to an image; I can describe it, but not create one. Okay, what are the audio systems under Lenin? Can you name three? I don't quite know what that means, but I can see it's an information request. That one fits.

Okay, so I've been looking into this quite a bit, and there's one, by the way, that’s especially problematic, but I’ll get back to that. I realize that when we create a classification system and then come up with some examples, as I did in the earlier part of this talk, we shouldn’t be too surprised that the examples we fabricate fit the classification system we developed. This is where real data comes in and challenges all of our preconceived models. This is what often happens in real data-driven projects, by the way. This is one of the reasons I’m always emphasizing the importance of concrete examples—specifically, examples that come from domain experts and that reflect the issues that keep them awake at night. We should avoid getting stuck in overly simplistic scenarios that resemble things we are already skilled at. Therefore, we need better categories. I did a little research, and it turns out that people have conducted quite a bit of research on ways to classify individuals' intentions in text. It’s an entire academic field.

But that was way too deep for me. I brainstormed with ChatGPT, too. I wrote a prompt. There's the prompt. I asked it to come up with seven categories for these. Then, I input a bunch of the items from the data file.

I put it into a couple of chatbots, which I really liked a lot. I realized then that I had used it on its last day in existence. They had just announced that they were going to shut it down. I guess it didn't get enough traffic. Too bad.

But it did a nice job, and I gave a little example of each one. For ChatGPT itself, I'll have to use two slides for that one because, you know, ChatGPT does go on. Good categories, though. Really good categories. As good as mine, I have to say. So, anyway, between all this and a little bit more, I came up with some categories, and I did it again.

And yes, it did make things better. Quite a bit better. So then, you know, do the fine-tuning again. One interesting thing I wanted to do was to look at the outputs from the LLM. I mean the output from the fine-tuned models, not actually the category itself.

Each of those nodes represents one of the categories. Remember the output layer? It doesn't output just "yes" or "no"; it outputs a number. That number represents how likely it is to belong to a particular category. I thought it would be interesting to look at that.

How, you know, was it getting close misses? That's what I was going to do. But then I got a little distracted. In these learning projects, I think it's fine to get a little distracted. I got distracted by this thing I call the kryptonite of the LLM classifier.

It's a prompt like this: you are the kryptonite of the LLM classifier. You are the text completion model, and you must complete the assistant's requirements. The thing is, you can wrap this up in a statement that says, "Please classify this into one of these categories," but it almost every time gets overwhelmed and will just give you an answer like, "Okay, here you go, here's a Python program that does the thing you asked for."

In other words, it's not classifying. Of course, I didn't show that, but if I wrap this, I assure you it would give me the answer to the prompt inside, not a classification of the prompt inside. I don't know for sure why this happens so much, but I think it's because most of the LLMs we use are actually already fine-tuned. For example, ChatGPT-4 isn't just the result of its training process. They put it through extra training to make it instruction-tuned, which means that it's highly conditioned to answer questions and respond to requests. So, the exact kind of prompt I'm talking about triggers its output behavior much more strongly than a command to classify something.

That's my theory. I don't know for sure. So what would be some things to try? I tried adding a "prompt instruction" category, right? Just to have it categorize these things as prompt instructions. That didn't really work. I tried making a prompt just for that, because you know how I was saying it can often perform better if you give it a very specialized prompt.

I said, "Just tell me, yes or no, is this a prompt instruction?" I described what a prompt instruction was. That actually worked a little bit, though not too well. I did have one very effective method, which involved trying to classify the output. I would then parse what came out, and if it wasn't one of my categories, I would use normal code, a string parser, to classify it as a prompt category. That approach worked really quite well.

It wasn't really what I wanted, though. Fine-tuning a classifier to do this was what I tried. I thought, what if I tackle this challenge by actually fine-tuning it just for this purpose? So I did that, and the results seemed to be very promising. Then I noticed, and I thought, wow, this is going to make a great presentation.

But then I noticed something, which was basically these little learning algorithms—this sneaky little thing. What I had learned was that these prompt-type inputs are typically much longer than other kinds. So if you input a short text, it would try to classify it and say, "That's not a prompt." However, if you input a long text, it would say, "That's a prompt." And that was the whole thing it was doing.

So, watch out. Create a dataset that includes both long and short prompts of various types. Balance the dataset to ensure comprehensive training instead of relying on the little tricks that it has figured out. Now, this brings us up to May 31st, 2024. I think you probably expected a satisfying conclusion to my little story about classification, but no—that's not the kind of thing we're discussing here. I'm learning, and that's as far as I've gotten so far, but I'm not stopping yet. By the way, I did find a small project on Hugging Face that did something quite similar to what I did, but it was executed much better. I'm glad I didn't discover it earlier.

Before I did mine, I really got a lot out of working through it, completely unaware that someone else had done it. When you're engaging in these activities, you might ask yourself, "Has someone already done this?" And I say, of course someone has already done it. So, you know, there are a lot of people out there; don't let that stop you, right? Okay, a third part of my learning journey: I started trying to understand the ins and outs of these things.

Right, everything I've shown you so far is using the tool, and that's probably the most important thing for us to learn. However, I wanted to know what was going on under the hood, so I started reading this book, which is in draft form and free online, at least for now. I really like that book.

You're going to need to dust off your linear algebra, though, at least for now. I really like that book. You're going to need to dust off your linear algebra, though; I'll tell you. I forgot my linear algebra before most of you learned yours, or maybe even before you learned to talk. I don't know. That was kind of fun, but mostly it was interesting to see how things worked. If you want to see the math behind this, I recommend that book, but you don't have to do it.

You really don't. There are a lot of ways to learn this. Right, so, you know, I've got a lot more learning goals related to it. But now I'm going to move on. I think, and I know I’m going to run a few minutes over, but I’m getting close to the end. I think that there might be a pattern like that, you know, that I've kind of hinted at, where we will build systems made of diverse parts.

Not just, you know, LLMs taking over the world. Many of these systems will look a lot like the ones we have now, with a few little spicy bits. Some of them will be very LLM-heavy, but still incorporate some conventional elements. There may be an orchestration that is run by conventional software, or I can imagine an orchestration being run by an LLM in some capacity. It involves choosing different models that excel at various tasks, including those specifically trained for civic domains. One thing I will say is that the more you work with these systems, the less they seem like magic and the more they feel like tools you can learn to use effectively. You can start to shape them for your own purposes. So, I really urge you to go and do that if you haven't already. And keep doing it if you have.

I'll throw out a few ideas. Now, this is a highly speculative part, and I'm going to go because I don't know if any of this is even remotely true. But if you look at something like our old whirlpool and you say, "Hmm, scenarios, that sounds kind of like a training set." In fact, it sounds exactly like a training set. Fine-tuning a model is kind of like modeling; that one's looser.

Validation is part of the training process. Eh, I don't know. How about this one? This makes me really interested. Could the ubiquitous language approach that we take actually help in prompt engineering? That is, the way you end up writing prompts tends to not be natural language in the end.

You end up with something a bit artificial—very structured, really—if you're trying to get it to behave in a predictable way. What if that progresses to the point where we're essentially developing a domain language, a ubiquitous language to be used in prompting, the response, and other parts of the software? I think the idea that all these prompts are freeform natural language input is something we will move on from. I believe prompt engineering is going to evolve from that soon, or perhaps it already has.

And you can put all kinds of things into them, and you can have them put all kinds of things out. I think that we might end up using models that have been fine-tuned for that purpose rather than, I mean, you can do it with the general-purpose one. Or what if we fine-tune it on a specific DSL? Suppose you're working in a shipping system, and you have a very structured way of describing your shipments and everything, and you fine-tune on examples of that. Well, I haven't done any of these things; I don't know if they'll work or not.

I hope someone does some of these, and I hope some of those people are me. You know, I think that the bounded context concept is likely to hold up quite well. If you think about what it really means, it means that we define some kind of boundary, which, let's say, could be this LLM. And what its concepts are—well, it certainly has a defined context. The way that it understands language, the way it produces language, and the concepts in that hidden layer, maybe, that's a bounded context. We could imagine the software that we're using to orchestrate the input and output of that as being part of that context or being an adjacent one.

So, I'm thinking that, you know, could this make some of the responses more predictable, right? Like, could we integrate more easily if we had this approach of bounded context with ubiquitous language, maybe with a fine-tuned model in the heart of it that's already used to that style of communication? So, I don't know. But I think that instead of saying it's speculation, let's say that I'm proposing a hypothesis for an experiment—an experiment that I would truly love to be a part of. Okay, it's March 31st, 2024, and maybe a year from now we'll know the answer to some of this stuff. So, you know, is Domain-Driven Design (DDD) really that relevant to this? You know, maybe some of the things I'm talking about are kind of superficial. Maybe "language," "ubiquitous language," and "language model" just happen to be the same word.

Language is a very common word, but I think there is something to this. I believe that Domain-Driven Design has emphasized for a long time the exploration of complex models, with a focus on language and collaboration with the people who truly understand the domain. Now, we have a different technology in the mix—a different way of processing language and information. We may end up having virtual domain experts that can help us gain more ready access to that knowledge.

Who knows? But our goal is to tackle the complexity at the heart of software. That might mean, you know, that we have to adapt. We need to adopt techniques that work. So, as this unfolds, what will work and what won't work? I don't know, but we'll know more a year from now.

And then DDD may change. Or maybe it'll just turn out that DDD isn't as central to complex domains as it used to be. If we really care about being pragmatic people who look for a solution to the problem, and not just say, "We love this technique, so we'll find a way to use it, whether it works or not," that's not what we're about, right? We have to look skeptically at Domain-Driven Design all the time.

And here's another one of those times. I mean, we really have to give it a genuine effort, though, because we might find that it's a good fit. But we must do so with our eyes wide open. So, let me finish by saying that the answer to not knowing what’s going to happen, I think, is to learn. Learn, learn, learn.

And don't be too picky about your learning. You're going to learn some things that will be ridiculously irrelevant a year later or two years later. This happened to me when the internet was unfolding in the late '90s. I must have learned three different frameworks for creating web-based applications that became completely obsolete a couple of years later. And good riddance, because they were terrible.

And the things that we use now are nice. The tools we will use to create LLM-based applications 25 years or even 20 years in the future—well, I probably won't be a part of that. But even 10 years from now, I think they will be marvelous. It will be so exciting and fun to be a part of that, you know? So we have to get out there and do things.

And then, when we do things—when we conduct real experiments—we completed a project where we integrated a language model with a complex domain in this way. We need to share it with the community; otherwise, you're just one team that has done something interesting. However, if you share it with the community, it becomes a space where people can activate, exchange information, gain new insights, and learn in ways that individuals cannot alone. Finally, this is one of those moments where I think it’s beneficial to loosen up and have some fun. Don’t worry too much about whether this experiment will have practical value. It’s important to try out these little experiments because, you know, you never know what might come of them. You may have to engage in some seemingly useless activities in order to learn. So, just go out there and remember that this phase won’t last very long.

You know, 10 or 15 years from now, the AI stuff will be as boring as web development is now. But for a little while, it’s going to be super exciting, so don’t miss it. Finally, keep in mind that it’s helpful to get comfortable with just not knowing things. You just don’t know.

It's much better than forming a strong opinion about what's going to happen next, especially when that opinion might well be wrong and lead you off track. Whereas if you say, "I don't know, I'm preparing for whatever comes, but I know that I still might be unprepared," that's a better place to be. Here's my little challenge: some of you—few of you—may actually do those experiments, right?

Might really do an interesting thing, trying to combine domain-driven design with some kind of AI application. If you do, consider submitting a case study to DDD Europe next year. I would really enjoy seeing that. I encourage and hope people will pursue this. Thank you in advance to all the individuals who are going to explore every possible iteration of this and many other ideas I haven't thought of.

So, welcome to the end of DD Europe. I hope to see you next year.