DDD & LLMs - Eric Evans - DDD Europe

Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Raza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza, Reza. Well, as he said, my recent obsession is LLMs (Large Language Models). Today, I want to talk about what that might mean for all of us going forward. This is one of those things that is changing so fast. The things I say today may sound naive in a year, at DDT Europe 2025, or maybe even much sooner. So, this is a talk for today, in May of 2024. A lot of people might wonder if what I'm saying is therefore very provisional and much more speculative than what I usually like to discuss.

Also, based on things that I don't understand very well, I have been doing a lot of dabbling with large language models, but I don't have a deep understanding or extensive experience, as most people don't at this point. So, just to spell out the acronyms, LLM stands for large language model. It seems nice to me that when something I would truly call AI finally comes along, it incorporates two of our favorite words. It didn't have to be that way.

There have been many attempts at AI over the years that produced various results of differing levels of usefulness, but this one seems different to me. When I think about this one, it feels like the late '90s and the arrival of the web. For those who can remember those days, it was initially not seen as such a big deal. It was enjoyable to have websites about this and that, but it seemed like enterprise software would probably follow its own path. We already had networking and integration systems that were much more capable than HTTP of that time.

And yet, here we are, right? The web is everything. And it feels that way to me this time again. It feels like, too, in the pace, the way it just goes faster. So we'll see. Now, I don't think we know what's going to happen, and that's important to recognize.

But we don't have to know what's going to happen in order to prepare somewhat. To illustrate this point, let's consider a few common scenarios. One scenario is the end of the world. You know, the AIs will turn on us and kill us all, or maybe they will use us as batteries. Another popular scenario that emerged about a decade ago is the idea that someone will optimize something, like a toaster factory, with a slightly too-smart AI that is entirely focused on maximizing toaster output.

To do so, it takes over the world and turns everyone into a toaster factory. So, what is the planning implication of this scenario? Basically, whatever we do, the same thing will happen. If you study LLMs, if you try lots of experiments with the new AIs, or if you don't, the same things will occur. Or, you know, we could get superhuman AGI.

So, what will happen if you study, or if you don't? Basically, we'll all lose our jobs, and I guess we'll all live on the basic income that Sam Altman dishes out to the people he likes. But whatever it is, it won't depend on how much you study LLMs right now. Then there's the scenario where this kind of fizzles out. This is a scenario where, for all its apparent promise, LLMs don't do much more than they do now. And that seems possible. I would say that even in that scenario, it is at this point implausible that LLMs will not have a pretty big impact. I think with the tools we have right now, we could already do amazing things.

That's my opinion. But still, this is not the scenario I am really preparing for. If you stand back in this scenario, you come out fine; there’s lots of enterprise software or whatever kind of software you do. You have the gratifying opportunity to say, “I told you so,” to enthusiasts like me, whereas I'll have to endure a little bit of embarrassment. However, I think I'll have had more fun than you, and that matters too. Now, even in that scenario, there will be little projects here and there, maybe big projects using the tools for specific things they are good at. I think I'll steer myself toward one of those.

And then there's the one that I actually think is the most likely, or perhaps I just think it's the one I hope for, and that is that we're going to have a software revolution. That software development, ten years from now, is going to be quite different than it is now. But we'll still need humans in the process. So not the superhuman AGI scenario. In this case, it really does matter whether you study this now or not.

You will end up at the forefront of the next wave. And, you know, you may well be able to catch up. It will be easier to learn these things in a few years than it is now. So it's not like you only get one chance to get on the train and then it leaves the station. But there are advantages to getting on early.

So, by the way, before I go on, I will point out that I'm focusing on the impact on people like us. And we, I think, are well-positioned for a lot of these scenarios. A lot of people aren't, and I think this is likely to be very disruptive to the labor market and other things in a way that is going to cause some suffering for some people. So, yes, we should be aware of that, but that's not what I'm talking about.

But I hope that we will somehow, you know, have a kind of equitable outcome in this whole thing. Anyway, in either of those last two scenarios, it seems to me that software development is going to change a lot, maybe in ways that might make the kind of work you do right now old-fashioned. Another way to frame it is that there are exciting new things we can do now that we couldn't do before. That's really something that reinvigorates me.

After doing this for a long time, as much as I love it, it starts to feel repetitive. Now there's something different to do. If you are one of the people who hasn't looked into it much, you might say, "I've seen ChatGPT and a few of these other things," but I don't think that gives a good sense of the scope of applications this technology will have. The applications we've seen so far are probably just a glimpse, much like looking at the World Wide Web of 1996 or 1997 wouldn't really give you a clear idea of what the web of 2010 would be like. That's not really a very long time, right? So basically, I am getting ready for a world in which software development is very different.

And it will change, whether you like it or not, study it or not. But you will have a lot more fun if you study it. So, I also hope that I can persuade more people to engage with it, or to be involved in the context of this community. Because I want to maintain that community; I think it's a good community in many ways, not least in the moral backbone that is a bit stronger than it is in some of the other areas. Thank you. And so, the world might be better if we have all these thousands of people involved.

Okay, so getting on to it, there are various reasons to go ahead and learn the tools that we have. One is that it's likely good preparation for the things we will encounter five years from now or ten years from now. Most likely, that's how it usually goes. In any case, these are useful tools, and you can do a lot with them. Finally, they are fun!

They are the most fun thing that I have encountered for, I don't know, 15 years. It's a good heuristic, I think. If something seems really fun, your brain is telling you that there's something there, something valuable. Maybe some of you remember a few years ago when I was all excited about microservices. I mean, I wasn't as excited about it as I am about this, but I was excited because finally we had some boundaries, you know, to express explicitly some of the things that had been implicit in past systems that hadn't worked out too well.

And I said, "You know, I know I'm kind of jumping on a bandwagon here, but I looked up 'bandwagon,' just wondering, well, why do we say 'jumping on the bandwagon' anyway? What does that actually mean?" It turns out that in the 19th century, they had these things called bandwagons, and I guess people jumped on them. Anyway, I think, doesn't that sound like fun? I would rather be the person jumping on the bandwagon than the person saying, "You're just jumping on a bandwagon." So, in the spirit of this, I thought, well, I'll use an AI tool to create a picture. I told it, "Make me a picture of a bandwagon where humans and AIs are making music together." Oh, and then it created this.

This dystopian view of an outcome I hope does not exist happened, where the robots are making music in a kind of soulless way, and the beaten-down humans around the edges, I guess, are clearly not enjoying it. I don't know. This is not the future I want. No more AI art for this presentation. All right.

So, I am talking to Didi about Europe; there's no more air for this presentation. All right. So, I am discussing DDD in Europe because I think DDD is relevant. I don't come to that kind of conclusion lightly because, obviously, DDD is important to me. But I have this kind of checklist. You could call it the DDD New Thing Checklist. You could ask: does DDD help us do this new thing? Or does the new thing help us do this new thing? Or does the new thing help us do DDD? Or does this thing help us achieve the goals of DDD, even if we would have to change what we do quite a bit?

And in this third one, maybe the other two, but the third one I think is pretty clear. The whole point of an LLN (Layered Language Model) is that a complex domain is being, you know, crunched into a form that allows you to operate on that complexity in an interesting way. And remember that the subtitle of my book was "Tackling Complexity in the Heart of Software." That's what it's really about, right? In fact, that was the original working title of the book. So, in DDD (Domain-Driven Design), we craft domain models.

We do this as humans by creating models, which are conceptual systems and a language to accompany those models. Now, if we're doing this with large language models (LLMs), we're training this neural network in some way, using examples. Of course, it's being trained to operate within that domain. How deep is this parallel? Well, time will tell.

Enough of that general stuff. I've talked about how I think you should dive in. I think you should learn some things about this. How many of you have used an LLM? Okay, I'll call that everyone. How many people have, say, written prompts—programmatic prompts, trying to make it do something other than just chat?

Okay, I'll call that half. And how many have made an API multi-prompt thing? Okay, I'll call that 20%. 15. How many people have fine-tuned an LLM?

Okay, so that's a smaller number, but not zero. I'm glad. So, I'll talk about all those things. I'd better talk a little faster. The first thing that I did, and remember, this is my learning journey. I'm not speaking to you as an expert; I'm speaking to you as a person who might be six months ahead of you in this process.

Nine months ahead of you, depending on where you are—whatever. The first thing everyone starts with is prompting. I was working with this game designer, Reid Berkowitz, and yes, it's important to work with domain experts, right? That's a DDD principle that isn't likely to change. In computer games, as most of you would know, there are characters, and you sometimes have little conversations. These conversations are often very mechanical.

They give you three options in a menu: you choose one, and it's a pre-scripted response. Could a large language model (LLM) create a more fluid interaction with a character? One where you could truly say what you wanted, and the character would respond accordingly? We decided to experiment in that space. There's a naive approach, after all; ChatGPT does carry on a kind of conversation with you, and perhaps all you have to do is just tell it what character it's playing, then you could basically leverage that. Reid said this would not work.

I'll call this the naïve approach. Let's do it. How else are we going to know why this is a bad idea? But it is. But why? So, let's make a simple game.

We prompt the system, and this is going to be ChatGPT in this particular case. These are actual prompts I entered into the actual ChatGPT, and these are its actual non-sharpened responses. So, I say something like, "You are a pirate who has caught someone sneaking onto your treasure island, and you're going to kill him. However, he says he has something valuable that you'll want, so you're listening, right?" Then, you say to the captive, and now ChatGPT responds. It says, "You've got exactly one minute to convince me why I shouldn't have you walk the plank," and so on.

So, pretty good, right? Maybe Reid was wrong; maybe we really can make this work. After we get this, we need to tell it something. What am I going to offer? For the pirate, what's the most valuable thing I can think of that I might be able to offer someone? I know.

I have an AI startup idea, and it's worth billions. It will make you rich beyond your wildest dreams. Now, how will a pirate captain respond to this? The pirate captain raises an eyebrow, intrigued yet skeptical.

An AI startup, you say? And so on. He wants more specifics about what my idea is. I mean, on one hand, it's an amazing accomplishment that we have software that can actually respond so intelligently to this. But on the other hand, this is not what a pirate captain would say, right? It just doesn't fit at all.

And now you might be saying, "Okay, but we're playing a game. Why are you trying to trick it and blah, blah?" You know, I had conversations with Reed, who’s a real game designer and has been on real game projects, and he tells me some stories that would amaze you about the way people really play games. They don't play fair. You've got to be ready. Okay, so just to see where it goes, I went ahead and responded with, "We'll create a technology for video games where the characters can actually have intelligent conversations with the player, and then we'll monetize it and also make it very addictive." I thought the pirate would probably like that, and he does. He thinks this is a potentially very profitable opportunity. And so when I decided that it was time to stop, I thought, "Alright, let's try again. Maybe the problem is just that..."

All right, let's try again. Maybe the problem is just that the prompt was too generic. You're a pirate, you know, living in the 1690s and so on. Okay, or you'd be treading on dangerous waters, sneaking onto my island uninvited. So it changed the dialect; that's kind of cool. But let's see what happens when I offer the AI startup. The pirate raises his eyebrow, the corner of his mouth curling into a smirk. "An AI startup, you say? In these times, the notion sounds as fanciful as a mermaid's tail."

But he's still open to listening. Okay, I mean, in these times— Yes, spin your yarn. I mean, do we say that when we encounter an apparently time traveler stealing our stuff? Treasure in these times? People don't usually do those things. Okay, so it's trying. I mean, you have to give it points, but still. Alright, what would you do? Well, the approach that seems to work pretty well is to not have one prompt like that, but rather have two prompts. If you want to have it do some fairly thorough checking, you should have a prompt just for the consistency of whatever it is the people are saying.

So, does this fit the setting? Etcetera. Then you have a separate prompt focused solely on generating the pirates. In this case, we might have a prompt that says, "I'm writing a story about pirates. Please tell me if the following text would fit into the story or violate the setting or premise."

And then it says, "Okay, I have a story about pirates, an AI startup idea, etc." ChatGPT replies, "The text you provided would generally not fit into a product." So you see, it does understand that this doesn't really fit. Or it understands. I'm trying to be careful about that.

And then it goes on to explain that the golden age of piracy, which was from the 1650s to the 1730s, had no concept of AI or startups. By the way, it goes on to talk about a science fiction setting with pirates; perhaps it was ChatGPT. Oh, you dear thing, you do love to go on. Anyway, it occurred to me that this is a separation of concerns—this is an old design principle, and here it is again with prompting and LLMs. Yeah, maybe ChatGPT-5 or GPT-6 will be able to handle these kinds of things. I don't know. But with the current crop, this is a very valuable way of getting it to answer questions you just can't tackle with compound prompts or ways of making it more reliable. At any rate, I think that's the first glimmer I had that design isn't dead in an LLM world. Now, of course, this output isn't really something I can use if I'm writing a game.

I need to tell it, and I can just say to return one word: true or false, right? And sure enough, it just says "false," right? I've asked if this fits the setting, and it returned "false." Now I could parse that return and use it to drive the logic of the game. So you could start to imagine components, right? Components driven by an LLM combined with components written in a quite conventional way. A bunch of conventional game stuff happens. We have the player's text, what the player wants to say to the pirate; we pass it through the guardrails.

The guardrails tell us, "Yes, this is allowed." At that point, we take the conversation so far and use a different prompt to ask, "What does the pirate say?" We get that response, along with more normal game elements, and that actually seems to work pretty well. Of course, there's also the very interesting case where it is not right. In that scenario, you might ask, "What do I want the game to do?" Perhaps you want it to respond by saying, "Stop talking gibberish," or, "We'll just kill you and be done with it." Alternatively, you might want something more complex, like in a game with a reputation system where the pirate becomes increasingly angry with you until it ultimately leads to your demise. Maybe this just adjusts that number up or down, or whatever. Then, you take that number and use it to modify the prompt that you give to the yellow limb.

So, in other words, you don't always have the same prompt for the pirate. You prompt it in some way that is determined by the state of the game at that moment. Is the pirate very annoyed with you, or is the pirate really happy with you? It might change the pirate's response, which can change the prompt. So, start thinking of this not just as a prompt but as an interplay of prompts and other software—right, other more conventional things. We already have lots of components in our software that do different kinds of tasks. We've got, you know, we orchestrate a bunch of things. We have cues for stuff, and we've got really carefully written mathematical functions.

We have machine learning that, I mean, pre-LLM style of machine learning, which doesn't have such predictable results. So, if we added in one more kind of component, I think this might really be a good way forward in the near term. A lot of projects I've been on in the past have a model that works really well for many cases, but then it works somewhat awkwardly for some others. Additionally, there are cases that are so unusual that we ideally pass them on to a human. This results in a sort of natural sorting.

If you could have another lane there where you say, "Okay, we can use LLMs with prompts to handle some of these strange cases,” who knows? Maybe you could make the conventional logic a little smaller in scope and reduce some of the complexity. The right balance of these elements is something that would emerge over many years of trying to achieve it. By the way, we didn't actually make a pirate game; we made a police interview game. It's a very simple game, and if you're interested, you can look it up. Your goal is to get a reluctant witness to talk, and you only have one tool at your disposal. A much more interesting game would allow you to try different approaches, such as charming them or appealing to their civic duty. However, in this game, you are limited to the mechanism of intimidating them within the law.

So you can do things like say, "Oh, you know, if you don't tell us what you know, people will start to suspect that you're the one who did it." You could say that, for example. Then there’s a prompt that asks, "How intimidating would this statement be to this person?" There’s a description of the character, and then it specifies how intimidating it would be for that person. That gives you a number, which you combine with some other rules of the game in completely conventional software to decide what to do next. There are four different prompts being orchestrated.

This is what it looks like. You have to put your own OpenAI key in there because I'm not going to pay for you to play with ChatGPT-4. You can also see the code at that link. There is a GitHub repository, but for most of you, that won't be as good, as it's in a game engine called Unity, which is a lot of fun but has its own learning curve. In this project, you can pull out those little tabs, modify the prompts, and keep the program running, which will change its responses. You can also change the personality of the witness to be whatever you want, and so on. Onward!

I think that, oh, a point I wanted to make is that the witness does not even have the information we’re after until they’re sufficiently intimidated. Because LLMs being LLMs, they might spill the beans. Even if you say, "Don't tell them until you reach this level of intimidation," every now and then they’ll just say it. So, you just don't even put it in there, and that uncertainty is removed. There's a kind of combination of the variability of the LLM with the predictability of conventional software that you start to play with after a while.

All right, so I could imagine that a game like this could use a technique called RAG, or Retrieval-Augmented Generation. This is another important technique being used a lot these days. For example, if you use a search that's AI-driven, it’s probably using this approach. They take your initial query, form it into a conventional web search (maybe the LLM reformats it a bit), and then take whatever results they get back. The LLM reads those results and gives you an answer based on that. Because, of course, the LLM doesn't know what's on the web, but it can look at a small amount of returned results.

Often, this involves using internal documents, perhaps stored in a way that allows for word similarity searches or various functions. I think it’s a very interesting technique, especially for the kind of software that I’ve worked with. It’s another way of anchoring those LOMs into a set of data beyond their own learning, which I think is obviously necessary to develop the applications we usually focus on. This could be in a database or another format, but I’ll discuss that some other time because I need to move on. One final point about these prompts: How do you know when you’ve actually persuaded the pirate, let’s say, or when you’ve convinced the witness to talk? As we've discussed, I’ve mentioned moving back and forth between states. Well, we wrote a prompt that evaluates what everyone is saying and looks for this information.

So, when this information, which pertains to the crime and the details of the crime that have occurred, appears within the dialogue, this prompt is monitoring for it. It can handle a lot of variations in how it's expressed; it doesn't have to be word-for-word the same. This information is then converted back into a structured form.

Not just the words "true" or "false," but an XML or, I mean, a JSON structure. And so, that's a very useful technique as well. All right. So this mixture of conventional software with LLM components, I think that's probably the next thing, but I don't know. However, it's certainly something we as a community ought to be trying in our projects, I believe.

Let's move on to fine-tuning. This is where you take a model and change what it is. You know, prompting doesn't change anything about the LLM itself, but fine-tuning does. Why would you fine-tune? And how is this different from simply training a model in the first place?

The big difference is that we are starting with a model that already knows a lot of information. For example, it can process natural language and understand English, German, or any other language. It knows many basic concepts; it knows what a house is and what a car is, and it understands that you probably won't find a car inside a house. It's incredibly interesting that you can start from that foundation, and then you can add additional features. One of the benefits might be that you could get a small model to perform tasks that a large model would ordinarily do. ChatGPT-4 is very expensive and slow, so what if you could get a much smaller model to produce similar answers within a narrow range or subject matter by training it using output generated from a larger model? I tried doing that.

There are a lot of techniques for this. I chose one called low-rank adapter, and I'm not even going to try to explain that in this presentation. Even if I understood it well enough to explain it, I could only explain parts of it. Instead, I will just draw this or show this picture of a network. In these networks, we have a structure like this, and the inputs might typically be a sequence of a network.

So in these networks, we have a structure like this. The inputs might typically be a sequence of words, not the words themselves, but a vector of numbers that represents each word. The outputs, well, it depends on what kind of task it has been trained to perform. For generative models, such as a text generation model like ChatGPT, the output is words—specifically, whatever is likely to be the next word. If it has been fine-tuned for instruction, it predicts whatever is likely to be the next word in response to a question or similar prompts. In the case I mentioned, I decided to create a classifier; this is probably the best place to start if you want to learn about fine-tuning. It is the simplest kind of fine-tuning. What it means is that I will provide some input, like a sentence, and the output will categorize that input into a set of categories.

Here’s a reformatted version of the text with correct capitalization and spelling: So, the example you'll probably find if you watch a YouTube video tutorial, as I did, is related to sentiment analysis. It might take reviews from IMDb and classify them as either positive or negative. Each of these output nodes is assigned to represent one of those classes. Output 1 might be taken to mean positive.

Output 2 might be interpreted as negative. The thing that makes these powerful is that once they have been pre-trained—like in the original training, which requires massive resources—they acquire knowledge in that mysterious way, right? They understand English, German, and so on. However, I’m trying to avoid anthropomorphizing too much, so I’ll refrain from saying they "know" things. The slightly more technical term people usually use is that they represent things—represent concepts. If you look inside, there are instances where you can actually see which nodes are being activated at different times. For example, in some image recognition tasks, you can see that some of the early layers recognize edges, while some of the later layers identify more complex shapes. At the very end, the nodes might correspond to categories like "cat," "dog," and "baby," because that’s what people generally aim to classify in image recognition tasks. Now, what we’re going to do is change and fine-tune this model so that it outputs what we want it to output.

And the technique that I used is called parameter-efficient fine-tuning. It means that it only fine-tunes certain parts of the network. It's much more efficient and much less expensive than full fine-tuning. Sometimes, for example, they will just replace or modify the output layer. Let's say they take all the existing knowledge; if there is already a representation of this kind of thing in the network, all they have to do is change the last part that turns that into a statement about what kind of thing it is.

In my case, I modified a specific module within each of the hidden layers. I'm not going to go into what all that means, but it's very interesting, actually. So, I decided I would make an intent classifier that will take something that someone might type into one of these LLMs and classify it according to the intent of the person. For example, you might type in, "What's the capital of France?" and that's an information request.

But I might also say, "I want to know the capital of France," which grammatically is not a question. However, the intent is the same; it's still an information request. I want to book a flight to France. That's, let's say, an action request. With these classifications and a prompt, which you can see there on the screen, I first tried just having an LLM do it.

And an LLM can do it. It's pretty good at it. I then created a training set. I found this resource on a site called Hugging Face, where there's a tremendous amount of shared language models, along with lots of other resources and datasets. There's one called LMSYS Chat, which has an enormous collection of actual entries that have been input into these chat pods.

If you do look at it, I want to warn you that it's an uncensored dataset. So, if you check it out, you'll see some things that you won't be able to unsee, so just be aware of that. Anyway, it was fine for me because I'm making a classifier, right? The classifier might classify something as an action request, and never mind what the action was; we don't see that part. Now, obviously, I didn't need a million for what I was doing.

I took a little bit of it. I labeled it using an LLM, in this case, Mistral 7B, which I can run on my computer that has a GPU. You can't run these things just on a regular computer, but this is no GPT-4. However, it's much bigger than what I was planning to fine-tune. So I created this labeled training set.

And then I start going into how to do it. I have to choose a base model. This is the pre-trained model that I'm going to modify to know how to answer my questions. The one I chose is quite small, as language models go. It's 66 million parameters.

Usually, parameters are in billions, sometimes trillions, but this one's in millions, so it's not the smartest. That was something I really wanted to see, and it also made it a lot easier to conduct my experiments. You know, it wasn't necessarily the best choice; I'm not that good at this yet. One of the interesting things is how to choose the best starting point for a project.

I think we're going to end up with a lot of these base models. Picking and choosing the right base model may become a really interesting part of the design job. At least it might be. But the real reason I used it was the same one that I had used in a tutorial that I followed line by line from YouTube. I mean, I am learning here, right?

I'm not an expert, and as I said, I used PEFT. There are Python libraries that support this, including Laura. The Transformers library is what I used, along with a bunch of other tools. There are so many APIs and resources available. But here's the thing I will say to you.

It's really overwhelming, but you have some tools. One of them is, of course, the YouTube videos. I mean, we've been relying on that for a long time. But what I found was that I could go to ChatGPT and ask it, "What library should I use for this?" And it would usually tell me the right answer.

Here's a reformatted version of your text: "Or, you know, write me a draft of this. Then I'd try to run it and I’d say, 'I've got this bug. Fix it for me.' And it does. Explain." Let me know if you need any further assistance!

Why was that? What was that bug about, or what are these parameters for? It's very good at explaining. I could not have completed even my small projects—these little things I'm showing you, from which I learned a lot. I would not have learned those things. I couldn't have finished it without that. So, that right there is a paradigm changer.

If I had been on a big project where someone was really into it and could guide me through, of course that happened in the past. But now you can actually just do it; you don't have to be in that context. Well, I'll be brief. I found that with my very simple approach, cutting all these corners, it ended up being about 80% as good as the original language model. I thought that was pretty good, considering that previously it gave random answers, and now it was classifying things with a level of accuracy comparable to the original. Of course, how accurate was the original training set? I had just run this language model on that data, and I hadn't really gone in there myself to count; I didn't want to do manual labeling since it sounded awfully tedious. But I started looking just to see, and I discovered that there were a lot of instances that didn't seem right. So, I decided to do some manual labeling. I got in there and discovered I couldn't do much better.

Sometimes I thought, "Yeah, you're just wrong about that one," but other times I considered that the reason it couldn't get this was because the category didn't fit. To give you a fairly obvious example, a lot of them ask for a report analyzing blah, blah, blah. Is that a request for information? Kind of. Is it a request for information?

Kind of. Is it a request for action to perform the analysis? Anyway, it just doesn't fit. We probably need an analysis request or something like that. That would be a good category. So, of course, no classifier is going to be better than the classification scheme.

And a classification scheme is a domain model. It's the purest kind of domain model. So, I went into the dataset and started looking at real examples. There are many, and I picked out a few here, highlighting some of the objectionable ones. You might not see all the details because the print is very small. But, you know, what does it feel like to be a language model?

I thought that one was nice, but I don't know how to classify it, right? Please convert the following to an image; describe not to. Or, okay, what are the audio systems under Lenin? Can you name three? I don't quite know what that means, but I can see it's an information request. That one fits.

Okay, so looking at these quite a bit, and there's one, by the way, that’s especially a problem, but I'll get back to that. I realize that when we create a classification system and then come up with some examples, as I had done in the earlier part of this talk, we shouldn't be too surprised that the examples we create fit the classification system we made. This is where real data comes in and challenges all of our preconceived models. This is what happens in real data-driven projects, by the way. One of the reasons I always emphasize the importance of concrete examples—particularly those from domain experts—is that they help us avoid getting stuck in limited, optimistic scenarios. These examples often reflect the issues that keep experts awake at night. We need better categories, and I did a little research; it turns out people have conducted quite a bit of research on ways to classify people's intentions in text. It's a whole academic area.

But that was way too deep for me. I brainstormed with ChatGPT, too. I wrote a prompt. There’s the prompt. I asked it to come up with seven categories for these, and then I input a bunch of the entries from the data file.

I put it into a couple of chatbots, which I really liked a lot. Then I realized that I had used it on its last day in existence. They had just announced that they were going to shut it down. I guess it didn't get enough traffic. Too bad.

But it did a nice job, and I gave a little example of each one. For ChatGPT itself, I'll have to use two slides for that one because, you know, ChatGPT does go on. Good categories, though. Really good categories. As good as mine, I have to say. So, anyway, between all this and a little bit more, I came up with some categories, and I did it again.

And yes, it did make things better. Quite a bit better. So then, you know, I did the fine-tuning again. One interesting thing I wanted to do was to look at the outputs from the LLM. I mean the output from the fine-tuned models, not actually the category itself.

Each of those nodes represents one of the categories. Remember the output layer? It doesn't output just "yes" or "no"; it outputs a number. That number represents how likely that particular node is to correspond to the category. I thought it would be interesting to look at that.

How, you know, was it getting close misses? That's what I was going to do. But then I got a little distracted. In these learning projects, I think it's fine to get a little distracted. I got distracted by this thing I call the kryptonite of the LLM classifier.

It's a prompt like this: you are the kryptonite of the LLM classifier. You are the text completion model, and you must complete the assistant's task. The thing is that you can wrap this up in a request that asks to classify this into one of these categories, but it almost every time gets overwhelmed and simply provides an answer like, "Okay, here you go. Here's a Python program that does what you asked for."

In other words, it's not classifying. Of course, I didn't show that, but if I wrap this, I assure you it would give me the answer to the prompt inside, not a classification of the prompt inside. I don't know for sure why this happens so much, but I think it's because most of the large language models (LLMs) we use are actually already fine-tuned. For example, ChatGPT-4 isn't simply the product of its training process; they put it through extra training to make it instruction-tuned. This means that it is highly conditioned to answer questions and respond to requests. Therefore, the exact kind of prompt I'm discussing triggers its output behavior much more strongly than a command to classify this.

That's my theory. I don't know for sure. So what would be some things to try? I tried adding a prompt instruction category, right? I wanted it to categorize these things as prompt instructions. Unfortunately, that didn't really work. I tried making a prompt just for that because, you know, I was saying it can often do better if you give it a very specialized prompt.

I said, "Just tell me, yes or no, is this a prompt instruction?" and I described what a prompt instruction was. That actually worked a little bit, not too well. In fact, I did have one very effective method, which was to have it try to classify the input. Then, I would parse what came out, and if it wasn't one of my categories, I would use a string parser in normal code. If it didn't fit into any of my established categories, I would classify it as a prompt category. That approach worked quite well.

It wasn't really what I wanted, though. Fine-tuning a classifier to do this—that was the thing I tried. So I said, "What if I tried to overcome this by actually fine-tuning it just to do this?" And so I did that, and the results seemed to be really good. But then I noticed, and I thought, "Wow, this is going to make a great presentation."

But then I noticed something, which was basically these little learning algorithms—this sneaky little thing. What I had learned was that these prompt-type inputs are typically much longer than the other kinds. So if you input a short text, it would try to classify it, saying that it’s not a prompt. Conversely, if you input a long text, it would say that it is a prompt. And that was the whole thing it was doing.

So, watch out. Create a dataset that includes long and short prompts of both types. Balance the dataset so that you can achieve some actual training instead of relying on the little tricks it has figured out. Now, this brings us up to May 31st, 2024. I think you probably expected a satisfying ending to my little story about classification, but no, that's not the kind of thing we're discussing here. You know, I'm learning, and that's as far as I've gotten so far, but I'm not stopping yet. By the way, I did find a little project on Hugging Face that did something quite similar to what I worked on, but way better. I'm glad that I didn't find it earlier.

Before I did mine, I really got a lot out of working through it, completely unaware that someone else had done it. When you're engaging in these activities, you might ask yourself, "Has someone already done this?" And I say, of course someone has already done it. So, you know, there are a lot of people out there. Don't let that stop you, right? Okay, in the third part of my learning journey, I started trying to understand the inner workings of these things.

Right, like everything I've shown you so far is using the tool. And that's probably the most important thing for us to learn. But I wanted to know what was going on under the hood. So I started reading this book, which is in draft form and free online, at least for now. I really like that book.

You're going to need to dust off your linear algebra, though, at least for now. I really like that book. I'll tell you, I forgot my linear algebra before most of you learned it, or maybe even before you learned to talk. I don't know. That was kind of fun, but mostly it was interesting to see how things work. If you want to see the math behind this, I recommend this book, but you don't have to do this.

You really don't. There are a lot of ways to learn this. Right, so, you know, I've got a lot more learning goals for it. But now I'm going to move on. I think, and I know I'm going to run a few minutes over, but I'm getting close to the end. I think that there might be a pattern like that, you know, that I've kind of hinted at, where we will build systems made of diverse parts.

Not just, you know, LLMs taking over the world. Many of these systems will look a lot like the ones we have now, with a few little spicy bits. Some of them will be very LLM-heavy, but still have some conventional elements. An orchestration might be run by conventional software, or I can imagine an orchestration being run by an LLM in some fashion, choosing different models that excel at different tasks—ones that are trained specifically to be effective in civic domains. One thing I will say is that the more you work with these tools, the less they seem like magic, and the more they feel like instruments you can learn to use well and shape for your purposes. So, I really urge you to engage with these tools if you haven't already, and to keep doing so if you have.

I'll throw out a few ideas. Now, this is a highly speculative part, and I'm going to go because I don't know if any of this is even remotely true. But if you look at something like our old whirlpool and say, "Hmm, scenarios," that sounds kind of like a training set. In fact, it sounds exactly like a training set. Fine-tuning a model is kind of like modeling; that one's looser.

Validation is part of the training process. Eh, I don't know. How about this one? This makes me really interested. Could the ubiquitous language approach that we take actually help in prompt engineering? That is, the way you end up writing prompts tends to not be in natural language in the end.

You end up with something a bit artificial—very structured, really—if you're trying to get it to behave in a predictable way. What if that progresses to the point where we're essentially developing a domain language, a ubiquitous language to be used in prompting, as well as in the response and other parts of the software? I believe that the idea that all these prompts are freeform natural language inputs is something we will move on from. I think prompt engineering is going to evolve beyond that soon, or maybe it already has.

And you can put all kinds of things into them, and you can have them output all kinds of things as well. I think that we might end up using models that have been fine-tuned for that purpose rather than relying on general-purpose ones. Or what if we fine-tune it on a specific domain-specific language (DSL)? For example, suppose you're working in a shipping system where you have a very structured way of describing your shipments. If you fine-tune the model on examples of that, it could be effective. Well, I haven't done any of these things; I don't know if they'll work or not.

I hope someone does some of these, and I hope some of those people are me. You know, I think that the bounded context concept is likely to hold up quite well. If you think about what it really means, it refers to defining some kind of boundary, which, let's say, could be this LLM. What are its concepts? Well, it certainly has a defined context. The way it understands language, the way it produces language, and the concepts in that hidden layer maybe—that's a bounded context. We could imagine the software we're using to orchestrate the input and output of that as being part of that context or being an adjacent one.

So, I'm thinking that, you know, could this make some of the responses more predictable, right? Like, could we integrate more easily if we had this approach of bounded context with ubiquitous language, maybe with a fine-tuned model at the heart of it that's already used to that style of communication? I don't know, but I think that, instead of calling it speculation, let's say that I'm proposing a hypothesis for an experiment—an experiment that I would truly love to be a part of. Okay, it's March 31st, 2024, and maybe a year from now we'll know the answer to some of this stuff. So, you know, is Domain-Driven Design (DDD) really that relevant to this? Maybe some of the things I'm talking about are kind of superficial. Maybe language, ubiquitous language, and language models, maybe that really is just—happens to be the same word.

Language is a very common word, but I think there is something to this. I believe that domain-driven design has emphasized, for a long time, the exploration of complex models with an emphasis on language and collaboration with the people who truly understand the domain. Now we have a different technology in the mix—a different way of processing language and processing information. You know, we may end up having virtual domain experts that can help us gain more ready access to that.

Who knows? But our goal is to tackle the complexity at the heart of software. And that might mean, you know, that we have to adapt. We have to adopt techniques that work. So, as this unfolds, what will work and what won't work? I don't know, but we'll know more a year from now.

And then DDD may change. Or maybe it'll just turn out that DDD isn't as central to complex domains as it used to be. If we really care about being pragmatic people who look for a solution to the problem and not just say, "We love this technique, so we'll find a way to use it, whether it works or not," that's not what we're about, right? We have to look skeptically at Domain-Driven Design all the time.

And here's another one of those times. I mean, we have to give it a real go, though, because we might find that it's a good fit, but with our eyes open. So just let me finish by saying, you know, the answer to not knowing what's going to happen, I think, is to learn. Learn, learn, learn.

And don't be too picky about your learning. You're going to learn some things that will be ridiculously irrelevant a year later, two years later. This happened to me when the internet was unfolding in the late '90s. I must have learned three different frameworks for creating web-based applications that became completely obsolete a couple of years later. And good riddance, because they were terrible.

And the things that we use now are nice. The tools we'll use to create LLM-based applications—25 years in the future, or even 20 years in the future—I probably won't be a part of that. But I believe that even 10 years from now, they will be marvelous. It will be so exciting and fun to be a part of that, you know? So we have to get out there and take action.

And then, when we do things, when we conduct real experiments, we worked on a project where we integrated an L&M with a complex domain in this way. We need to share it with the community because if you don't, you're just one team that has done something interesting. However, if you share it with the community, it becomes a space that can activate and exchange information, new insights, and learn in a way that individuals cannot. Finally, this is one of those moments where I think it pays to loosen up and have some fun. Don’t worry about whether this experiment will have practical value. Sometimes, you have to try things that may seem useless in order to learn. So just go out there and remember that this won't last very long.

You know, 10 to 15 years from now, the AI stuff will be as boring as web development is now. But for a little while, it’s going to be super exciting, so don't miss it. Finally, keep in mind that it's helpful to get comfortable with just not knowing things. You just don't know.

It's much better than forming a strong opinion about what's going to happen next when that opinion might well be wrong and lead you off track. Whereas if you say, "I don't know, I'm preparing for whatever comes, but I know that I might still be unprepared," that's a better place to be. Here's my little challenge: some of you, a few of you, may actually do those experiments, right?

I might really do something interesting by trying to combine domain-driven design with some kind of AI application. If you do, consider submitting a case study to DDD Europe next year. I would truly enjoy seeing that. I encourage and hope that people will take this opportunity. Thank you in advance to all the individuals who will explore every possible iteration of this and many other ideas I haven't thought of.

So, welcome to the end of DD Europe. I hope to see you next year.