Ed Jones: “Fearless Refactoring & the Art of Argument-Free Rust” | Rust Global @ RustConf 2024

Hello everyone, I'm Ed, and today I'm going to talk to you about a kind of unsung superpower that I think Rust has, as well as a way in which our company has been trying to use this. For the purposes of this presentation, there are three things that you need to know about me. Firstly, I work at Canonical in the office of the CTO, where, among other duties, I am responsible for promoting Rust among our various teams. I'm the guy who goes around and says, "Hey, you have a problem. Have you tried using Rust?"

I think it will be really good, and we are having some success with it. I believe we will see that there are some lessons we have learned. I also really like clean code; there is a certain zen to it.

And when I find a piece of code that I don't understand, I immediately think, okay, do I not understand this because it's complicated? Or do I not understand this because it's a little bit weird? That weirdness is something I generally like to avoid. This kind of rubs off slightly on others, and I have the effect of really annoying my coworkers over this. I'm that guy who, on pull requests, will be there saying, after everyone else has approved, "Oh, there are actually like 20 things we need to go through here."

I don't think this is quite ready. Sorry. Yeah, I'm not too keen on that. But anyway, before starting in earnest, I would like to tell you a little story about a project that we were working on a couple of months ago where we had to limit the amount of CPU time used in a part of one of our processes. For the sake of solving this, we needed to have a solution that was quite lightweight. It had to work effectively in a very asynchronous environment.

It needed to be simple, and it proved to be quite a challenge. The idea we came up with was that the program would declare a certain amount of work it had done, and then we would bound that at runtime to prevent the code from running away. We needed to relate this back to reality, and we did this through testing. Essentially, we conducted tests over multiple input sizes to ensure that the amount of time declared as being used on a particular green thread was above the actual time used.

Now, wouldn't it be great if this were actually reality? It's not. The data was very noisy, and the property we were looking for is that the white line is always above the orange one. If we can bound the white one, then the orange one is not going to go too badly. This data here has a lot going on. I'm not going to go into too much detail as to why it's so noisy.

But what we essentially needed to do was reduce the amount of noise because a lot of this data was not replicable. It would happen once, and then maybe it wouldn't happen again. Having flaky tests was something we really wanted to avoid. So, we denoised the data to make it resemble this more clearly. The problem was that denoising it like this was really complicated.

The methods that we needed to implement simply weren't available. My coworker and I had to look at papers and translate some code from GNU Octave. Essentially, we had to create a mathematical model that roughly represented the noise we were experiencing so that we could then reduce it. This process took about two months of work, which was quite grueling, I might add. The result was a substantial amount of code that was heavily steeped in mathematics.

There were a bunch of seemingly magical constants that emerged from that. However, when we examined the situation, we concluded that currently, between one and two members of this team understand this code, and in the future, that number is only going to decrease. So we decided to scrap the code. We recognized that it would simply not be maintainable, so we opted for a much simpler approach. We re-evaluated our requirements and discovered that if our goal is simply to prevent users from running away, we can take a straightforward approach by ensuring that the testing encourages users to engage in approximate counting, rather than requiring perfect counting as these graphs would suggest.

This kind of got me thinking: what could we really learn from this? I believe there are a couple of challenging problems that we as an industry face—not necessarily hard coding problems, but rather difficult people problems. Firstly, we need to elicit requirements. In our case, the two of us who were writing the code had slightly different versions in our heads of what was required compared to what the business actually needed. It took some negotiation to guide us from the initial path we were on to a more collaborative approach that ultimately led us to a solution.

Secondly, there's a certain amount of politics involved. If we weren't careful, and if we were really pushing for what we believe to be right, then although we may have won that argument, it would have been quite unhealthy for the process. This was a good lesson for us in learning to lose certain battles strategically in order to win the war. Finally, maintenance. Maintenance is a tough one because, unlike the other two aspects here, you don't really have anyone to talk to when it comes to maintenance. Best case, you've got some good documentation, or maybe the original person who wrote this.

And if you don't, good luck. That's not going to be fun. So, what is maintainable? At its core, I think maintainable is quite difficult to define, but here are some properties that I've come up with. Firstly, if you have very easily described concepts, then I think that's quite maintainable.

If you can explain to someone what a little bit of code is doing, then they are going to be more comfortable with it. If the code can describe itself, then they will have a much easier time working with it in the future. When multiple concepts come together, you need to ensure that these concepts flow naturally. If there is an awkward break, it will cause some real head-scratching later. Finally, we need to make sure that we have very predictable patterns. Although this may lead to localized sections of code being slightly less clean, the overall quality of the codebase will improve. This means that the person reading the code will have to keep fewer patterns in their head as they go through it.

So, who are these people for whom we are trying to write our code? Firstly, our coders should be writing for our customers, I hope. Secondly, they should be writing for their coworkers—those who will be building on their work and will be the ones looking at their API and thinking, "Oh, that's a little bit strange. Why did you do it like that?" If you're in close contact with these people, some interesting conversations will happen.

But of course, you're not always in close contact. What if your co-worker is someone on the other side of the world who is downstream from you and is reading your work, trying to make the best use of it? Finally, we need to think about ourselves when we are coding, but not just about our current selves. That's an easy target. We need to consider who we will be in six months, 12 months, or even 18 months, when all of our current decisions may have been forgotten.

When you look back at what you wrote and thought, "What was I on at that moment?" it can feel really odd. I think we've probably all been there. This kind of brings us to the notion of fearless refactoring. We've identified some problems. How can we start to address them?

And this is a place where I think Rust really shines. I would define fearless refactoring as the ability to make changes and ensure that, the next time you run your checks, your code still behaves as expected. Thanks to its type system, the fantastic error messages the compiler provides, and the comprehensive linting and ecosystem surrounding Rust, we find ourselves in a uniquely advantageous position among the languages commonly used in industry. While refactoring may not always be easy, we can approach it with confidence, knowing that we will likely end up with code that works.

Which gives us a unique opportunity because it means that once we have working code, we can easily go back and improve it, which is nice. There are a couple of things we can do to help the people who will be working with our code in the future. Here are five techniques, or best practices, that I've found useful. There are many more that will be mentioned later. Firstly, I would say you should avoid using `unwrap`.

If you can explain something using the type system, your code will be far easier to work with and far less error-prone. However, if it's not possible to reasonably and simply express something in terms of the type system, I would argue that using `expect` is acceptable as a last resort. At least by using `expect`, you're forced to document the preconditions that will have been violated if you used `unwrap`. Secondly, I would advise matching exhaustively. If you are matching over an enum exhaustively and someone else adds a variant, they will receive an error message pointing to your section of code, indicating that there is an unaddressed case.

Go do your homework, dot your i's, cross your t's. It forces the person who is making that change to propagate what they have done throughout the codebase. As a result, you no longer end up in a state where your program compiles, but it isn't actually 100% correct, which I think is really good. I would also advise mutating quite small scopes. Mutation really requires you to keep invariance in your head.

And the fewer invariants you have to keep in mind, the easier the code is to write. I'd also advise really avoiding the use of unsafe code. You're turning off your compiler's checks. If the compiler is unable to help you, you're the only one who can help yourself. It is just far harder to work with.

So, you should really ask yourself the question: does that 1 or 2% speedup actually merit losing the safety that the compiler would otherwise provide? In most cases, I don’t think so—unless you can prove, through profiling, that your hot loop genuinely requires every last bit of performance. I would always advise opting for safe and more maintainable code. Finally, I believe we need to use "weirdness," but quite sparingly. A certain amount of weirdness is inevitable when we are gluing together different systems that have varying models for how to represent your problem.

So we're always going to end up with some of it, but we can use it strategically. I want to explore weirdness for a moment here. In my definition, I would say that weirdness is somewhat equivalent to surprise. For example, you look at a piece of code and think, "Oh, what's going on there? That's weird." If that happens, you should probably do something about it if possible.

So, when weirdness occurs, it really diverts attention away from what the reader was already thinking about and focuses them on a particular part of the code. If a codebase is weird, they'll be met with so much noise that it's just going to be unpleasant for them to work with. However, if you restrict weirdness to very small, specific places, it underlines where extra care and attention need to be paid. Therefore, I think it can be quite useful. Of course, we shouldn't strive to be weird. Fundamentally, I believe the best way to avoid weirdness is by having a debate.

Because oftentimes this weirdness comes from the idiosyncratic things that one coder may have done that another coder might not be 100% familiar with. So if we discuss, things will just naturally get better. This kind of brings me to argument-free Rust. I really want to focus our arguments on the places that matter. However, it's worth noting that I would differentiate between arguments and debates.

Fundamentally, I think a debate is a positive action, as multiple parties come together to try to settle their differences. In an argument, however, one or both sides may come away feeling very bad about themselves. They might feel unheard or find the process much more difficult than necessary to achieve what they wanted. I don't think that's very productive, and it can be quite draining as well. If you have a team that is constantly engaged in arguments, you're likely to see more churn, increased turnover, and a rise in apathy, which we definitely want to avoid. Therefore, we really need to foster constructive debate.

So at this point, I'd like to describe a little bit more about how our company works and how we have tried to foster debate within Canonical. Canonical is a company of about a thousand people. We hire all over the world and are currently hiring. We don't discriminate by time zone, which, unfortunately and fortunately, means that we end up with people in very distant places, making it difficult to have conversations.

If I am working quite closely, I am based in the UK, with someone in Australia. One of us is going to have to wake up and attend a meeting at midnight, possibly repeatedly, and that's just not very nice. I wouldn't wish that on anyone, quite frankly. So we need to find a way to have these debates that is both productive and globally considerate. We came up with a couple of approaches for this. Firstly, we tried to use a cross-team code. We just set up a channel in our company messenger, Mattermost, where people could post their Rust pull requests, and then people from other teams could come along and comment on the patterns they had used.

In this way, we built up a shared sense of community and some of the idioms that we're all accustomed to using. Unfortunately, there wasn't really a consensus here, and we need that consensus across all of our codebases if we are to succeed in Rust. We have a thousand people, but we also have hundreds of different projects, it seems. So, it's a real challenge. The next approach we tried, roughly at the same time as the cross-team code reviews, was having open office hours. We would gather a group of people to chat about Rust.

It's a bit of a social event, and there, it’s a lot easier to build up and agree on one shared sense of what our vision of canonical Rust should look like. This also ran into the same problems, as it’s quite difficult to talk to someone on the other side of the planet. So, this kind of merited a different approach, which I think is important. What we came up with was standardization documents. This has probably been the most effective approach, where we literally just document all of the style guidelines we have in one place for everyone to see. So, what does one of these standardization documents look like?

The content here isn't too important, but roughly speaking, we have a description of the concept we would like to standardize. There is a short argument—for lack of a better term—explaining why we would prefer it to be this way, in hopes of convincing those who might otherwise disagree. Additionally, there are visual examples illustrating what we are aiming for and what we are trying to avoid. A lot of the material in these documents is quite low-level and straightforward. Here’s a list of some of the items included, which is an excerpt from the table of contents. While the specifics may not be crucial, you will notice that much of this information is relatively simple. We want to avoid debates over topics that don’t significantly impact our work. You and your coworkers have a finite amount of energy to devote to these discussions, so it is important that our debates focus on interesting issues rather than trivial ones. I believe this approach will yield better results.

Every time we have one of these low-level debates, we can always just point to this and say, "Hey, this is the way we're doing it." So how did we put one of these together? And how would you go about putting one together for your organization? Firstly, we took note of the repeated arguments that we were having, where different outcomes were occurring, so that we could standardize upon them. If you do any coding in your own time—an exercise I did as well—just as I'm writing, I have in the back of my mind, "Oh, if I've done something, and there could be another way of possibly doing it that is equally good but just different, I'll note that down and maybe discuss it later."

Once all of these points have been gathered, you should write them up. Focus on the generally applicable ones to keep it relatively short. These documents will be quite lengthy, so you need to be selective. After the write-up is complete, have a discussion with the most experienced consultants you can find. There will likely be many changes that need to be made.

That is a good thing. The extra experience that they will have brought from other places will allow for far better-informed decisions regarding the conventions you wish to enforce. Once you've worked with the small group, you can then expand to a larger group of participants from all levels of ability. Now, the key here is that we want something on which everyone can roughly agree without it feeling like a design-by-committee exercise. We are trying to create something with a very strong personality, and if everyone contributes their ideas from the beginning, it might get a little messy. Therefore, a certain amount of restriction is necessary here.

And finally, you do a 1.0 release. Of course, there's always more stuff that is going to be added to these things in the future. But the releases come at the moment where you say, "Okay, guys, this is what we're going to be doing. This is how we're going to be doing it." So that's kind of the process on hard mode.

I appreciate that not all organizations will have the time and money available to dedicate a couple of people to work on something as low level as standardization. So, I want to offer you an easier option. All of the guidelines we are putting together are fully open source. You can use them if you like; please do. If you agree with them, thank you. If you find yourself in agreement, great—fantastic!

Please take them and use them. If you disagree, we have an RFC process, and I would be very, very happy to have debates on all the minor nitty-gritty points about this. Even if you disagree with some parts of this, feel free to fork it and use it for your own organization. It's licensed under GPL version 3 to keep it nice and open.

"Go for it," is what I'd say. So, what's next in this effort? We have our standardization documents. We possibly have our developers reading these documents and applying what they have learned. What comes next is, well, more standards—sorry! Because as this process continues, we're just going to find more areas where we have trivial differences, and we can create standards for those as well. There are standards associated with these.

But reading all of these large documents is going to get a little bit taxing after a while. So I've actually been running a bit of an experiment into automating many of these tasks. It's not possible to automate every single one of them, because a lot of them have their roots in natural language, and I don't want to go there, quite frankly. However, there are quite a few of these guidelines that can be mechanically enforced. So, on my GitHub, if you can.

But there are quite a few of these guidelines that can be mechanically enforced. So, on my GitHub, if you come and find me, you will be able to see some of the work I’ve been doing on that, and I hope to present it at some point in the future as well. On a more human front, we’re also going to have open office hours. The office hours that we have inside the company will be opened up to others to join. If you want to have a discussion about these minor differences you might be encountering, feel free to come along.

I'm going to share some details about that in the Discord, and if this is on YouTube, in the comments or description, I guess. And that's about it. Thank you for listening and have fun!