"Transactions: myths, surprises and opportunities" by Martin Kleppmann

"Transactions, Myths, Surprises, and Opportunities" is a title I considered, and I eventually settled on it because it conveys a sense of optimism. My original working title was more along the lines of "Transactions, Joys, Challenges, and Misery," possibly with a subtitle of "Consistency Guarantees." You want consistency guarantees? Consistency guarantees! So, as you can see, exploring this topic has driven me a bit insane, and it's possible that the forbidden knowledge in this talk could drive you insane as well.

So, this is a little experiment to see if I can drive 200 people insane within the space of about 40 minutes. This talk came out of some research I did for a book that I've been writing for ages and which still isn't finished. However, this talk is kind of an extract from Chapter 7 of this book called "Designing Data-Intensive Applications." You can find Chapters 1 to 8 in early release at data-intensive.net if you are interested. Anyway, when we're talking about transactions, I should really briefly go back into the history of where they actually came from.

The origin of the concept can be traced back to the mid-1970s when IBM was developing what is now regarded as the first SQL database, known as System R. This groundwork laid the foundation for many subsequent relational database systems, and if you examine the architecture of contemporary relational databases, you can see that many of the ideas can be traced directly back to System R. The concept of transactions established in System R remained relevant for approximately the next 30 years, until the late 2000s, when the NoSQL movement began to gain traction. Subsequently, an inevitable counter-movement arose, referred to as NewSQL, though the exact implications of that term can be ambiguous. Ultimately, the discussion often centers less on SQL itself and more on transactions. The specific query language and data model being used are somewhat significant, but the transactional guarantees provided by these databases are what truly differentiate these technologies. When discussing transactions, it’s essential to mention ACID properties, which have been referenced many times. However, I would like to delve into what this means in more detail because we often assume we understand it without fully unpacking its implications.

We think we know it means Atomicity, Consistency, Isolation, and Durability. But what do those words actually mean? If you take a closer look, you may realize that they're a bit vague. Eric Brewer put this very politely when he said that ACID, as a term, is more mnemonic than precise, which is a polite way of saying it doesn't really mean anything specific; it's just kind of a marketing term that we use. Nevertheless, there are certain definitions associated with these concepts.

Let's start at the back with durability, which is kind of the easiest aspect to discuss. Back in the day, durability meant writing your transaction redo log to an archive tape. Disks were not very reliable, so you could ensure that whatever you wrote to the tape would be appended sequentially to that tape. If your disk died, you could replay the tape to restore the state of the database. However, tapes eventually fell out of fashion, and disks became the norm. As a result, people began defining durability as the database syncing to disks to prevent data loss in the event of a crash.

And then more recently, that's kind of been replaced with replication as a new way of thinking about durability. But that's really all I'll say about durability; otherwise, there's nothing too interesting there. What about the "C"? The consistency bit? Now, it should hopefully be clear that the "C" in ACID is absolutely not the same as the "C" in the CAP theorem. If somebody comes up to you and says, "We can't have ACID because the CAP theorem says so," then I suggest you very quickly run in the opposite direction because you're probably not going to have a very productive conversation.

So, I wrote this paper about CAP, analyzing it in a bit more detail if you're interested. We discussed it last night and are going into detail today. The "C" in CAP is kind of an odd one, in that it doesn't really have the same kind of structure or property as the "A," the "P," and the "I" do. Joe Hellestein suggested that it was basically tossed in by the authors to make the acronym work, but they didn't really attach much importance to it. So, "C" is now generally associated with the database. The transactions that the application executes on the database move the database from one consistent state to another.

Where consistent state is defined through meeting certain invariants. These invariants could be expressed as integrity constraints that are checked by the database, but they might also be implicit. For example, in an accounting system, you might have the invariant that all of the credits must sum up to all of the debits or something similar. In some systems, you can express that kind of invariant declaratively, while in others, you cannot. However, the odd thing about consistency is that it is fundamentally a property of how the application uses the database; it is not a property of the database itself.

So that's why we won't really go into consistency in too much more detail either. What about the "A" in atomicity? This is a kind of confusing topic, because in the context of multi-threaded programming, when discussing concurrency, you have concepts like an atomic increment. This means that multiple threads can increment a value simultaneously without experiencing lost updates due to reading, delay, or writing. However, in the context of ACID—a set of properties for database transactions—atomicity is not about concurrency, which can be confusing. Concurrency is addressed under isolation, represented by the "I" in ACID.

We'll come to that in a minute. So here, in the context of ACID, atomicity is about how faults are handled. If things crash or go wrong, what atomicity provides is that you're going to get either all or none of the writes that occur during a transaction. You can think about this, for example, like this: consider a timeline where time moves from left to right.

Say you've got an auction website, and somebody has just won an auction. Now, you want to do several things with the winner of that auction. You want to update the listing to indicate that the buyer is the user who purchased it. You probably want to send the buyer an invoice and maybe handle some other tasks as well. These actions involve several writes that need to go to different places. What happens if something crashes halfway through making those changes? You could end up in an inconsistent state where some changes have been made while others have not. The whole point of atomicity is that if you wrap these actions in a transaction, you can ensure that any incomplete changes resulting from an aborted transaction will be rolled back and won't be visible.

So fundamentally, this means that transactions are about manipulating multiple objects as one, but doing so as a single unit. If someone claims to have transactions, but what they actually have is just operations on a single object, like a single document in a document database, for example, that doesn't qualify as a true transaction. The whole point is that you can wrap several different mutations into one logical unit. If something fails, you can roll back any partial modifications that you've made. If I were to reinterpret the terminology, rather than calling it atomicity, I would actually refer to this as abortability. This is really the key feature of a transaction: there is always the possibility that it can abort, and there are well-defined semantics regarding what happens when it does. Namely, all of the writes made and all of the side effects are null and void and rolled back. The nice thing about aborting is that it allows you to consolidate a whole range of different failure classes into just one scenario that you need to handle.

So it doesn't matter whether the problem is a deadlock due to concurrent transaction locks, a network issue, a violation of an integrity constraint, an application server crash, a database server crash, or someone yanked out a power cable. It doesn't matter. All of these issues ultimately reduce to a single situation that you have to handle, which is an abort. You could, for example, then retry the transaction. So that's all about atomicity.

So let's move on to isolation, which is kind of the trickiest of all of these, which is why I've left it for last. If you read the database textbook, it will state that when you refer to isolation in the context of serializable isolation, it presents a rather strong definition. Serializable isolation means that the effect of all transactions is as though they had executed serially, one after another, on the database. In other words, each transaction feels as though it has the entire database to itself, even though other transactions might be executing concurrently; however, those concurrent transactions are never visible to the transaction. In practice, while serializable isolation exists, there is also a wide range of different isolation levels. The history of these can be traced directly back to System R, where they first implemented serializable isolation and then realized that it can be quite slow.

So, what do we do? Well, let's fiddle with our lock implementation a bit. Let's change some of the locks so that they're not held for so long. Then, we'll give you a little bit of information about these levels with different lock configurations and the names associated with them. However, the definition of these is rooted in the implementation details of databases from the 1970s.

That's why these concepts are so hard to understand; they are not defined in a way that intuitively makes sense. Instead, they are defined as implementation details of a particular database. So, is there anyone here who thinks they can explain off the top of their head precisely what the difference is between "read committed" and "repeatable read"? Does anyone want to volunteer? If the assembled bright minds at Strange Loop cannot distinguish between these different isolation levels, even though I assume many of you have used a relational database at some point, in theory, you should be able to make an educated choice about which isolation level to use. However, in practice, if you don't understand the differences between these isolation levels, how are you supposed to make that choice? Let me look at this in a bit more detail.

Here's a wonderful list that Peter Baillies and his team at Berkeley created, examining the maximum isolation level that various databases support and what the default is if you don't change it. If you look at it, the maximum level is Serializable in only about half of the databases. This means that half of the databases do not even support Serializable isolation. Furthermore, Serializable is actually the default in only about three out of twenty or so databases. This implies that you have a lot of complexity to consider if you want to develop an application using one of these databases that does not have race conditions. Let me elaborate on the differences among these isolation levels, as I believe it is quite useful to examine examples of race conditions that can occur at these various isolation levels. This will help us unpack what we mean by isolation a bit more thoroughly.

I'll start with Read Committed, which is actually a fairly easy concept to understand, and it mostly means what you think it should mean. Sorry, I skipped over a slide there. Read Committed means that two phenomena, or two anomalies, are not allowed to ever occur. These two things are prevented by the database: dirty reads and dirty dirty reads. The concept of dirty reads is fairly simple.

It simply means that one transaction reads data that another transaction has written but not yet committed. So that's what "read committed" means. You only read the data once it's been committed. That's fairly clear. "Dirty writes" may be not quite as obvious.

Imagine you have an auction website, for example. You have two transactions, A and B, that are writing to the data. These transactions are writing to two different keys in a key-value store, which correspond to two different rows in a relational database, referred to as X and Y. Both transactions concurrently write to these entries, and the outcome is that either X becomes A and Y becomes A, or X becomes B and Y becomes B. These are the two possibilities that could occur in a serial execution.

But in this execution, you actually end up with x first set to a, then set to b, and the final value is b, whereas y is first set to b, then set to a, and the final value is a. So, these two are now inconsistent. However, read committed prevents this. This is already one type of race condition that you don't have to worry about. But we have all of these other factors, so what do they actually mean? There must be some other kinds of race conditions that can occur at read committed, which these higher isolation levels prevent.

Let's have a look at this one. It's called read skew. The example here could be an accounting system or a bank, where you have two accounts, X and Y, each with a starting balance of $500. Now, you want to execute a transaction that transfers $100 from one account to another. The way to do that is to decrement account X by $100 and increment account Y by $100. This works fine, but now consider that you have a concurrently running read-only transaction.

It's only reading the database, not changing anything. It could be a backup process or perhaps an analytics query that needs to look at various keys in the database. At one point, it reads Y and gets 500 back. Later, it reads X and gets 400 back. It's valid to receive 400 back because, at the time it reads X, the transaction that performed the transfer has already been committed.

So it's reading committed data. This is fine, but we've now got this weird inconsistency. Originally, there was $1,000 in total in the system, but to this orange process here, it looks as though there are only $900 in the system. It appears that $100 has somehow vanished into thin air. If this is indeed a backup, then it's an inconsistent backup.

If you restore from this backup, your database will be meaningless because you've seen different parts of the database at different points in time. This can occur under read committed. It's important to note that read committed is the default isolation level for many of the databases mentioned here. To prevent this read skew anomaly, repeatable read and snapshot isolation were introduced. Although these two isolation levels may appear fairly similar from a user's perspective, in terms of implementation, they are quite different.

In many cases, when databases now say "repeatable read," they actually mean "snapshot isolation," just to be confusing. There are basically two ways you can prevent read skew. One is by taking lots of locks, which is what SQL Server does, for example, if you use repeatable read. This is also what System R did back in the day. However, more common nowadays is snapshot isolation, which is implemented as multi-version concurrency control (MVCC for short). This is what PostgreSQL and MySQL do, as well as SQL Server if you use the snapshot mode, and Oracle if you choose the serializable mode.

Note that this is not serializable, but it is referred to as such anyway. The purpose of snapshot isolation is that if you are a transaction reading the database, you see the entire database as it was at a specific point in time. Even if other writes are subsequently committed, that read transaction will not see them, even if it runs for an extended period. This is where the multiversion aspect comes in. The database internally maintains several different versions of a data item and presents each transaction with the version that is appropriate to its time cut.

Okay, but we're still not at the top. There's still a difference here between snapshot isolation and serializable. Can we find another example that lets us distinguish these? What do we need to do? Now, imagine you have, say, an emergency dispatch service, like an ambulance service, for example. The service needs to ensure the invariant that at least one doctor is always on call. Because if no doctor is on call, then no ambulances can be dispatched.

So, there can be multiple doctors on call, and that’s okay. Doctors can trade their shifts; if a doctor is not feeling well, they can ask a colleague to take over for them, and that’s no problem. Now, this is the transaction that will be executed if a doctor wants to go off call. They can say, "I'm not feeling well," and click a button on the web interface that says, "Can someone else take my shift, please?" This transaction will first check how many doctors are currently on call. If this number is greater than or equal to 2, then it is safe for that person to go off call. The database can then be updated to indicate that the doctor is no longer on call.

It's a very simplified example, but it's enough to show a particular anomaly occurring. So imagine you have this database with three doctors: Alice, Bob, and Carol. Alice and Bob are currently on call. Alice says, "Oh, I'm not feeling well. Can I go off call, please?" The transaction says, "Yeah, okay, we have two doctors currently on call."

I'm feeling well. Can I go off call, please? The transaction indicates that we currently have two doctors on call. It's safe for you to go off call. So, she gets updated to off call status.

And then concurrently, Bob does the same thing. He's also not feeling well, but his transactions also see two doctors on call. Now both go through, both meet their condition, and we have a database in which no doctors are on call, which means our invariant has been violated. This problem is called right skew.

And it really does happen. I created a little test case that demonstrates in which databases and under which isolation levels this issue occurs. For example, in Oracle, you simply cannot prevent this unless you implement explicit locks or materialize the conflict in some way that requires careful consideration of your actions. So, just hope that the ambulance service in your town is not built on Oracle, I guess. The general principle of this right skew can occur in many different situations. The fundamental idea is that you have a transaction that reads something and then makes a decision based on what it has just read.

So in this case, the decision is: Are there enough doctors on call? Then it writes its decision to the database. However, different transactions may write their decisions to different places in the database, so they won't conflict. Therefore, a simple lock will not account for the fact that those transactions might conflict. The problem here is that with concurrent execution, by the time a transaction commits, it is based on the premise of the decision that the transaction made.

That premise may no longer be true, and this would then be a serializability violation. Okay, so we've worked our way up to serializable, and we've realized that there are some fairly subtle but important race conditions that can happen at these various isolation levels. So how can we actually implement serializability? Well, for about 30 years, the only real answer was to use lots and lots and lots of locking.

And this technique is called Two-Phase Locking. I'll abbreviate it here as 2PL. The way it generally works is as follows: whenever you read something, you need to take a shared lock on all of the items that you've read. The shared lock does not prevent other transactions from reading it, but it does prevent other transactions from writing to it.

So you can be sure that once you've obtained a lock and you hold that lock until the end of the transaction, nobody can modify the data you have read. The important thing here is that you must hold the lock until the end of the transaction, which can lead to serious performance issues. For example, if you have a transaction that needs to read all the rows in the database, such as a large analytics query that scans everything, it will require a shared lock on the entire database. This means that no one can write to the database until that transaction is completed. Essentially, you are locking the entire database for writes.

And this is why all of these weak isolation levels came into existence: because people said, "There's no way I'm going to lock my entire database for writes. It's just not going to happen." Fortunately, better techniques were developed. In the late 2000s, two competing approaches emerged regarding how we could implement serializability without two-phase locking. I’ll run through each of them briefly. The first example is from HStore, which was an academic prototype that later became VoltDB.

The second example is from Postgres. In the first example, you can simply take all of the transactions and execute them in serial order, one after another. It sounds stupidly simple and obvious, but actually, it's a really good way of ensuring that your transactions are serializable because, by construction, they are. There’s no concurrency going on. You simply take what would normally be these various interleaved threads of execution and turn them into a timeline. Of course, this is only going to work if you make each transaction really, really fast to execute.

And this is why it took 30 years to reach this point. Of course, theoretically, people could have done this for ages, but if you want to actually execute transactions in serial order, each transaction must wait for all of the preceding ones. To achieve this, you probably need to have all of the data in memory because waiting for the disk to respond would be too slow if you're aiming for serial execution. Additionally, you would likely need to take the entire transaction and send it to the database as a stored procedure, allowing the database to execute it quickly as soon as it's received. You wouldn't have an interactive client-server conversation with slow back-and-forth communication over the network, as that would take far too long. However, if you can get your transaction executed within, I don't know, 100 microseconds, then you can achieve reasonable throughput.

And so this is the principle that's used in VaultDB. John Hugg gave a good talk about this yesterday, so if you didn't see that, you can check out the video. Datomic uses a similar transaction commit principle as well, I think. Okay, so that's one way of doing it. The third way, which I find quite interesting, is that you can detect any serialization conflicts and abort transactions.

This technique is called Serializable Snapshot Isolation. It is based on Snapshot Isolation, which we discussed earlier, but it adds additional checks to ensure serializability. This approach has been used in PostgreSQL since version 9.1. In the initial instance, it actually resembles Two-Phase Locking. Specifically, you acquire locks whenever you read or write something; however, the key difference is that these locks do not block. Therefore, if someone wants to write to data that has already been read and there is a read lock in place, the write operation is not blocked and can still proceed.

These locks just record everything. They take a little ledger and write down, "Oh, okay, so this write from this transaction appears to conflict with this read from this other transaction," and they record all of those facts. Then, at the end of a transaction, when it's time to commit, an analysis process occurs. It looks at the various conflicts that happened and decides if it is safe for this transaction to commit. If there's a possibility that this transaction would introduce a serializability violation, that transaction simply aborts and has to retry. So, this is an optimistic way of ensuring serializability.

You're assuming that the conflict rate is not going to be too high because, if there's a lot of contention, then you will experience many transaction aborts and restarts. However, if contention is not too high, then this can work very well. In contrast, two-phase locking is the pessimistic approach. It ensures that you can never get into a situation where transactions yield non-serializable outcomes, but this comes at the cost of preventing a lot of concurrency. Okay, that's our overview of these different serialization levels.

There's still more to say, I think. So, there's kind of a big elephant in the room, I feel, with all of these transaction models, and that is that all of them work just within a single database. Generally, that database is assumed to be in one geographic location. But we're building bigger systems than this now. What if you're doing microservices and you've got several different services, each with their own databases? What if you're doing stream processing, and you have these flows that take in input streams and produce output streams? What do transactions mean in this kind of world?

So we've got here these various kinds of boxes, and there's a cylinder inside each box. The idea, generally what I've heard recommended with microservices, is that you shouldn't try to share a database between several different services because that would break a lot of the goals of introducing microservices in the first place. So, you've got these wrappers of application code around each database. If you're doing stream processing, the picture actually looks very similar. The difference is that, with microservices, these arrows here represent request-response flows like RPC or REST type data flows. In stream processing, there are queues and message brokers, but otherwise, the structure is very similar.

And generally, what people do, and what seems to be the recommended approach, is to draw a box around each of these services and say, "This is a transaction boundary." Transactions only occur within one of these boundaries. If it's using a transactional data store, it can utilize whatever facilities that data store provides. However, there won't be any transactions across different services. There is a very good reason for this recommendation: if you wanted serializable transactions across multiple services, you would have to run some kind of atomic commit protocol. These are distributed systems protocols that ensure that if various parties are all participating in a transaction—different services—either all of them commit or none of them commit.

Here is the text reformatted with correct capitalization and spelling: And protocols such as two-phase commit, three-phase commit, transaction managers, etc., implement this kind of thing. However, as we saw earlier, the whole idea of serializability is that transactions appear as if they were in a total serial order. Now, you can execute literally in serial order or not; this is definitely defined. So whenever two transactions conflict, somebody has to decide which of the two came first. Hey, hey, stop, stop, stop.

From a distributed systems point of view, this effectively makes it an atomic broadcast problem; that is, the challenge of delivering a series of messages to different nodes in exactly the same order. However, the issue with atomic broadcast is that, from both a theoretical and practical perspective, it is actually equivalent to consensus. There are deep connections among all three concepts: atomic commits, total ordering, and consensus. We can certainly achieve consensus, but it can be quite expensive. It requires extensive coordination among various services to ensure that they agree on a common outcome.

And people tend to not like doing things like two-phase commit in services because it makes them feel very brittle and very sensitive to failures. It only takes one service that is running slowly or is temporarily not responding, and suddenly everyone else can't get any work done either. This tends to amplify failures. This coordination makes systems very brittle, whereas the whole point of wanting to introduce several services was to make them more decoupled from each other, allowing for independent failure domains. The situation gets even more complex if they are actually distributed geographically because trying to achieve consensus over a wide area network with very high and highly variable latencies poses significant challenges. Many current consensus protocols do not perform well in this context. That's why people recommend running Zookeeper only in one data center, for example.

Don't try to run it across different data centers. We seem to be stuck in a place where there are no transactions across different services. But what do people do instead? What I've seen recommended in the context of microservices is that you might have compensating transactions, for example. That is, you try to do something, and the service completes it, but then you do something elsewhere, and that fails. Now, you have to go back to the first service and say, "Oh, actually, could you undo that again, please?"

Because this other thing that was also supposed to happen didn't happen, and so now it's not. Now we have to go back to the first service and say, "Oh, actually, could you undo that again, please?" Because this other thing that was also supposed to happen didn't occur. In order to return to a consistent state, you need to do some cleanup afterward. This resembles a transaction abort and rollback, except that you're implementing it at the application level rather than having it managed by the database in the infrastructure.

Look at sagas, for example. There's a paper describing this kind of stuff. What do people do about integrity constraints in this world? So, say you have, I don't know, a constraint like you can only have as many items as you have in the warehouse. What happens if you accidentally accept more purchases than you have items in the warehouse?

Well, you're going to have to apologize to some of the customers and say, "Oh, sorry, actually, we ran out. Here's a discount code for your next purchase, and we've refunded you," or something like that. This is reasonable; businesses do this all the time. You can think of these kinds of apologies as having some integrity constraints, but you don't enforce them upfront. Instead, you check after the fact whether the integrity constraint was violated, and if it was, then you fix it up again.

So, bringing this back to the ACID terminology, it seems to me like these compensating transactions are a bit like the atomicity aspect. We're kind of implementing aborts at the application level. Additionally, processing apologies is similar to the "C" in ACID, as we are implementing integrity and validating invariants, again at the application level. This leads me to think—you’re going to hate me for this—that every sufficiently complex and large deployment of microservices contains an ad hoc, informally specified, bug-ridden, slow implementation of half of transactions.

I have no evidence whatsoever to back this up. I'm just asserting it boldly. So, what about the "I" in ACID? I said that compensating transactions are kind of like implementing atomicity, and apologies are kind of like implementing consistency. What about isolation? Let’s take another example.

Say you've implemented a social network, and, I don't know, one of your users has just broken up with their significant other and is really unhappy. They go to your website and unfriend their ex-girlfriend or boyfriend. After that, they send a message to all of their friends expressing how horrible this person is. The user's intention here is clearly that their ex-boyfriend or girlfriend does not receive that message; they first unfriended them and then posted the message. If I were a user, this would make sense to me in my mental model of the world. However, if you have the friends database handled by one microservice and the posting database managed by another microservice, this ordering information between the two events is lost.

So you could have, for example, a notification service that listens to these events. Due to some transient glitch in the network, the unfriend event may arrive at that notification service later than the post event. By the time the unfriend event arrives, the notification service has already sent notifications to all of the people, including the ex-boyfriend or girlfriend, who was not supposed to receive the notification. The problem here is that there is a causal relationship between these two events, and that causal relationship is implicit. The user hasn't explicitly told us that posting a message to friends depends on the unfriending event.

It's just implicit through their timeline. In this microservices world, we somehow lost that information. Now, this leads us to some interesting areas of speculation and research, which is a similar thing. At the moment, the way we're building microservice-based systems is just pure eventual consistency. Hopefully, it's at least eventually consistent. Maybe it might be perpetually inconsistent; who knows? We can achieve eventual consistency, hopefully. We probably can't achieve serializability because, as I said, that requires you to do two-phase commit, a kind of variant of consensus, and so it just becomes too brittle, too fragile.

So, we want something in between that will still capture causality, which will also reflect the ordering of events and allow people to reason more logically about this. Causality is quite significant, especially when we consider what happens in snapshot isolation. In snapshot isolation, a transaction receives a consistent snapshot at a specific point in time that it can read from. But what does a consistent snapshot actually mean? It means that it adheres to causality. In other words, if you see one item in the database that was written after another item, then that other item must also be present in the database.

That's really what a game-consistent snapshot here means. The interesting thing is that this can all be implemented without requiring global coordination or reaching consensus. There are substantial proofs showing that this represents the upper bounds of consistency that can be achieved without coordination. Whether we can do it efficiently is another question. Furthermore, there are other interesting proofs indicating that you incur a significant amount of metadata overhead if you want to track causality. This is going to be a very interesting and fruitful area of research.

Can we make causality efficient? To summarize, we have a hierarchy of different isolation levels that we inherited from System R in the mid-1970s. Unfortunately, nobody really understands what they mean. However, most people are using something somewhere along that hierarchy. In fact, a lot of people are using "read committed" because that's the default for about half of the databases we observed on that list.

But the interesting thing about read committed is that it can actually be implemented without coordination. Even something slightly stronger, like introducing causality, can still be implemented without global coordination. So maybe there’s some way we can still do transactions across some kind of microservices or stream processing architecture that will obey causality and will adhere to read committed, but which doesn't introduce the whole overhead and problems of coordination. There is a lot of research going on in this area. I have a completely oversized reference list for anyone who wants to read papers. This is page one of the reference list.

This is page two of the references list, along with pages three and four. To help you find them, all of the slides are available online. I've tweeted the link to the slides already. It's a big topic, and I've mostly described problems rather than solutions because many of the solutions are still unknown. However, people are actively working on them. If you have any questions or would like to discuss this further, please feel free to contact me.

As a final note, I wanted to mention that I'm actually going to be giving away free copies of this book. O'Reilly has kindly printed off 25 copies, even though it's not finished yet, but they include the chapters that are available so far. I will be giving those away during the lunch break at 12:30 down by the entrance. Additionally, there is a discount code available if you would like to purchase the e-book. Unfortunately, I think I'm out of time.

So, if you want to ask me questions, I'll be around here. Thank you very much for coming.