Advanced JSON Handling in Go

Hi, I'm Jonathan Hall. In this video, I'm going to talk about some advanced techniques for JSON handling in Go. This is material I originally prepared for a meetup where I presented a few months ago, but the recording from that day wasn't very good quality, so I decided to re-record it here. I also go into greater detail on these topics and related subjects on my blog and in a book I'm writing called *Data Serialization in Go*. The book is available for pre-purchase on LeanPub. You can see a free sample, and if you're interested, you can, of course, buy the book. As new chapters are added, you'll receive the additional information at no extra fee.

Let's get started. In this video, I will talk very briefly about what JSON is. I will cover the basics of JSON and discuss the basic usage of maps and structs in Go when dealing with JSON. I will also talk about handling inputs of unknown types and provide some examples of that in a moment.

And handling data with unknown fields. These are based on some questions I see frequently on Stack Overflow and other places, so I hope you'll find at least some of this to be valuable. Of course, I expect most are familiar with what JSON is. It's the JavaScript Object Notation as defined in RFC 8259. Basically, it's a human-readable textual representation of arbitrary data.

Compared to Go's type system, it's very limited, of course. It has just five basic types, plus null. However, it is used for many things, from configuration files to REST APIs. The uses are endless. There are many alternatives to JSON, and most, if not all, of the techniques I will describe in this video also apply to others like YAML, TOML, MessagePack, etc. Possibly even XML and Protobuf.

But JSON I like to use as a baseline because it's so common, and practically everybody deals with it at some point. So just to get started, when you create JSON from a Go object, in this example, a map of string to string, it's pretty straightforward. You can more or less pass any arbitrary data type from Go into the `json.Marshal` function, which will then produce valid JSON. Now, one note: throughout this talk, I often skip error checking, which is not a good idea—obviously, in production, never do that unless you have a really good reason. But for this video, to make the demonstrations a little bit shorter, I often skip error checking. Just be aware of that. So this simple example program takes a Go map of string to string, with the key of "foo" and the value of "bar," and converts it to JSON, which is pretty expected: foo to bar. Of course, we often don't use just maps in Go.

We use structs, and that's where JSON usually gets a little bit more interesting and easier to work with as well. So, here's a really simple example. I have a Person struct that goes to Go. JSON usually gets a little bit more interesting and easier to work with. So, here's a really simple example.

I have a `Person` struct that contains a name, age, and description fields. By using the JSON tags, I can specify the JSON names of the respective fields so they don't have to match exactly. Additionally, I have a fourth field here called `secret`, which is unexported, as you can see because it does not begin with a capital letter. This is a common mistake that I see happening a lot. I probably encounter anywhere from two to three, maybe four questions about this on Stack Overflow each week from people who are new to JSON handling in Go. This aspect confuses many people, so let me explain it here.

When you have an unexported field in a struct, no package outside of the package that defines it can access that struct. This includes the JSON package in the standard library, which is the reason that unexported fields are excluded when either marshaling or unmarshaling JSON in Go. While this is often seen as a limitation or a hassle, I actually see it as a benefit that provides us with flexibility. We can define structs with unexported fields that do not get marshaled into JSON when that is our intention. In some cases, it is also useful during unmarshaling JSON, as we can exclude certain data.

We'll talk about both of these in a little bit. Mainly, I just want you to be aware of this and consider it as you go forward. So now that we have our struct defined, here's just an example of it. We define a person named Bob, who is 32 years old and has a secret. If we marshal this, we will see that we get the expected JSON output: the name is Bob and the age is 32.

And the interesting part here, of course, is that the secret is excluded. Now, unmarshalling can often be a little bit trickier, and that's just because it takes a little more forethought to map arbitrary JSON data to a Go struct. Here’s a really simple example: we have JSON, the same JSON we saw a moment ago, being converted back into Go, and we get a map of `string` to `interface{}`. Now, you often should avoid using just a map, specifically a map of `string` to `string` or `string` to `interface{}` when unmarshalling JSON. The reason is that those types are just cumbersome to work with.

Of course, they're necessary sometimes, but whenever it's possible and when it makes sense, you should use a struct instead— a struct with defined data types and a defined structure. So here's an example of that. Using the same Person struct we had a moment ago, this time I'm converting JSON into a Go struct. Here we have our sample JSON input, which is, as before, name: "Bob", age: 32, and it contains a secret. Now when we execute this code, as we might expect, we get a Go struct with name "Bob", age 32, and, as described a moment ago, the secret is blank, even though it's included in the JSON because it's unexported.

The unmarshaller cannot access that, and it's left alone. So, structs are nice, and maps are useful, but what happens when you don't even know the data type before you unmarshal it? Let's talk about some examples of that. One common example that I've dealt with many times is when you receive either a number or a string representing a number. Some APIs are very inconsistent; they may return one or the other depending on various factors, such as the phase of the moon, changes between software versions, or the specific options you send to the API. Another common pattern I see is an API that returns either an object or an array of objects. Instead of returning an array with a single object when there's only one, it just returns that object alone.

This is something I've seen fairly commonly in more dynamically typed languages, like JavaScript, Perl, or Python, where that's simple. However, it's a little more complicated in Go, but I'll talk about how to work around that. The other case I'll discuss is when you get completely different object types. For example, an API that returns one type of object for success and a different type of object for failure. I will go into more detail about this, including the concept of literal numbers or literal string options. This is an example of something I actually dealt with not long ago.

We had an API that was returning a literal number, and then, I guess through a change and upgrade in the service, it started returning a string instead—just a quoted number—and that broke our API. So how can you deal with both without having to change your code all the time? In this example, I've created a custom type called `IntOrString`. Of course, please use more descriptive names in your own code, but for this example, it's fine. Then, on that type, I've defined a custom unmarshaller, which you do by defining a function called `UnmarshalJSON`.

And then, inside that function, I simply unmarshal to a number, to an integer called v. I then assign that integer to the receiver, which is called i. Now, the magical part here is that I remove any leading or trailing quote marks before doing that in the unmarshal. In a simple example like this, that's sufficient. Supposing I receive a string, the quoted string "quote one, two, three," for example. By removing those quote marks, I'm left with just "one, two, three," and I can unmarshal it as an integer normally.

And if I get just an integer that's just one, two, three without quotation marks, then this trim does nothing, and I still unmarshal one, two, three. Now, suppose I received a string that was not a number. Suppose somebody sent a quote, "hello world," end quote. Then I would remove the quotes and try to unmarshal it to this integer. That would produce an error, and I would return that error.

So let's watch this in action. I've created a JSON input here that contains an array of two elements. The first element is a literal number, and the second is a quoted number. Let's see what happens.

So now our result is a slice of strings. The first is a value of one, two, three. The second is a value of three, two, one. Now let's talk about the next example, which can either be a single value or an array of values.

So, this is common in some APIs. Maybe you're querying a database or something similar. If you get a single result, you receive one object.

If you get multiple results, you receive an array of objects. In this example, I'm going to use strings rather than objects, but the principles are the same. Of course, you can update the code to use objects if that's what you need. In my example, I have a custom type called `Slice` of `string`. The underlying value is of type `Slice` of `string`, along with my custom `UnmarshalJSON` function. The first thing this function does is check whether the first character of my input data is a quote mark. If it is, I know I'm dealing with a string, and I can act accordingly.

So, if I have a string, I first unmarshal it to a temporary variable called `v`, which is of type string. Then, I set my receiver to a slice that contains a single value of that string, and I return any error I receive. If the first character is not a string, then I presumably have an array. In this case, I create a temporary slice variable, unmarshal into that, and then set the receiver to that value. Once again, if I receive an error, I return that error. If I were to receive an object, a number, or something other than a string or an array, then this would behave accordingly. So, let's look at an example.

My input to test is an array of two elements. The first element is just a string, and the second is an array of two strings. As expected, my Go result is a slice of string types: a single element for the first input and two elements for the second input. What happens when you don't know the data type?

So, let's use an example of an API that returns one type of object for success and another type for failure. For the sake of argument, we'll assume you cannot determine which is which based on, say, the HTTP response code. You only know by reading the JSON result. What I've done here is create a success type and an error type. My goal, of course, is to unmarshal success objects into success types and error objects into error types. With these two types defined, I then create a third type that embeds both of these, so this response is sort of a generic wrapper container, if you will—a wrapper type that includes both success and error. In this case, I don't actually need to create a custom unmarshal function.

And the reason for that is that the success and error structs have no common fields. So there's no ambiguity when the JSON unmarshaller tries to unmarshal data into the embedded structs. Now, let's look at an example. Here, our input is an array of two values. The first is a success object.

The second is a failure object. We can see here that our result for the success case contains a success object with populated values and then an empty error object. You can see that both the error and reason are empty strings. In the error case, we have an empty response, and the error is populated. But what happens when we have conflicting or overlapping fields?

Let's look at an example. In this example, our API returns a status key for both success and failure. Therefore, we update our success and error types to include a status. You can see that this is duplicated between success and error. However, it is not sufficient to simply embed them because the JSON unmarshaller won't know where the status field belongs.

We now need to create our custom unmarshal function. The first thing this function does is attempt to unmarshal into success. If it returns an error, we return that error as well. However, it is not enough to simply check for an error; we must also verify the type of the error.

We also need to check for an error. Check the value of the status because if we encounter an error value, it will still successfully unmarshal; it just won't have the error field. The error field reason will be blank. After we've unmarshaled to success, we check the status value. If it's as expected for success, then we're done and return nil. If we get any other value, we assume it's an error. In that case, we set the response.success value to its zero value and then unmarshal to error. Let's look at our demonstration. Once again, we have a success case and a failure case, both of which have different status values.

And as before, you can see that in the success case, we have a success object. The status is okay, and the other values are set. The error is empty. For the error case, we have an empty success value, and the error field is set with its own status. However, the response type I've been using here may be a little bit cumbersome.

You always have to check whether it's the success or the error value within that response that is populated. Maybe you don't want to have two objects; you want to have just a single one and then deal with that one. So, let's discuss a way to use a different container type to solve that problem. However, here I'm still using the same success and error structs as before.

I'm only changing the response. Rather than having both success and error embedded, I've just created this result field that's an empty interface. Now, let's look at the custom unmarshal function. Here, I create a value of type success again, the one I defined a moment ago. I unmarshal to that same value as before, and also the same as before, I check to see if the status is okay. If it is, then I proceed if the status is okay.

If it is, then I set the result to success and return nil. If our status is anything other than okay, we continue. We define a value called fail, which is of type error. We unmarshal to that, set the result to fail, and then return nil. Let's look at an example. We have the same input as before, but now our output has only one result value for each input, rather than one populated and one empty.

So, for success, you can see that we have the success case, and for error, we have the error case. Let's go a little bit further with container options. In this example, I'm defining my unmarshal JSON function on a slice of the empty interface rather than on each individual object. The way this works is that I start by unmarshalling the JSON into a slice of JSON raw message. As we can see from the Go documentation for the JSON package, JSON raw message is just a byte slice, but it has custom marshal and unmarshal methods that simply pass the data through unaltered.

And this is useful, as you'll see in just a moment, where all I care about right now is chopping my JSON into individual elements rather than parsing the entire thing, so that I can process each element one at a time. So, let's do that here. I unmarshal into my slice of JSON raw messages. Then, I define a success and a fail value, similar to before. After that, I pre-allocate a result slice of the same length as my input, and then I loop through the input.

For each value in my raw input slice, I unmarshal it to the success struct. If the status value is "okay," then I assign that success value to the result and continue. If it's anything other than "okay," then I unmarshal it to a failure struct and set that value to the result. At the end, I set the result to the receiver and return nil. So you can see it's essentially the same logic as before, but in a loop this time. Let's look at the example.

Our output is pretty much exactly what we would expect, almost identical to before. For the success case, we have one success object. For the failure case, we have one error object. Of course, it can get much more confusing. Let's suppose that rather than just success or failure, we have three, or maybe four, or even a thousand different object types. For my example, I have three.

I have a person, an animal, or an address. Other than the type field, everything else about the object is arbitrary or completely different. So, I’ve defined my three core types: person, animal, and address. Then, I create my container, my responses, which is a slice of empty interfaces.

My unmarshal JSON method operates on the container type, `responses`. As before, I create a temporary slice of `json.RawMessage` and unmarshal to that. Now, I have created a variable called `header`, which is of an anonymous struct type that contains a single field called `type`. As before, I define my result slice and pre-allocate it to the same length as my input. Now, my loop, of course, is a little more complicated.

For each case in my input slice of JSON raw messages, I first unmarshal to this header. Then, I check the value of that header, specifically the type field. If the type is "person," I unmarshal to a person. If it’s "animal," I unmarshal to an animal. For this address, I unmarshal to an address. In each case, I assign that value to the corresponding index in the result. Finally, as before, I assign the result to the receiver and return nil.

Let's look at an example. Our input here contains three different objects of three different types, and the output result is a slice of empty interfaces. The first element is of type Person, the second is an Animal, and the third is an Address, as you would expect. While this last example works fairly well and is perfectly adequate for small bits of JSON, it's not particularly efficient. The reason for that is that it reads the entire JSON array into memory when it parses it to the slice of `json.RawMessage`. Then, it goes through each of those elements individually and parses them to the final result.

And then it goes through each one of those elements individually and parses them to the final result. However, this process creates a completely extraneous copy of the entire array in memory, which we don't need. It's especially wasteful if an error occurs early in processing. In such cases, you've read the entire array into memory, encountered an error, and then discarded the whole thing. Even in successful cases, it would be much more efficient if we could read each element individually, rather than consuming all that memory.

Let's look at a way to accomplish that. For this example, I'm using the exact same configuration as before. I still have the person, animal, and address types, and I still have a responses type that is a slice of the empty interface. The only thing I'm changing here is my implementation of unmarshalling JSON. The functionality should be the same; it's just hopefully more efficient now.

My new implementation of unmarshalling JSON takes advantage of the JSON decoder, which accepts an `io.Reader` as input. The decoder allows you to utilize a tokenizer interface to interact with it. I define my decoder here, and it reads the input using `bytes.NewReader`. The first thing I want to do is consume the first token, which I am assuming for this example is correctly an opening square bracket. Of course, to be more robust, I would actually validate that this is what it is and return an error if it is not.

I trust that you can add that to your own implementation. Then I create my result output. Now, in this case, I don't know the size of my output, so I can't pre-allocate it. So I just set it to zero length. Once again, I create my variable called "header," which is of this anonymous struct type that contains just the type field, the same as before. Now, my loop is slightly different.

Now, I still use this raw JSON message, but I just have a single instance, as you can see, rather than an entire slice. That's because I need to unmarshal the value twice still. So, I start by decoding the next symbol from the decoder, the tokenizer, into this raw value. Then, I decode that raw value into the header. Once I've done that, I have my `header.type` that I can use to see where to unmarshal the rest. If the type is "person," I unmarshal it to "person." If it's "animal," I unmarshal it to "animal." If it's "address," I unmarshal it to "address." Essentially, it's the same as before. Now, this time, rather than assigning to a particular index of the result, I just append since I don't know how many results I have. At the end, as before, I assign the result to the receiver and return a nil error.

Now, our example should be identical. We have the same inputs, we expect the same outputs, and indeed we see the same outputs. It's just a more efficient implementation. Now, let's talk about something that's even more interesting, at least for me. Suppose you have a data structure that has some known fields that would map well to a struct and some unknown fields that would require a map, say from string to string or from string to interface. However, you don't want to do everything as a map. Let's discuss a way to do a hybrid approach. So, here’s our example input data. In this case, we have two fields that are the same across all objects: the underscore id and the type. Everything else is arbitrary, and we don't have any way to know what it is.

It might be that these fields vary based on the type, or they might be completely arbitrary. Therefore, we would like to access the ID and type through struct fields rather than using a map. Everything else must be in a map because we don't know what it is. So here's our target type: I've created a type called Item.

It represents a generic entity that has an ID, a type, and arbitrary data. Let's look at the custom `unmarshal` function I've created. The first thing I do in this function is define a variable called `x`, which is of struct type. This anonymous struct embeds the `Item` type that I defined on the previous page and also adds a field called `unmarshaledJSON`. The only reason this field exists is to overshadow the `UnmarshaledJSON` method on the `Item` type.

If I don't do that, then on this next line, when I call `json.Unmarshal` on `x`, it will find that `item` has the `UnmarshalJSON` method, which is actually defined right here. This will create an infinite loop and cause the program to crash. So, with that in mind, I do call `json.Unmarshal` on the raw data into `x`. Then, down here, I unmarshal the data a second time, this time into a map of string to interface. Next, I delete the two fields that were previously handled up here, the ones defined in `item` for `id` and `type`. I assign the item here to my receiver, and then I set my data—my arbitrary data.

So let's look at an example. Here are our three example input objects from the beginning. When we execute the code, we see that we get a Go slice of type Item. The first one has an ID of "bob," a type of "user," and arbitrary data with the name. The second one has an ID of "meetup," a type of "website," and arbitrary data with a URL. Again, the third one has an ID of "soup," a type of "recipe," and arbitrary data. Of course, doing this is most useful if you can reverse the process. So let's talk about that. Now, instead of creating an unmarshal JSON method, we're creating a marshal JSON method that works on the same data type, the same Item struct.

The first thing I do is marshal the data. So this is the arbitrary map data assigned to a value called "data." Again, I use an anonymous struct type for marshaling. I embed "item" inside of it, and, as before, I create this empty struct field called "marshalJSON" to avoid the infinite loop. In addition, I added this JSON tag. If I don't do this, then the output will contain something called "marshalJSON" with no fields, which I don't want.

Then I marshal this temporary value into raw data called `obj`. At this point, both `data` and `obj` should be completely valid JSON objects, each containing a subset of the data we're concerned with. `data` contains the arbitrary data values, and `obj` contains the `id` and `type` fields. To combine those, I replace the final character of `obj`, which should be a closing curly brace, with a comma, and then I append all except the first character of `data` to that, returning the result. So, here we have our input data. This time it’s JSON objects, but they essentially contain the same data we had before: Bob, meetup, and soup. Let's see what the JSON output looks like. In the first case, we have an object with three fields: `id`, `type`, and `name`, as expected.

The second one is a type URL, and the third one is a type with ingredients. Thank you for watching! I hope you find this information valuable. If you have any questions, please reach out to me either in the comments or on my blog. I love to hear from you, and don't forget to like and subscribe if you found this valuable.