Introduction

Moshi 1.6 was recently released with a brand new Kotlin code gen artifact. It’s an alternative to the existing reflection-based artifact, and in my opinion one of the most interesting recent evolutions in Kotlin’s open source ecosystem.

There are a lot of annotation processors today that have some degree of support for Kotlin in the sense that they see Kotlin as it is in bytecode: an extremely… meticulous form of Java. Some work mostly without a hitch (such as Butter Knife), but some have quirks you’ll probably step into. Dagger is a classic example due to how users have to manipulate companion objects and @JvmSuppressWildcard.Some even have partial understanding of Kotlin language constructs (such as Room and property names). All of them usually come with some degree of caveats/limitations, usually still generate Java code (imposing slow mixed sources compilation times), don’t understand a lot of/any language features, and certainly can’t lean on those features in most cases due to the lack of understanding or interoperability from Java. In a sense, you’re writing Kotlin that can feel restrained by Java anyway, coercing it into something that makes the compiler happy but maybe sacrificing code design and build speeds along the way.

Overview

Moshi’s new code gen artifact is one of the first (if not the first?) annotation processor to fully support Kotlin code. Not just being able to read Kotlin code, but to natively understand its language features, generate Kotlin code (using KotlinPoet), and be able to lean on those understood language features in the generated code!

A rough overview of language features that are understood and supported:

• Mutability (val vs. var)
• Collection mutability (e.g. List vs. MutableList)
• Nullability (?, including on generic type parameters!)
• Property requiredness
• Property names (no having to decipher bean-style names)
• Property annotations
• Default values
• Variance/projection (e.g. in, out)
• Typealiases
• Companion objects
• Primary constructors

The result is that generated adapters can work wonderfully with language features used in the classes they are generated again. Not only that, but you get all the speed and safety that comes with using a streaming JSON API and none of the overhead or runtime impact that comes with including the kotlin-reflect library.

Show me the code! 👩‍💻👨‍💻

This is a simple example for just a single, simple bar property:

The generated adapter reads and writes via Moshi’s streaming API and throws an exception if bar isn’t present during reading, recognizing that it’s a required property. Sure, some language features could be used to prettify it, but this is code gen and focused on correctness, not being pretty.

Below is a more robust example with parameterized types, @Json naming annotations, default values, generics, nullability, and Kotlin collections.

There’s a lot more going on here, but what we’re seeing is a generated adapter that clearly understands the source type it’s generated for and even able to lean on Kotlin language features for its implementation. This is really powerful, as you now have all the runtime benefits of writing out your JsonAdapters without any of the work!

How does it work?

The Kotlin code gen is heavily influenced by auto-value-moshi/gson. The design principle is simple: annotate your Kotlin class (can be a data class or just a regular class) with @JsonClass(generateAdapter = true).

@JsonClass(generateAdapter = true)
data class Foo(val a: String)

… and that’s it! Under the hood there’s a lot going on to make this work. At the really nitty gritty level, this works by reading Kotlin @Metadata annotations to understand language features, but this post is going to focus on the design of how Moshi generates adapters. If you want to learn a little more about how Metadata annotations work, stay tuned for another post or check out Eugenio Marletti’s talk about them.

Basic language features

Kotlin, in its most raw form, is a language that allows you to succinctly express at the language level what has always required extra keywords or annotations (plus static analysis tools) to work in Java. Specifically, mutability and nullability.

In the generated adapters, mutability is respected at the collection level. Kotlin collections differentiate between mutable and immutable variants (List vs MutableList, etc). Generated JsonAdapters make the same distinction in the types used to construct.

Similarly for nullability, generated JsonAdapters understand the nullability of both the properties themselves as well as generic types. If a property is nullable, it’s read with a nullSafe() adapter. If it’s not nullable, it’s read with a nonNull() adapter. nonNull() is a new API on JsonAdapter in Moshi 1.6 to return a new adapter that fails hard on explicit nullable values (but not absent values). If a nonnull property with no default value has no value read in JSON, it will throw a JsonDataException.

Something that didn’t make the cut for 1.6, but might be an exploration in the future is recognizing mutable var properties. Currently, default values (covered in the next section) are implemented with a copy(), but if a property is mutable we could bypass this by knowing we can set the value directly.

Default values

One of Kotlin’s best features is the ability to define default values for parameters. This allows for consumers to only supply a subset of the parameters required, and also allows generated JsonAdapters to defer to them in the absence of that property in the deserialized JSON.

This is tricky to get right, so what we’ve opted to do is something clever using the copy() function.

• First, we instantiate an instance supplying only the properties that have no default values.
• Then, we call copy() on the returned instance, overriding values that had defaults only if we read a value out of the JSON.

The result is that you do technically have one extra allocation, but it’s a small overhead for significant gain. The only other solution to this would be to generate code that covers every possibly permutation of different defaulting properties (which would be crazy!).

Note that the adapters differentiate between null and absent in these cases. If an adapter saw a key but it was null, that nullwill override the default value of the property it backs. The default value is deferred to only if that key was absent entirely from the JSON. In the example above, typeParamSet is an extra flag generated to check if the typeParam was set, even if it was to null.

Shared adapters

The adapters are generated, but it doesn’t mean we don’t want to avoid making them smaller if we can. In a naive implementation, we’d get or create one delegate JsonAdapter for each property, even if some can be shared. The processor tries to be intelligent about these though, and will reuse shared adapter properties where possible. What this means is that if you have two String properties, they’ll use the same JsonAdapter<String>.

This can also bridge across Kotlin language features, so if you have two properties of type String and a typealias that also aliases to String, only one single JsonAdapter<String> property will be generated and reused for both.

The heuristics the processors uses are: Type, nullability, backing type (in the case of a typealias), and @JsonQualifier annotations.

@JsonQualifier annotations

@JsonQualifier annotations are one of the most powerful features of Moshi, and it took a lot of exploration to get these working in a sane way in generated adapters. Under the hood, qualifier adapters on properties are copied directly onto the delegate adapter property in the generated JsonAdapter. The property then reflectively reads these annotations off itself (meta, I know) to read their data. We do this because annotations in Kotlin are final, and cannot be extended or implemented like they can in Java. We also can’t rely on proxies for cases where they have readable method values. If we had a String property annotated with an @UnEscape qualifier annotation, the resulting adapter would look like this.

One thing to note here is that they have a site target of field:. This is actually an optimization we chose to do after inspecting the Java bytecode of properties with annotations applied without this target. Without this site target, the Kotlin compiler synthesizes a helper method like this:

Kotlin does this so it can access the annotations at runtime, but is unnecessary in our case and avoidable by specifying the field target. Avoiding it saves us a method, but it also means that any @JsonQualifier annotations you use must have at least FIELD target support. This includes both Java and Kotlin annotations.

selectName() API

Moshi uses Okio under the hood, which has a select() API for more performant reading of data. Moshi exposes support for this via the selectName() API, which generated adapters use to squeeze extra performance out of deserialization. This is what the JsonOptions property is in generated adapters, as seen in the linked Gist examples above.

@Generated

There is an optional processor option to enable generation of @Generated annotations onto the generated adapters. This is disabled by default, but can be enabled by specifying the moshi.generated option with the value of either javax.annotation.Generated (for JRE < 9) or javax.annotation.processing.Generated (for JRE 9+).

Companion objects

If you define a companion object on your model class, the processor will generate an extension jsonAdapter(moshi: Moshi) function onto it for easy retrieval. For example, the following code:

@JsonClass(generateAdapter = true)
data class Foo(val bar: String = "bar") {
  companion object
}

Would result in a generated adapter and an extension function on the companion object:

fun Foo.Companion.jsonAdapter(moshi: Moshi): JsonAdapter<Foo> = FooJsonAdapter(moshi)// Generated FooJsonAdapter...

This allows you to do simple direct JsonAdapter lookups per class via Foo.jsonAdapter(moshi). This works with generic classes too, where the extension function will just have an extra Array<Type> parameter to define the type variables.

Kotlin in, Kotlin out

Reiterating this point: the processor generates Kotlin code! Language features aside, the simple fact of avoiding generating Java code and creating mixed source sets can substantially improve your build times by not having to do multiple passes to link the languages together.

Caveat: Dynamic default values

Default values support works in most cases, but there are some cases that are not feasible. Specifically, dynamic default values are finicky to deal with. Consider the following case:

@JsonClass(generateAdapter = true)
data class Foo(val bar: String = "burrito", val baz: String = bar)

Since baz is conditionally initialized to the value of bar, this is a case we can’t really statically reason about in the generated adapter. Remember that adapters will initialize an instance once without default values before calling copy() with recorded values from the JSON. Consider the following JSON snippet:

{ bar: "taco" }

If we deserialize this JSON, we would want a Foo with bar value of "taco" and baz default value of also "taco". Instead though, we would end up with a baz value of "burrito"! This is because the initial instantiation with default values will leave us with that "burrito" value dynamically assigned to baz based on the initial default value of bar.

Unfortunately, there’s nothing we can do about this case in adapters other than advise caution. We can’t detect these cases at compile-time due to property values not being available in metadata, so our best hope is that in the future Kotlin allows for specifying certain named parameters as “undefined” during invocation.

Misc

• Proguard: The generated adapters (assuming you use the recommended proguard configuration) are completely proguard friendly with a small runtime reflection lookup. If you use @JsonQualifier annotations, you’ll need to keep the names of anything they annotate as well.
• Generated factory: You could also write your own processor to read the @JsonClass annotations and generate your own JsonAdapter.Factory if you wanted to as well (similar to the factory processors of auto-value-moshi or Kotshi) to completely remove the runtime reflection cost and need for proguard rules at all. If you leverage the companion object extension functions described above, you can generate very clean factories that statically link all the generated adapters.
• Source sets: If you use kapt in Gradle and want to reference the generated code (i.e. to call jsonAdapter() extension functions), you’ll need to include the build/generated/source/kaptKotlin/... directories in your sources. Here’s an example of how to define this.

Closing

I’m really excited to see to see released and looking forward to hear feedback from the community about how it works for them. This project has a neat backstory too. Notice how I kept saying “we” above? It actually started as a prototype in my personal CatchUp project before eventually making its way to Moshi as a collaboration. The full story is for another post another time though :).

Thanks to Mike, Florina, and Eugenio for proofreading this.

This was originally posted on my Medium account, but I've since migrated to this personal blog post.