A Closer Look At Moshi 1.9

Moshi 1.9 is here! It's been in the oven for a year and excited to have it out there. Most of this is covered in the changelog, but I wanted to dive into a few larger changes under the hood in code gen and some incubating APIs.

Incremental Annotation Processing

Build times rejoice! Kapt added support for incremental annotation processing in March as one of the many fantastic contributions from Ivan Gavrilovic over the past few months. Unfortunately though, supporting incremental annotation processing also involved some external blockers.

• KotlinPoet support for Filer and originating elements
• Initially, Kapt did not support Filer-generated Kotlin source files yet. This meant that the incremental support effectively only really supported processors that generated Java code. Partially fixed in Kotlin 1.3.40.
• Second fix was needed for mixed source set projects, not released until Kotlin 1.3.50.

Documenting all this can only go so far. Prior to Kotlin 1.3.40, this wouldn't have worked at all and only partially worked until 1.3.50. Documentation is especially limited as a tool in annotation processing since users aren't actively looking at the processor source code itself. It would have been unclear which tool was at fault. If you get an obscure gradle compilation error not tied to a specific source, who's to blame? Moshi? Kapt? Gradle? Android gradle plugin? 🤷‍♂️. This would've caused unnecessary churn and noise for other tools that had nothing to with the issue and the issue was already fixed on master in Kapt and just waiting for the next release! With all this, we couldn't in good conscience release with that many caveats.

Java reflection will now reject Kotlin classes

The built-in Java reflection (ClassJsonAdapter.FACTORY) has rejected Kotlin platform types (i.e. classes in the kotlin.* package) for a while now. In 1.9, it will reject all Kotlin classes, not just ones in the kotlin.* package.

This is for a good reason. Before - if a Kotlin class was unhandled by any other JsonAdapter.Factory, it would fall through to ClassJsonAdapter. Sometimes this worked fine, but many times it didn't and would fail in esoteric ways. ClassJsonAdapter's factory would best effort this, but at the end of the day it was not built to handle Kotlin classes and doesn't understand their behavior. That's what KotlinJsonAdapterFactory or Kotlin code gen is for. Now, for Kotlin classes, you either need to use code gen, KotlinJsonAdapterFactory, or supply your own custom JsonAdapter.

This is a potentially dangerous change! In projects using code gen, there are cases where Kotlin classes could have (appeared to) Just Work™️ before if you forgot to annotate them with @JsonClass. These will fail at runtime now. If you're worried about this, I suggest using Moshi 1.9 only in debug builds for a period of time to tease these out before releasing production builds with it.

Dynamic constructor invocation from code gen

In Moshi 1.9, generated adapters will now invoke constructors via the same synthetic constructor that Kotlin uses for constructors with default parameter values. How these constructors work merits a dedicated post in the future, but in short they accept some extra parameters to indicate which of the "real" parameters are actually present. Previously, we had to use an awkward combination of the declared constructor + copy() to set the default properties. Aside from avoiding the awkward double instantiation, this also allows for construction with dynamic parameter values.

data class Fox(val color: String, val winterColor: String = color)

This makes code gen functionally at parity with the reflective KotlinJsonAdapter in this regard, albeit it does require a minimal bit of (cached) reflection to look up the synthetic constructor. We plan to remove the need for reflection by generating raw bytecode directly in the future, but can't do this just yet due to a bug in kapt. Generated code will only use this reflective mechanism IFF the target model's constructor has any parameters with default values, otherwise it just invokes the regular constructor directly.

KotlinPoet-Metadata backend

Moshi's initial code gen implementation used kotlin-metadata, but it had its downsides that were starting to show. Namely - it was a shade of the kotlin compiler metadata protos and not officially supported. This would manifest itself as compatibility/versioning warnings and lacked support for newer Kotlin language features like inline classes or unsigned types. JetBrains later released an official library - kotlinx-metadata. We built KotlinPoet API over this that can create spec representations from metadata and a given classpath (reflection or javax elements). Moshi is now powered by this, giving us longer term reliability and as well as an improved and simplified API.

If you want to learn more about how metadata works, see my KotlinConf talk.

Code gen flexibility

Adapter name lookup API is now public.

Using Types#generatedAdapterName, you can write or generate your own adapter from a given name and Moshi will look it up from @JsonClass-annotated types. This is useful if you want to reuse this automatic mechanism for your own purposes or are generating your own adapters (more on this later).

If using this API from an annotation processor, we have some ideas for improving this but right now it can be a bit awkward. The simplest approach is to use a KotlinPoet ClassName representation of the target type and then read the target generated class type back.

val targetClassName: ClassName = // The target class

val generatedAdapterFqcn: String = Types.generatedJsonAdapterName(targetClassName.reflectionName())

val generatedClassName = ClassName.bestGuess(generatedAdapterFqcn)

Custom generators

@JsonClass now has a new property: generator. By default it's an empty string, but you could use a different value there if you want to write your own generator that reads it. Moshi's own generator will skip over any classes with a custom generator string.

@JsonClass(generateAdapter = true, generator = "sealed:type")
sealed class Message {
  @JsonClass(generateAdapter = true)
  data class Success(val data: String): Message()
  
  @JsonClass(generateAdapter = true)
  data class Failure(val errors: List<String>): Message()
}

Code gen API

This isn't public API yet, but it's in source if you're interested in taking a peek. Currently code gen is solely coupled to annotation processing. This is an experiment to decouple it, which could allow for custom generators in different environments to reuse the standard code generation tools from sources other than annotation processing. Stay tuned for updates here.

More consistent Kotlin behavior across reflection and code gen

We want the reflection and code gen artifacts to have the same behavior at runtime. To this end, we've done some work to consolidate tests and APIs to improve consistency. With the addition of dynamic invocation in code gen and the multiple transient properties fix in reflection, we should have functional parity at this point. Please let us know if you see any issues.

Aside from consistency, another benefit that this helps achieve is that you could use moshi-kotlin just in debug builds (for faster builds, since the code gen isn't running) and code gen for just release and/or CI builds. This is an explicit case we want to support, and have tests to ensure that classes annotated @JsonClass(generateAdapter = true) will gracefully fall back to the reflective KotlinJsonAdapter if present.

New valueSink() API

There is a new Okio-based JsonWriter#valueSink API. This is a simple, yet powerful API that affords a lot of flexibility for custom value types and allows you to stream values through to the writer's underlying Okio BufferedSink. This is great if you want fine-grained IO access at the byte level such as writing encoded blobs (images, compressed data, etc). Consider the below example of serializing an Android Bitmap image to Base64 encoding in JSON:

class Taco(val thumbnail: Bitmap)

// In your Taco adapter
override fun toJson(writer: JsonWriter, value: Taco) {
  writer.name("thumbnail")
  
  writer.valueSink().use { valueSink ->
    valueSink.writeByte('"'.toInt())
    val base64Sink = Base64EncodingSink(valueSink).buffer()
    value.image.compress(
        Bitmap.CompressFormat.PNG,
        0,
        base64Sink.outputStream()
    )
    // Don't close base64Sink, otherwise it'll close valueSink as well
    base64Sink.emit()

    valueSink.writeByte('"'.toInt())
  }
}

We plan to add an analogous JsonReader#valueSource API in a future release as well, so stay tuned!

Misc

Inline classes

Inline classes are an experimental feature of Kotlin. We have a couple of tests to track Moshi's behavior with them and, while they appear to "work" at the time of writing, these are not officially supported by Moshi. We are looking forward to them though, as they'll also allow Moshi to do some neat things. We'll revisit once they are stable in Kotlin!

Optimizations

JsonAdapter has a couple of commonly used convenience methods to better control null-safety - nullSafe() and nonNull(). These are often called defensively (especially nullSafe()) and can cause a steady buildup of logically-duplicating wrappers. In Moshi 1.9, these APIs will avoid wrapping again if the type passed in is already the target behavior.

Generated code has optimized toString() and error messages to better de-dupe Strings. See the "String Duplication" section of this blog post for more details on why this is helpful: https://jakewharton.com/the-economics-of-generated-code/#string-duplication.

ClassFactory APIs and Android App Compatibility

With Android SDK 28, Android has a grey list of APIs that shouldn't be accessed via reflection. There are a couple of these that Moshi uses to reflectively create classes for POJOs when no default constructor present. By default, you'll see a warning in logcat about this. Note however that Android will crash your app if you have strict mode set up to penaltyDeath() these issues. The workaround is to add an empty default constructor, or switch to something non-reflective such as auto-value-moshi.

Tons of bugfixes!

Nuff said, a year is a long time. Check out the changelog for full details!

Thanks to Dan Lew, Jesse Wilson, and Ryan Harter for reviewing this.