In this post, I'm talking about how I spent a shameful amount of time understanding how Gradle handles input files.

The problem

The Apollo Gradle Plugin is generating Kotlin code from GraphQL files.

If you have GraphQL files like this:

.
└── src
    └── main
        └── graphql
            └── com
                └── example
                    ├── GetUser.graphql
                    ├── GetProduct.graphql
                    └── GetReview.graphql

The Apollo Compiler generates Kotlin classes like this:

com.example.GetUser
com.example.GetProduct
com.example.GetReview

In a nutshell, the compiler looks at the relative path of the file and uses that as the package name.

Build cache relocation

A naive implementation could use the "base" directory as input so that it can compute the package names:

  @get:PathSensitive(PathSensitivity.RELATIVE)
  @get:InputFiles
  abstract val inputFiles: Set<File>

  // The base directory to compute the package name
  // This isn't great
  @get:Input
  abstract var baseDir: String

  @TaskAction
  fun taskAction() {
    inputFiles.forEach {
      val packageName = it.relativeTo(File(baseDir))
                          .canonicalPath
                          .replace(File.separatorChar, '.')
      // ... 
    }
  }

That works but it also completely breaks build cache relocation. If we were to copy our repository to another directory on our machine (or in CI), then no cache results could ever be reused.

Entering FileCollection

The solution is to use FileCollection and its mutable cousin ConfigurableFileCollection.

FileCollection is a lazy API that can resolve Configurations but most importantly may contain "roots". In other words, a FileCollection contains `File`s but also their relative path to the root(s).

At a high level, Gradle doesn't havejava.io.Fileinputs. It's always java.io.File + "fileIdentifier".

fileIdentifier being a name, a relative or absolute path, or even empty if only the contents of the file matter and you're using PathSensitivity.NONE.

So how do we get that identifier? Well, for NAME_ONLY we can use file.name, for NONE, we can use "" and for ABSOLUTE, we can use file.asbolutePath. But what about RELATIVE?

The secret to getting the relative path from a FileCollection is using asFileTree which returns the underlying FileTree if any (which is really a FileForest because FileTrees can contain multiple roots):

  @get:PathSensitive(PathSensitivity.RELATIVE)
  @get:InputFiles
  abstract val inputFiles: ConfigurableFileCollection

  @TaskAction
  fun taskAction() {
    inputFiles.asFileTree.visit {
      // Yay, we have the File here alongside its relative path 🎉
      val packageName = path.replace(File.separatorChar, '.')
      // do codegen
    }
  }

Using ConfigurableFileCollection, we get all the information about our files, including their relative path.

It's a very convenient API that non only allows resolving dependencies (Configurations are FileCollections) and filtering (FileTree.include()) and do so in a lazy way but most importantly, they make it possible to model input files without breaking cache relocation! SUCCESS!

Looking ahead, Gratatouille will not allow simple `File` inputs to model the fact that task inputs also have a name (even if empty). Hopefully that will make the task of writing Gradle tasks easier!

Brainteasers pictures frommtairymdonInstructables

I just realized some of the Gradle APIs are like those metal wire brain teasers games:

• Are they easy? No.

• Do you have to turn them upside down for hours (or months...) trying to understand what you're supposed to do? Yup, most probably.

• Do you get a nice feeling of completion once you figure out how to untangle them? Absolutely!

I just had a couple of "Aha!" moment very recently. This post is about the "Aha!" moment of untangling dependency resolution and project isolation.
A follow up one will be about input files.
Warning, spoilers ahead, do not read if you want to play the game yourself.

The problem

If you have a multi-project build, there are a number of occasions where aggregating artifacts is desirable. Dokka is an example. Maven publication is another one. Each project (also known as module) creates their artifacts and you want to collect them in the root project to upload everything all at once.

# aggregate all publications accross all modules and publish them
# to the Maven Central Portal
./gradlew publishToMavenCentral

Sounds easy, right?

Well... Not that much. There are a lot of traps along the way.

Project isolation

Project isolation is an initiative to make builds faster by parallelizing the creation of the task graph. You enable it with the Dorg.gradle.unsafe.isolated-projects flag.

Up until recently it wasn't really an option to enable it but it's been picking up steam lately!

Let's take a simple build:

By using project isolation, building the task graph for project1 and project2can be executed in parallel. That's right, actually using all the CPUs in your machine 🎉.

If you've read the documentation about sharing outputs between projects, you know that you shouldn't do things like this:

// build.gradle.kts
dependencies {
   // Don't do this!
   add("aggregate", project(":project1").tasks.named("generatePublication")
}

From the documentation:

This publication model is unsafe and can lead to non-reproducible and hard to parallelize builds.

If the projects are evaluated concurrently then accessing the mutable Project.tasks is prone to race conditions.

If we want to support project isolation, we need to do this instead in our root (consumer) project:

// build.gradle.kts
dependencies {
   // Do this
   // This is project isolation compatible
   // (PS: this doesn't use attributes for REASONs!)
   add("aggregate", project(":project1", "outgoingPublication")
}

To expose the artifacts in our producer project, we use configurations and the artifacts {} API:

// project1/build.gradle.kts
val configuration = configurations.consumable("outgoingPublication").get()

val m2Dir = layout.buildDirectory.dir("m2/").get()
publishing {
  // Add a local repository
  repositories {
    maven {
      name = "m2"
      setUrl(uri(m2Dir.asFile.path))
    }
  }
}
artifacts {
  artifacts {
    // expose the repository to other projects
    add(configuration.name, m2Dir) {
      builtBy("publishAllPublicationsToM2Repository")
    }
  }
}

(There's a lot in there and if you're curious about how that works in details, take a look at this other post about Gradle configurations).

Using the configurations and artifacts API, we can express cross-project dependencies. But it's a bit verbose.

Avoiding repetition

The above works fine but assuming you have 50 modules, you now have to do something like this:

// build.gradle.kts
dependencies {
   add("aggregate", project(":project1", "outgoingPublication")
   add("aggregate", project(":project2", "outgoingPublication")
   // ...
   // Pfewwwww, that's a long list of projects to maintain
   add("aggregate", project(":project50", "outgoingPublication")
}

Wouldn't it be nice if instead we could register the dependency automatically? After all, the subprojects have some build logic that runs already. Could they automatically register themselves as dependencies of the root project?

To this day I haven't found a way to add cross-project dependencies in a project isolation compatible way (Gradle issue).

But luckily there is another solution!

Lenient artifact views

The trick is in using subprojects and lenient artifact views:

// Create the configuration
val aggregate = configurations.create("aggregate")
// Add all subprojects as dependency
// ℹ️ subprojects {} is not PI compatible
// (but subprojects.forEach {} is)
subprojects.forEach {
  aggregate.dependencies.add(
    dependencies.project(":${it.name}", "outgoingPublication")
  )
}

tasks.register("zipAggregate", Zip::class.java) {
  // Use a lenient artifact view to avoid failing if some subprojects
  // do not publish anything.
  from(aggregate.incoming.artifactView { lenient(true) }.files.asFileTree)
  destinationDirectory.set(layout.buildDirectory.dir("zip"))
  archiveBaseName.set("aggregate")
}

By using a lenient artifact view, the root project can still collect all subprojects. Because the dependency is added on the explicit "outgoingPublication" configuration, there is no risk of resolving other (wrong) artifacts. And because the resolution is lenient, it will fail silently if a project doesn't publish anything. SUCCESS!

Wrap-up

I've been toying with publishing and Maven Central APIs for a while now. Project isolation was something that has been itching me for the last 6 months or so.
Using lenient artifact views provides a simple solution that would otherwise require a lot of fragile code.

With this itch gone, new opportunities for publishing arise, stay tuned!

PS: this only works for cases where a single root projects depends on all the other ones but modeling a more complex project graph is still itching me...

Brainteasers pictures from mtairymd on Instructables

This is usually the last resort solution. But in my life of Gradle scripts engineer, I found the odds of afterEvaluate {} actually "fixing" your issue are in effect quite high. II guess that explains while you'll still find it in a lot of places.

Because afterEvaluate {} postpones your code's execution, it gives more time to other plugins and build logic to execute their "stuff", whatever that is. If you code relies on some of that "stuff" then your build is "fixed" (note all the quotes there).

This is very fragile though for a number of reasons:

• Anyone wanting to use your code now also needs to use afterEvaluate {} which is not obvious at all
• Anyone wanting to use the code that uses your code also needs to use afterEvaluate{} and so on, all the way down
• It becomes even more complicated as more plugins/scripts are involved
• More generally, it makes dependencies between plugins/scripts implicit and error prone

So how can we make things more solid? The good news is solutions exist!

Usually, afterEvaluate {} is required in two occasions:

1. reading extension properties
2. configuring defaults

(if you have more use cases, please reach out, I'm curious to learn about them and will update this article)

Reading extension properties

Gradle plugins are configured with extensions. I won't go into the details of extensions here, the Gradle doc is, as usual, very comprehensive on the matter. The tldr; is that extensions are simple classes that expose options for your plugin. In a sense, it is the public API of your plugin.

This is working well until you need something in your extension to change how the task graph is created. Assume you want to customise the name of your task for an exemple. It doesn't work.

open class MyExtension {
  // The name of the person to greet
  var personToGreet: String? = null
}

open class MyPlugin: Plugin<Project> {
  override fun apply(target: Project) {
    val extension = target.extensions.create("myPlugin", MyExtension::class.java)

    // DOES NOT WORK 😞
    // extension.personToGreet is not set here because your 
    // `build.gradle.kts` file is not evaluated yet
    // Your task will be named "greetnull"
    target.tasks.register("greet${extension.personToGreet}") {
      it.doLast {
        println("Hello ${extension.personToGreet}")
      }
    }
  }
}

It is tempting to wrap everything in afterEvaluate{} and wait for your build.gradle.kts to be evaluated but please don't do this for the reasons explained above.

Instead, you can expose a function in your extension. This way, you can execute logic and tweak the task graph from your code. You have full control over when your code executes without relying on the afterEvaluate {} chaos:

class GreetingOptions {
  // The name of the person to greet
  var personToGreet: String? = null
}

open class MyExtension(val project: Project) {
  fun greeting(action: Action<GreetingOptions>) {
    val options = GreetingOptions()
    action.execute(options)

    // WORKS 🤩
    // personToGreet is always set here
    // You have full control of the timing
    project.tasks.register("greet${options.personToGreet}") {
      it.doLast {
        println("Hello ${options.personToGreet}")
      }
    }
  }
}

There is one drawback to this solution is that it makes the callsite a bit more verbose:

// build.gradle.kts
myPlugin {
  // there's one additional level of nesting here
  greeting {
    personToGreet = "Björn"
  }
}

But trust me it's a minor drawback compared to the afterEvaluate {} hair pulling especially as your plugin grows, this initial function call will be very small in your total API surface.

Configuring defaults

The other case is about default behaviour. Sometimes, you don't want your users to specify anything and have valid defaults that can be overridden by the user.

apply {
  id("myplugin").version("1.0")
}

// no other configuration, just use defaults

This one is trickier. Because there is no extension, there is no function where you can run your code... I've looked at this from every direction and the more I look at this, the more I think you should not support this...

There are ways to do this without afterEvaluate. For an example, you could always create the default tasks and then disable them as soon as the extension gets configured. That could work. But this also adds more tasks in ./gradlew --tasks, more clutter to your build for no obvious reason.

So I think (but I still have to do it) that forcing the user to opt-in the default is an acceptable workaround here.

apply {
  id("myplugin").version("1.0")
}

// One time configuration
myPlugin {
  // opt-in the default behaviour
  greeting()
}

Sure it's 3 extra lines and it's not great but in the grand scheme of things, it's a one-time cost that's going to save you a bunch of head-aches over time. And I don't really see another way around.

Is afterEvaluate {} that bad?

Short answer as you guessed is yes!

There is one area though where I find it useful, it's to perform consistency checks. In the example above, it doesn't make sense to force the user to either use the default greeting or define their own.

In these cases, I found it useful. As it's localised to your own code and doesn't introduce timing issues or dependencies with other plugins, it usually works well.

abstract class MyPlugin : Plugin<Project> {
    override fun apply(target: Project) {
        val extension = target.extensions.create("myPlugin", MyExtension::class.java, target)

        target.afterEvaluate { 
            check(extension.setupDone) {
                """
                    myplugin: you either need to define your greeting or use the default one with `defaultGreeting()`
                """.trimIndent()
            }
        }
    }
}

Conclusion

All in all, whether you trying to read extension properties or provide defaults, I like to think of the entry point to your plugin as a "function" and not just a class. It's something where you can write code. It has drawbacks for sure, it's more verbose and all but once you do that, you can finally say goodbye to that afterEvaluate {} chaos and start sailing to new Gradle horizons!

Have you ever tried to do something like this?

val kotlinVersion = "1.7.21"

plugins {
    id("org.jetbrains.kotlin.jvm").version(kotlinVersion)
}

dependencies {
    implementation("org.jetbrains.kotlin:kotlin-stdlib:$kotlinVersion")
}

Looks pretty neat, right? Your Kotlin version is defined in a single place and it's easy to change it when a new version is release.

If you have tried this, you probably know it doesn't work. You'll get hit pretty quickly by this:

e: build.gradle.kts:4:44: Unresolved reference: kotlinVersion

That's surprising. There are very good reasons for this but they're not obvious at first sight.

Let's dive in.

Plugins and classpaths

Gradle support plugins. Plugins are very handy because they allow you augment your builds. If you're an Android developer, you're certainly familiar with snippets like this:

android {
    compileSdk = 32

    defaultConfig {
        applicationId = "com.example"
        minSdk = 23
        targetSdk = 32
        versionCode = 1
        versionName = "1.0"
    }
}

Looking a bit closer, this is all compiled and typesafe. Gradle knows compileSdk is a var Int, same for minSdk, targetSdk and others.

This means that Gradle knows about com.android.build.api.dsl.CommonExtension the class that defines compileSdk. Since Gradle cannot put the whole world on the build script classpath, this has to be conveyed somehow. This is what plugins {} do.

Multiple passes of compilation

Gradle parses your build.gradle.kts file and extracts the plugins {} block. It does so using the same KotlinLexer that the Kotlin compiler uses. Once the plugins {} block extracted, Gradle compiles it and runs it. This is also the moment when generated accessors are ... well... generated!

So after the plugins {} block is evaluated, Gradle has:

• the plugins used by the script and their matching jar
• generated accessors

In a second pass, it can compile and evaluate the script, with all the plugin jars on the classpath. It all makes sense!

This first pass is why the syntax of the plugins {} block is so constrained.

It's not only plugins {}

plugins {} is the most used block but this also applies to other blocks:

• buildscript {}
• pluginManagement {}
• iniscript {}

If you bump into errors, double check what block you're in. Chances are that your code is evaluated in a separate context.

What's working

Once we know that, how do we make the above work? In general, I'm not 100% sure what syntax is allowed or not in these blocks. The good news is that there are multiple solutions to define your Kotlin version in a single place (or do other things).

The most straightforward is to use version catalogs:

// libs.versions.toml
[versions]
kotlin = "1.7.21"

[libraries]
kotlin-stdlib = { group = "org.jetbrains.kotlin", name = "kotlin-stdlib", version.ref = "kotlin" }

[plugins]
kotlin = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }

// build.gradle.kts
plugins {
    alias(libs.plugins.kotlin)
}

dependencies {
    implementation(libs.kotlin.stdlib)
}

There are other solutions using pluginManagement and Gradle properties. Or you can build your own!

In all cases, I hope this post helped you understand how Gradle processes your build script. It's nothing magical!

This year, we decided to rewrite the Android app (github) in Jetpack Compose and this too was a lot of fun 😃! We got a functional app in no time and sent it out for public scrutiny. We got a lot of feedback and contributions (did I mention this community rocks? 🤘💙)!

This is the story of how (and why) we made our google-services.json public so that anyone could easily contribute.

About google-services.json

Whenever you setup a Firebase project, google-services.json comes up pretty quickly in the installation instructions. You usually download it from the Firebase console:

If you open it, you'll find something like this (some values redacted although I'm not 100% sure why, more on that later):

{
  "project_info": {
    "project_number": "378302699578",
    "project_id": "openfeedback-am-2022",
    "storage_bucket": "openfeedback-am-2022.appspot.com"
  },
  "client": [
    {
      "client_info": {
        "android_client_info": {
          "package_name": "fr.androidmakers"
        }
      },
      "api_key": [
        {
          "current_key": "[redacted]"
        }
      ],
    }
  ],
  "configuration_version": "1"
}

As you can see, there's something called api_key up there. That sounds pretty secret so certainly it's a good idea to hide it, right? Well... turns out it's not secret at all.

Your Android API key is not secret!

Because your app needs your API key at runtime, the API key must be accessible somewhere in the app. So really it can't be secret. To prove it, let's try to retrieve it.

Start by doing a google search for the Android Makers app apk. This takes you to an ApkPure website that allows you to download the apk:

A click and and a few downloaded MB later, you can open that APK in the Android Studio APK analyzer (Build -> Analyze APK...). Look for a resource that looks like an API key. This google_api_key string sounds like a good candidate 🙃:

Yup, zoom a little bit... That's the one! So I'm not sure why I redacted it above. Certainly because everything is made to make you believe this should be treated as a secret.

Scary warnings

If you ever commit such an API key to Github, all the repository admins will receive an email along these lines:

Well, that's scary 🙀...

Same thing if you publish an app to the Google play that contains your API key in Kotlin code:

This is also scary 😱...

Why all these warnings if the API key can be retrieved in a few minutes by any malicious user? That remains a mystery to me. My current guess is that there's no way to distinguish between a client key and a server key so that warning is sent to anyone indiscriminately. If someone can see of a good reason to show these warnings for an Android API key, please tell me so I can edit this article and redact everything for good 😅 (and invalidate all the things too!).

Developing in the open

After browsing the web a bunch, realising that there was an official answer in the firebase forums (edit: and it's even in the official Firebase documentation as @zsmb13 made me realize later) and double checking that the values were in the apk anyway, we decided to ignore the scary warnings, stop living in fear (🤘) and commit our google-services.json!

Now anyone can develop for the App using the real data. Special thanks to @underwindfall, @R4md4c and @oldergod for their awesome contributions 💙!

What could go wrong?

Assuming someone has your Android API key. What can they do with it?

In our case, the main thing was using our Firestore instance and using our quota. We could imagine someone developing their own service squatting our Firestore instance and therefore not paying the bill at the end of the month 💸.

While technically possible, I'm not sure how practical that would be. Such a service would be dependant on a completely foreign infra and any change would break it almost instantly. I'm not aware any such attack has happened already but maybe it's a matter of time...

Additional (actual) restrictions 🔐

The good news is that the Google Cloud console allows to restrict the API keys 🎉

Platform restrictions

You can restrict your API key per platform:

In the example above, we restricted the API key to only be callable from Android apps. As @RickClephas points out on Twitter, this works by using two HTTP headers: X-Android-Package and X-Android-Cert. So it's not bullet proof but it's still something.

APIs restrictions

In addition to platforms, you can also restrict your API key to some APIs:

This makes sure no other APIs can be used so it's an additional safety. Make sure to include Firebase Installations API and Token Service API or your app will stop working after a few calls.

If your feedback was not counted during Android Makers, this is what happened 😅. Thanks to Hugo Gresse and @GerardPaligot for catching this!

Conclusion

You can hide your Android API key from Github but this is not going to prevent anyone to get it. If you need to share it with colleagues or your community, it should be pretty safe to commit it despite all the scary warnings.

At the end of the day, security is never a "yes" or "no" kind of question and has more of a continuum of answers. Some might argue that despites all the flaws, hiding your API key from GitHub is still making it harder to retrieve and that's very true. It's also very true that every security measure has a cost, either in terms of raw dollars or in terms of developer experience.

Of all the possibilities:

• Add GCP API key restrictions
• Fine tune your Firestore security rules
• Check your app integrity with something like Firebase App Check
• Obfuscate your client code with something like DexGuard
• Implement server side rate limiting and fraud detection
• Hide your API key from Github

I'd argue that hiding your API key from Github is the one with the worst cost/security ratio. If you're relying on that for your security, you'd better add API key restrictions and double check your Firestore rules instead.

Note: none of the above applies to server API keys of course. If your API keys doesn't end up in a client, make sure to keep them secure in the depths of your infra.

This article was edited on May 10th to add a note about the Firebase official doc, the X-Android-Package and X-Android-Cert headers and rework the conclusion to display more possible solutions.

Many thanks to Dorian Cussen and Edouard Marquez for proof reading this article

Cover image background by Claire Tresse

Edit: Blog post updated for Kotlin 1.7, see the last paragraph for details.

Kotlin 1.6 has just been released 🎉 (blog post). This is great news for everyone in the Kotlin ecosystem. As with every feature release, there are new features, new deprecations and other changes that might (but hopefully not too much) break your code.

The Kotlin documentation has two awesome pages to go into the implications of new feature releases:

• The Kotlin evolution - compatibility tools page
• The Compatibility modes page

Additionally, a lot of additional information can be found in the Kotlin issue tracker, like this issue about Kotlin native compatibility

In practice though, I often have a hard time choosing what apiVersion, kotlin-stdlib version to use, etc... This post is an attempt to list common scenarios to help build a better mental model of what's happening under the hood. And also answer the question of:

Should you use Kotlin 1.6 in a library?

Let's try to answer this question.

If you prefer reading code than blog posts, the source for this post is available at: https://github.com/martinbonnin/kotlin-compatibility/tree/main

project compiled with 1.3 using a lib compiled with 1.6

• appCompiler=1.3
• libCompiler=1.6

e: /Users/mbonnin/git/kotlin-compatibility/lib-1-6/build/libs/lib-1-6.jar!
/META-INF/lib-1-6.kotlin_module: Module was compiled with an 
incompatible version of Kotlin. 
The binary version of its metadata is 1.6.0, expected version is 1.1.16.

That sounds appropriate. The evolution principles state that:

Preferably (but we can't guarantee it), the binary format is mostly forwards compatible with the next feature release, but not later ones (in the cases when new features are not used, e.g. 1.3 can understand most binaries from 1.4, but not 1.5).

Here, the 1.3 compiler cannot read the 1.6 metadata which sounds expected. Let's try with 1.4

project compiled with 1.4 using a lib compiled with 1.6

• appCompiler=1.4
• libCompiler=1.6

> Task :app-1-4:compileKotlin FAILED
e: /Users/mbonnin/git/kotlin-compatibility/lib-1-6/build/libs/lib-1-6.jar!
/META-INF/lib-1-6.kotlin_module: Module was compiled with an 
incompatible version of Kotlin. 
The binary version of its metadata is 1.6.0, expected version is 1.4.2.

Good 😌. This was expected. Note how 1.4 expects 1.4.2 metadata while 1.3 was expecting 1.1.16 so something changed but it's still not enough.

Let's try with 1.5

project compiled with 1.5 using a lib compiled with 1.6

• appCompiler=1.5
• libCompiler=1.6

> Task :app-1-5:compileKotlin

BUILD SUCCESSFUL in 5s

Huge success 🙌. Everything works as expected.

Now let's assume that we are a library author and we want our new shiny lib to use 1.6 while still allowing 1.3 users. Is that possible?

Make the lib use apiVersion=1.3, languageVersion=1.3

• appCompiler=1.3
• libCompiler=1.6
• libApiVersion=1.3
• libLanguageVersion=1.3

e: Language version 1.3 is no longer supported; please, use version 
1.4 or greater.

Fair enough. The 1.6 release notes state that:

Starting with Kotlin 1.6.0, you can now develop using three previous API versions instead of two (along with the current stable one). Currently, this includes API versions 1.3, 1.4, 1.5, and 1.6.

That doesn't say anything about languageVersion(Update: Starting with 1.7, languageVersion will also support three previous versions). Let's try with just apiVersion then:

Make the lib use apiVersion=1.3, languageVersion=1.6

• appCompiler=1.3
• libCompiler=1.6
• libApiVersion=1.3
• libLanguageVersion=1.6

e: /Users/mbonnin/git/kotlin-compatibility/lib-1-6-api-1-3-language-1-6
/build/libs/lib-1-6-api-1-3-language-1-6.jar!/META-INF
/lib-1-6-api-1-3-language-1-6.kotlin_module: Module was compiled 
with an incompatible version of Kotlin. The binary version of its 
metadata is 1.6.0, expected version is 1.1.16.

We're back to step one. The new metadata format cannot be read by the old compiler. Maybe if we could tell the 1.6 compiler to output "backward" compatible metadata that could help. Let's see if languageVersion=1.4 can help

Set languageVersion=1.4

• appCompiler=1.3
• libCompiler=1.6
• libApiVersion=1.3
• libLanguageVersion=1.4

> Task :lib-1-6-api-1-3-language-1-4:compileKotlin

BUILD SUCCESSFUL in 1s

🎉 That worked! So looks like languageVersion also affects the metadata format

Takeaways

As a library author wanting to use Kotlin 1.6, my current understanding is that:

• if your users are compiling against your lib with Kotlin 1.n

  • languageVersion should be set to n+1: languageVersion=1.(n+1)
  • apiVersion doesn't really matter as kotlin-stdlib should be resolved to 1.6, which is backward compatible with any previous kotlin-stdlib

• if your users run your lib with a fixed kotlin-stdlib:1.p at runtime (like Gradle)

  • apiVersion should be set to n: apiVersion=1.p

If you bump to 1.6 without configuring anything else, your consumers will have to update to Kotlin 1.5.

Note: The other way around (backward compatibility?) is a way better story: if you don't bump to 1.6 and keep using 1.5 in your library, then all consumers can upgrade to 1.6 and continue using your lib (including native ones).

Update for Kotlin 1.7

The same is still true. If you bump to Kotlin 1.7 in your lib, your consumers are forced into compiling with 1.6 by default (including for native).

You can keep compatibility with the 1.5 compiler with languageVersion="1.5" but that will only go so far because kotlin-stdlib itself contains 1.7 metadata so unless you're going to great length to not expose kotlin-stdlib:1.7 transitively, setting the languageVersion is most likely not going to change much.

Picture: way down by Romain L

If you prefer reading source code, read the matching pull request in apollo-android

Note: This post was written when Kotlin 1.5 was released and Gradle 7.1 was using 1.4. The same is true with Kotlin 1.7 (or any other newer versions)

The 🐔 and 🥚 problem

Say you have a Gradle build. This Gradle build uses Kotlin build.gradle.kts scripts that you wrote. These scripts themselves declare plugins:

// build.gradle.kts
plugins {
  id("org.jetbrains.kotlin.jvm").version("1.5.30")
  id("com.example").version("1.0")
}

Let's also assume that the com.example plugin uses kotlin-stdlib:1.5 and uses some new 1.5 APIs like lowercase.

Gradle needs to compile these scripts using the 1.4 embedded Kotlin compiler, run them, resolve plugins and their dependencies and put them in the buildscript classpath. So the build environment will be like:

+--- org.jetbrains.kotlin:kotlin-gradle-plugin:1.5.30
|    +--- org.jetbrains.kotlin:kotlin-gradle-plugin-api:1.5.30
|    |    \--- org.jetbrains.kotlin:kotlin-stdlib:1.5.30 -> 1.5.31
|    |         +--- org.jetbrains:annotations:13.0
|    |         \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.5.31
+--- com.example:plugin:1.0
|    +--- org.jetbrains.kotlin:kotlin-stdlib:1.5.31

Now in order to run the dependency resolution, and because the scripts themselves are written in Kotlin, Gradle needs a stdlib even before it starts the build [^1].

This is were the chicken and egg problem kicks in. It's too early for Gradle to know the build will ultimately require 1.5 so Gradle puts the version it knows about in the classpath. That's version 1.4 💥.

When our com.example plugin runs, it will crash because lowercase doesn't exist in 1.4 [^2]

Even if it doesn't crash, chances are that you will see a lot of these lines:

w: Runtime JAR files in the classpath should have the same version. These files were found in the classpath:
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-stdlib-1.4.31.jar (version 1.4)
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-stdlib-common-1.4.31.jar (version 1.4)
...

This problem is discussed at length in Github issue #16345. There are multiple workarounds or fixes. Let's go over a few of them and see how relocation can help.

Solution #1: Don't use Kotlin 1.5 in plugins!

After all, it's not such a huge deal, right? We've lived without lowercase for years so we can wait a bit more. What's more, the Kotlin compiler supports apiVersion. If you're a plugin author, you can make sure your bytecode doesn't use any 1.5 APIs:

compileKotlin {
  kotlinOptions {
    // Compile against 1.4 stdlib for the time being to make sure it works 
    // with a wide range of Gradle versions.
    apiVersion = "1.4"
  }
}

That works but not being able to use the latest APIs feels bad. What's even worse is that apiVersion only works for your code, not for dependencies of your plugin. If you want to use okio in your Gradle plugins, you're out of luck with this solution.

Solution #2: Use Gradle Worker's API

For plugins, Gradle exposes the Worker API. Especially, it has a classloaderIsolation mode that allows to isolate the plugin's classloader from the Gradle classloader.

It's a bit more work because you'll have to pass parameters to the worker but it gives full flexibility about the classpath. I'm still not 100% clear how much this is a rock solid solution given that the public API (extensions, tasks, etc...) will also use Kotlin and this cannot be moved to a worker. Maybe if you make sure the public API is simple enough to not use any new API and keep the tasks implementations for 1.5... I'd be curious if anyone has had any success with this.

Solution #3: Always update Gradle to the latest version

Given that Gradle is relatively fast to update the embedded version of Kotlin, you can wait until it's updated before using it in your plugin. That's always too much waiting but it can be acceptable in most cases. Of course, that also means any user of your plugin is also on this fast update path, which might or might not be acceptable.

Solution #4: Relocation!

Finally, the solution that should work in all cases!

No need to wait or change your plugin code, just ship your own kotlin-stdlib as part of your plugin jar. As a nice bonus, that even fixes other issues with buildSrc or classloaders. Sounds simple, right? Well... let's see what it takes to relocate the kotlin-stdlib

Relocating with Shadow?

The Gradle Shadow Plugin has been used to relocate countless Gradle plugins like SQLDelight. It's working well most of the time. Unfortunately, when it comes to Kotlin, there are some details that make it harder to use. The biggest issue is that Shadow uses MavenShade under the hood and this relocates constant strings that shouldn't be. In practice, using it to relocate kotlin will transform code like this:

// Get the Kotlin plugin extension (to get sourceSets or anything else)
project.extensions.getByName("kotlin")

into code like that:

// Yikes, there's no "com.your.plugin.relocated.kotlin" extension :-(
project.extensions.getByName("com.your.plugin.relocated.kotlin")

This is pretty unexpected and breaks most of plugins interacting with the Kotlin plugin.

For plugins that generate source code and contain a lot of package names, this might require even weirder workarounds.

Using R8

R8 is Google's replacement for Proguard. While R8 is mainly used for Android, it can be used with any jar file too, including Kotlin. By using proguard rules, we get a lot more control about the relocation, what is kept and what is not.

Unfortunately, R8 is not super easy to consume. It is only published in minified form at maven.google.com. I wrote a small Kotlin script to publish the non-minimized artifacts to Maven Central. I also wrote a small plugin to make it easier to use R8 with Gradle. It's named GR8 because it's great and you can find it on Github at https://github.com/GradleUp/gr8.
(Edit: you can also call R8 directly from your Gradle scripts if you prefer as shown in this repo from @ephemient )

The rest of this article goes through the process of setup up GR8 for a fictional com.example plugin using Kotlin 1.5.

Applying the GR8 plugin

Applying the GR8 plugin is similar to any other plugins:

plugins {
  id("org.jetbrains.kotlin.jvm").version(kotlinVersion)
  id("java-gradle-plugin")
  // You can use `kotlin-dsl`too, it's going to set apiLevel="1.4" automatically
  id("kotlin-dsl")
  id("com.gradleup.gr8-plugin").version("0.2")
}

Create a "shade" configuration that will contain all the dependencies to shadow (including kotlin-stdlib.jar):

// Configuration dependencies that will be shadowed
val shadeConfiguration = configurations.create("shade")

Declare your dependencies:

dependencies {
  // no need to specify the version, this will use the same version as the kotlin plugin version 
  add("shade", "org.jetbrains.kotlin:kotlin-stdlib")
  // add your other dependencies here
  // Dependencies can use whatever version of Kotlin they want \o/
  add("shade", "com.squareup.okio:okio:3.0.0")
  // ...

  // Add gradleApi() as a compile-only dependency, not shadowed
  compileOnly(gradleApi())
  // Alternatively, you can use Nokee distributions
  compileOnly("dev.gradleplugins:gradle-api:7.2")
}

Don't forget to remove kotlin-stdlib from the default dependencies to avoid having it in the pom file/Gradle module file. In your gradle.properties file, add:

kotlin.stdlib.default.dependency=false

(See the Kotlin docs for more details about kotlin.stdlib.default.dependency)

Now configure the GR8 plugin:

gr8 {
  val shadowedJar = create("shadow") {
    configuration("shade")

    // The R8 configuration
    proguardFile("rules.pro")

    // Remove proguard rules from dependencies, we'll manage them ourselves
    exclude("META-INF/proguard/.*")
  }

  // If you're using the `java-gradle-plugin` plugin. It will add `gradleApi` to the API configuration
  // We don't want that, we want to control what's going out
  removeGradleApiFromApi()

  // Needed for the plugin to compile
  configurations.named("compileOnly").configure {
    extendsFrom(shadeConfiguration)
  }
  // Needed for the tests to compile
  configurations.named("testImplementation").configure {
    extendsFrom(shadeConfiguration)
  }

  // When publishing, publish the shadowed Jar
  replaceOutgoingJar(shadowedJar)
}

That's it for the Gradle configuration! Now you need to tell R8 what to keep and what to relocate. This is done using rules.pro

Configuring R8 rules

This is where things begin to be project specific. If you use reflection or other dynamic features, you might need to fine tune the rules. As a rule of thumbs though, you will always need to keep your plugin public API:

# Keep the API as it's used from build scripts
-keep class com.example.gradle.api.** { *; }
-keep interface com.example.gradle.api.** { *; }
-keep enum com.example.gradle.api.** { *; }

Then tell R8 to repackage (relocate) classes:

-repackageclasses com.example.relocated

# Help the relocation process by allowing to change the visibility of some members
-allowaccessmodification

Some helpful options:

# Ignore warnings for all the compileOnly dependencies that we didn't pass to R8
-ignorewarnings

# Makes it easier to debug on MacOS case-insensitive filesystem when unzipping the jars
-dontusemixedcaseclassnames

# Keep annotation and other things (might be overkeeping a bit but that's without consequences on the relocation itself)
-keepattributes Signature,Exceptions,*Annotation*,InnerClasses,PermittedSubclasses,EnclosingMethod,Deprecated,SourceFile,LineNumberTable

That's the minimal set of rules. Depending on what your plugin uses, you will most likely need some other ones.

A note about kotlin.Metadata

An interesting question is whether to relocate kotlin.Metadata or not. The compiler uses kotlin.Metadata at compile time. That enables a lot of Kotlin-only features like extension functions, top-level functions, named parameters, default parameters, typealiases, properties, etc... If you relocate them, the compiler will miss a lot of information and fail to compile for things like top-level functions:

// build.gradle.kts
import com.example.gradle.api.someTopLevelFunction

// Unresolved reference: someTopLevelFunction
someTopLevelFunction()

// Default parameters, type aliases, etc... also won't compile

If your public API uses a lot of Kotlin features, and if you want your users to be able to use them, keep kotlin.Metadata:

// Keep metadata used by the compiler
-keep class kotlin.Metadata { *; }

Of course, that's taking the risk that a future version of kotlin changes the definition of kotlin.Metadata and overrides that annotation. I'm not 100% sure what would happen there but hopefully it shouldn't happen too much. And if it does happen, most likely other things will break.

If you're keeping kotlin.Metadata, you'll also need to keep kotlin.Unit:

// The compiler also uses Unit
-keep class kotlin.Unit { *; }

kotlin.Unit is a special value to the Kotlin compiler and if you relocated it to say, com.example.relocated.aa, a runtime error will happen on void methods because the bytecode references a method that returns com.example.relocated.aa instead.

Profit!

Run ./gradlew assemble to build the shadowed jar. It will be available in

build/gr8/shadow/projectName-version-shadowed.jar

In order to verify that your classes were repackaged correctly, unzip the jar. Most of the classes should be under com/example/relocated. If not, iterate on the rules.conf to allow more relocations. Make sure to also test your plugin to make sure runtime reflexive accesses are still working.

If everything works, congrats! You can now use Kotlin 1.5 and all its APIs in your Gradle plugin! To see it in action, check the matching PR in apollo-android

Should I use this in production?

That's always the million dollar question. I'll give the usual answer of "it depends" 🙃. This is all wildly experimental so come prepared for some hickups. Relocation is always fragile as you don't find issues until runtime. Also, running R8 takes some time. If you need to do this while debugging, that can become annoying.

All that being said, if you have a good test suite and it's working for you, you can actually make it a lot easier to consume your plugins.

Moving forward

We've seen that R8 gives you a lot of flexibility to shadow, and even shrink/optimize your Gradle plugins. It has a lot of advantages like avoiding dependencies conflicts, allow to use newer Kotlin APIs as well as making self-contained Gradle plugins. It is an effective solution to shipping plugins that use latest Kotlin APIs today. It also has drawbacks as the configuration is not straightforward and reflective accesses require extra care.

Another important drawback is that this requires more resources. If every plugin starts shadowing all their dependencies, it means loading the Kotlin stdlib N times, loading okio P times, etc... That's a lot of classes loaded in the JVM. All of these add up and will take more memory and make your builds slower.

As a wrap-up from this long post, please take the time to upvote https://github.com/gradle/gradle/issues/16345 to prioritize removing the Kotlin stdlib fixed runtime limitation in Gradle. If you're a plugin consumer, move buildSrc and rootProjects buildscript {} to convention plugins and hopefully one day we'll have Gradle plugin that can share their dependencies \o/.

Thanks for making it through! Have a wonderful day/evening/night!

[^1]: Actually that's not entirely true as the plugins {} block is a special block that doesn't allow all the Kotlin syntax so maybe it could have a special handling 🤷‍♂️

[^2]: This is not entirely true either 😅. In most cases, the compiler will just optimize the constant value, or use the 1.4 experimental lowercase but that was a nice example for a relatively simple and famous 1.5 API.

🙏 Many Thanks to LouisCAD for proofreading this article.

Cover Picture: Chiesetta sul Col del Nivolet by BORGHY52