Allow publishing to a pre-defined staging repository

[IgnatBeresnev]

Due to Gradle running publishing tasks in parallel, multiple staging repositories can be created within a couple of seconds with all artifact files scattered throughout them. Even though the publishing in done on the same build agent.

Interestingly enough, setting -Dorg.gradle.parallel=false didn't help, I believe it's due to us using included builds, but I'm not sure.

Setting --max-workers=1 additionally did help, but significant time was spent to find and test this solution.

Given it's easy to forget to set these params (especially when adding new build steps in TC), I think it's much easier to just allow to publish to a predefined staging repo, which will be created before the publishing build is run (as a prerequisite).

[adam-enko]

Makes sense to me.

It'd also be worth replacing the config-cache disabler here

dokka/build-logic/src/main/kotlin/dokkabuild.publish-base.gradle.kts

Lines 92 to 97 in 7fc9244

	tasks.withType<PublishToMavenRepository>().configureEach {
	// Configuration Cache allows tasks to run in parallel. Maven repositories (especially Maven Central)
	// can't cope with parallel uploads, and might drop or split publications.
	// As a workaround, disable Configuration Cache whenever publishing to remote Maven repos.
	notCompatibleWithConfigurationCache("Prevent parallel publishing tasks")
	}

And instead using a build service to limit task parallelization:

//region Maven Central can't handle parallel uploads, so limit parallel uploads with a service.
abstract class MavenPublishLimiter : BuildService<BuildServiceParameters.None>

val mavenPublishLimiter =
  gradle.sharedServices.registerIfAbsent("mavenPublishLimiter", MavenPublishLimiter::class) {
    maxParallelUsages = 1
  }

tasks.withType<PublishToMavenRepository>().configureEach {
  usesService(mavenPublishLimiter)
}
//endregion

[whyoleg]

It'd also be worth replacing the config-cache disabler here

Good idea!
Additionally, AFAIU, this could be dropped at all when DOKKA_MVN_CENTRAL_REPOSITORY_ID is set, as there should be no issues then. Am I correct?

If so, something like this could be done instead:

//region Maven Central can't handle parallel uploads, so limit parallel uploads with a service.
abstract class MavenPublishLimiter : BuildService<BuildServiceParameters.None>

val mavenPublishLimiter =
  gradle.sharedServices.registerIfAbsent("mavenPublishLimiter", MavenPublishLimiter::class) {
    maxParallelUsages = 1
  }

val repositoryId: String? = System.getenv("DOKKA_MVN_CENTRAL_REPOSITORY_ID")
tasks.withType<PublishToMavenRepository>().configureEach {
  if (!repositoryId.isNullOrBlank()) usesService(mavenPublishLimiter)
}
//endregion

But in the end, as we control DOKKA_MVN_CENTRAL_REPOSITORY_ID, and we do expect that it will be set on TC when publishing to MC, we can just drop notCompatibleWithConfigurationCache configuration.

[adam-enko]

Good idea! Additionally, AFAIU, this could be dropped at all when DOKKA_MVN_CENTRAL_REPOSITORY_ID is set, as there should be no issues then. Am I correct?

In theory yes, AFAIK Maven Central is the only problematic repo. But in practice I would lean towards keeping the build scripts simpler, even if it introduces a performance hit when publishing :). Is parallel publishing important for Dokka? Would it help significantly?

[whyoleg]

But in practice I would lean towards keeping the build scripts simpler, even if it introduces a performance hit when publishing :). Is parallel publishing important for Dokka? Would it help significantly?

Parallel publishing is not that important, but, as I mentioned in the end, as we intend to publish to MC only from TC with DOKKA_MVN_CENTRAL_REPOSITORY_ID set, then we don't need to have additional setup in build scripts to de-parallelize publishing, as it will be just not needed at all, as it's just a hack for MC behaviour.
So by removing this, we will keep build scripts simpler :)

[IgnatBeresnev]

@adam-enko @whyoleg thank you for the proposals!

Indeed, it seems like the only repo struggling with parallel uploads is Maven Central, so if we start using DOKKA_MVN_CENTRAL_REPOSITORY_ID, the parallelization limitations could be removed.

I'll merge this PR in to unblock me with adapting TeamCity build configurations, and will create a separate PR with the removal of config-cache disabler - there might be more questions there