Reproducible Builds

[IzzySoft]

At IzzyOnDroid we support Reproducible Builds (see: Reproducible Builds, special client support and more at IzzyOnDroid). Trying for yours, I was able to successfully generate the APK using ./gradlew assembleFossRelease, but the resulting APKs were not identical. One culprit is an embedded build timestamp – but even if I "override" that (using sed to replace ${buildtime()} with the timestamp from your APK), there are differences in classes.dex remaining, see contents of
diff.zip. The essential parts are:

-  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/lists/differ/AsyncDiffer$callback$1;" }
+  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/lists/differ/AsyncDiffer$callback$1<" "TA;TT;>;" }

-  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/viewbinding/ViewBindingProperty$onDestroyObserver$1;" }
+  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/viewbinding/ViewBindingProperty$onDestroyObserver$1<" "TComponentT;TBindingT;>;" }

(- is your APK, + is the one I built). In case it's relevant: the build here happened on Debian bookworm with OpenJDK 17.

I hope you have an idea where that difference comes from, and how it can be avoided? Also, could the buildtime() be replaced by something reproducible (e.g. commit count, commit hash, time of the last commit) or skipped altogether?

We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.

Thanks in advance – and looking forward to your reply!

[d4rken]

I can change the build-time being included, but I have no idea where the other differences come from. Don't have time to look into that either, sorry.

[IzzySoft]

Fair enough Matthias – thanks! So maybe fix that build-time thingy, and confirm which SDK you build with and on what OS (so we can rule out anything coming from that end)? Mine is stated above – but I can switch to e.g. ubuntu:jammy and OpenJDK-21 when needed.

[IzzySoft]

confirm which SDK

Oops, JDK I meant of course… But OpenJDK-21 on ubuntu:jammy results in the very same dex diff 🤷‍♂️

[d4rken]

and confirm which SDK you build with and on what OS (so we can rule out anything coming from that end)? Mine is stated above – but I can switch to e.g. ubuntu:jammy and OpenJDK-21 when needed.

permission-pilot/.github/workflows/release-tag.yml

Lines 38 to 41 in c66a1cf

	uses: actions/setup-java@v3
	with:
	distribution: 'adopt'
	java-version: 17

JDK17 on ubuntu-latest (which should be "Ubuntu 24.04" according to https://github.com/actions/runner-images)

[IzzySoft]

Thanks! So… wait: "adopt"? I must have missed that. Is there any reason not to use OpenJDK? Could be that Adopt deals a little different with annotations, so this could be the cause of the differences.

[d4rken]

Hm not that I remember, I think I took it from a blog post about setting up runners for Android 🤔

[IzzySoft]

Ah. Mind to make a build with OpenJDK, attach the APK here (just rename it to .zip so you can attach it) and mention the commit it was built from – so I test if that solves the issue before you start "really changing things"?

[IzzySoft]

@d4rken guess you're drowning in tasks the same as I am, and might have missed my last comment?

[d4rken]

I forgot, sorry.

This is from my local machine, though the releases are usually always done via GitHub CI.

java --version
openjdk 17.0.13 2024-10-15
OpenJDK Runtime Environment (build 17.0.13+11-Ubuntu-2ubuntu122.04)
OpenJDK 64-Bit Server VM (build 17.0.13+11-Ubuntu-2ubuntu122.04, mixed mode, sharing)

./gradlew assembleFoss

eu.darken.myperm-v1.7.0-rc0-10700000-FOSS-RELEASE-c66a1cf.zip

[IzzySoft]

though the releases are usually always done via GitHub CI.

So it should basically be what release-tag.yml specifies: Adopt JDK 17.

Here's what I use: OpenJDK-17 on Debian bookworm, and this recipe:

build:
  - sed -r '/signingConfigs\[/d ; s/\$\{buildTime\(\)\}/2024-07-12T17:26:33.187777090Z/' -i app/build.gradle.kts
  - chmod +x gradlew
  - ./gradlew assembleFossRelease

(the sed is needed as you use an embedded build timestamp, which makes the build not reproducible otherwise – that's something I'd ask to address once we got the app RB in general with this "dirty hack").

Looking at the diff again: are those "annotations" which differ? That could mean a different JDK causing the issue. Unfortunately, I cannot try with the Adopt JDK as that's not available in the standard Debian or Ubuntu packages. I often see Github workflows using Temurin, though, which usually produces identical output to that of OpenJDK.

PS: As you use OpenJDK at your machine, if you could provide me an APK built there from the latest tag (just rename to *.zip so Github allows you to attach it), I could compare against that so we know if the JDK is the culprit.

[d4rken]

PS: As you use OpenJDK at your machine, if you could provide me an APK built there from the latest tag (just rename to *.zip so Github allows you to attach it), I could compare against that so we know if the JDK is the culprit.

That's what I provided in my last comment.

[IzzySoft]

Oops, missed the link, apologies! Trying it now…

[IzzySoft]

OK, apk diff:

  -rw-r--r--  0.0 unx      120 b-      117 defN 1981-01-01 01:01:02 1992c814 META-INF/version-control-info.textproto
- -rw-r--r--  0.0 unx     3792 b-     3792 stor 1981-01-01 01:01:02 8200f577 assets/dexopt/baseline.prof
+ -rw-r--r--  0.0 unx     3792 b-     3792 stor 1981-01-01 01:01:02 0994c57c assets/dexopt/baseline.prof
  -rw-r--r--  0.0 unx      400 b-      400 stor 1981-01-01 01:01:02 ba76b15e assets/dexopt/baseline.profm
- -rw-r--r--  0.0 unx  9708784 b-  3592343 defN 1981-01-01 01:01:02 813563b0 classes.dex
+ -rw-r--r--  0.0 unx  9708784 b-  3592340 defN 1981-01-01 01:01:02 1fa60085 classes.dex
  -rw-r--r--  0.0 unx   689416 b-   255925 defN 1981-01-01 01:01:02 e34588d4 classes2.dex

Dex diff:

 |: sput-object v1, Leu/darken/myperm/common/BuildConfigWrap;.FLAVOR:Leu/darken/myperm/common/BuildConfigWrap$Flavor;
-|: const-string v2, "2024-11-26T07:59:52.197522310Z"
+|: const-string v2, "2024-07-12T17:26:33.187777090Z"
 |: check-cast v2, Ljava/lang/CharSequence;

OK, that's the build timestamp – apart from that it seems to be RB. Just to be sure, let me update my sed to reflect the timestamp from your build and make another run… Yupp, confirmed, then it is RB.

So 2 questions here:

• is that build timestamp really needed, or could it be removed (or replaced by something that's deterministic)?
• as it seems we've identified the other culprit (the JDK), how do we want to deal with that? Will you try replacing adopt with e.g. temurin (which I saw quite often in workflows and cannot remember it having caused any issues) or OpenJDK? I guess always using your local builds would not be the "ideal option" 😉

[d4rken]

is that build timestamp really needed, or could it be removed (or replaced by something that's deterministic)?

That is mostly part of a default project setup, I don't think I have need here and could remove that.

as it seems we've identified the other culprit (the JDK), how do we want to deal with that? Will you try replacing adopt with e.g. temurin (which I saw quite often in workflows and cannot remember it having caused any issues) or OpenJDK? I guess always using your local builds would not be the "ideal option" 😉

I'm not deep into the different CI runner setups. I'm open to replacing adopt. Do you know what other large projects use? The adopt is likely from some sample/example config I used some time ago.

[IzzySoft]

That is mostly part of a default project setup, I don't think I have need here and could remove that.

Thanks!

Do you know what other large projects use?

PS: funny. Though they mention OpenJDK, it's not in the list of supported distributions. From what I've seen so far, I'd tend to zulu or temurin for a try with the Github workflow then – in that order (zulu is even described as "Azul Zulu OpenJDK", temurin is (to my knowledge) at least based on OpenJDK).

TL;DR: Go with zulu? That's what I've seen most frequently (at least my brain tells me that).

I've seen adopt a few times, and my brain associates that with "problematic". Brain says it cannot remember issues with zulu, and is almost sure the same could be said for temurin (though something nags of a one-time issue in the past). I don't remember ever having seen one of the others so far.

If you want me to check before you make the next release, we could go the same path again: you switch the workflow to zulu, have it produce an APK (from the last release is fine, we can deal with the timestamp separately – or if you want to fix both at the same time I simply download the APK and check for the embedded commit hash 😜) and I see if it is RB.

[d4rken]

Release build is being run:
https://github.com/d4rken-org/permission-pilot/actions/runs/12047569321

[IzzySoft]

Thanks! Build timestamp is gone. But strangely, still the same annotation stuff in classes.dex:

   VISIBILITY_RUNTIME Lkotlin/Metadata; d1={ "..M\n\484850\485148\484850\n..\n\484850\485148\484850\n..\n\484850\485148\484850\n\484850\485148\484850\n\484850\485048..\n\484850\b\484850\n\484850\485148\484850\n\484850\485048\484951\n\4848
-  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/lists/differ/AsyncDiffer$callback$1;" }
+  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/lists/differ/AsyncDiffer$callback$1<" "TA;TT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/util/List<" "TT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Landroidx/recyclerview/widget/AsyncListDiffer<" "TT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(TA;" "Lkotlin/jvm/functions/Function2<" "-TT;-TT;" "Ljava/lang/Boolean;" ">;" "Lkotlin/jvm/functions/Function2<" "-TT;-TT;" "Ljava/lang/Boolean;" ">;" "Lkotlin/jvm/functions/Fun
@@ -1360128,7 +1360128,7 @@
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Lkotlin/jvm/functions/Function1<" "TComponentT;TBindingT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Lkotlin/jvm/functions/Function1<" "TComponentT;" "Landroidx/lifecycle/LifecycleOwner;" ">;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "TComponentT;" }
-  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/viewbinding/ViewBindingProperty$onDestroyObserver$1;" }
+  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/viewbinding/ViewBindingProperty$onDestroyObserver$1<" "TComponentT;TBindingT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "TBindingT;" }

And as before: running with ubuntu:jammy instead of debian:bookworm-slim results in the very same APK. I doubt that ubuntu:noble (aka "latest" aka "24.04") will make a difference there, but let me try… nope. That's as close as I can come matching your Github workflow. I only have OpenJDK available from the packages, so I cannot match zulu (or any other variant at that).

Leaves us trying with temurin I guess? I'm otherwise clueless. I only run ./gradlew assembleFossRelease, as your workflow does. The only other remaining difference is that your workflow runs ./gradlew assembleFossBeta immediately before that. As there's no clean with the second run, it might use artifacts from there. So options would be:

• try temurin
• have the "Assemble production APK" use ./gradlew clean assembleFossRelease
• switch the task order and build FossRelease before FossBeta

Guess you can change (and run) the workflow without making a new release, optionally using some test branch for that? I gladly match that with the test runs here then, building from the release tag (or any commit you name). I'd try the last variant (switching task order) first but up to you.

[IzzySoft]

OK, I've kept a closer watch on the JDKs projects use when walking my backlog the past days. Almost all I had problems with used zulu (ouch), while most of what I looked at had temurin (and less issues). And I have a good guess that oracle probably means "Oracle OpenJDK" and thus would match exactly what we use here. So it seems my suggestion to try zulu first was not exactly the best. As the APK is RB when you build with OpenJDK on your machine, could we give a CI build with temurin or oracle another try, please?

Apologies for the trouble here, @d4rken – especially for the part caused by my misjudgement. I'm still learning here, too…

[d4rken]

I changed it to oracle.

https://github.com/d4rken-org/permission-pilot/actions/runs/12256138407/job/34190733682

[IzzySoft]

Strange, the DEX diff looks almost identical to that from before (size of the `*.diff file is only 2 bytes smaller this time) – as if there were additional annotations in there:

   VISIBILITY_RUNTIME Lkotlin/Metadata; d1={ "..M\n\484850\485148\484850\n..\n\484850\485148\484850\n..\n\484850\485148\484850\n\484850\485148\484850\n\484850\485048..\n\484850\b\484850\n\484850\485148\484850\n\484850\485048\484951\n\4848
-  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/lists/differ/AsyncDiffer$callback$1;" }
+  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/lists/differ/AsyncDiffer$callback$1<" "TA;TT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Ljava/util/List<" "TT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Landroidx/recyclerview/widget/AsyncListDiffer<" "TT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "(TA;" "Lkotlin/jvm/functions/Function2<" "-TT;-TT;" "Ljava/lang/Boolean;" ">;" "Lkotlin/jvm/functions/Function2<" "-TT;-TT;" "Ljava/lang/Boolean;" ">;" "Lkotlin/jvm/functions/Fun
@@ -1566302,7 +1566302,7 @@
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Lkotlin/jvm/functions/Function1<" "TComponentT;TBindingT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Lkotlin/jvm/functions/Function1<" "TComponentT;" "Landroidx/lifecycle/LifecycleOwner;" ">;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "TComponentT;" }
-  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/viewbinding/ViewBindingProperty$onDestroyObserver$1;" }
+  VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "Leu/darken/myperm/common/viewbinding/ViewBindingProperty$onDestroyObserver$1<" "TComponentT;TBindingT;>;" }
   VISIBILITY_SYSTEM Ldalvik/annotation/Signature; value={ "TBindingT;" }

If temurin gives the very same results as well, I'm out of ideas… Interesting, though, that it's always the same classes. What was different when RB succeeded with v1.7.0-rc0 on November 26th (eu.darken.myperm-v1.7.0-rc0-10700000-FOSS-RELEASE-c66a1cf.apk)? That was the only one that was RB (according to what I've kept here locally).

PS: Ah, that was one you've built locally…