bug: (pkg/oci/auth) Importing the auth package causes "kubeconfig flag redefined" error

[vishal-chdhry]

Background

We at Kyverno were facing issues with the ACR keychain where there was a 30-second timeout when using an image from Azure ACR in a non-AKS cluster. To work around that problem we decided to use the auth pkg in this repo to create custom keychains. The implementation works fine, but there is a problem.

PR: kyverno/kyverno#7908

Description

After importing github.com/fluxcd/pkg/oci v0.30.0 in kyverno, we started getting an error when running the pod saying panic: /ko-app/kyverno flag redefined: kubeconfig . fluxcd/pkg/oci is the only new direct import in the PR. So I believe that is whats causing the issue here.

Any suggestion on how to fix it?

[makkes]

The flag is defined in an init function by controller-runtime here which is imported by various files in the oci package so it really is on controller-runtime to fix this. There exists an issue for this behaviour but it looks like they won't change this anytime soon so the only workaround is the one that you already implemented which is to check if the flag has already been defined and if it has, not re-define it.

[kingdonb]

We just had an interesting discussion about this at Bug Scrub, It looks like the controller-runtime are aware of this issue and I might have found this by clicking through @makkes links.

The working theory is that by importing fluxcd/pkg to get the oci package, we've changed the dependency order and moved controller-runtime up the import stack, and so this flag gets defined by controller-runtime in the updated version. Whereas it was getting defined by kyverno before and controller-runtime was observing the flag already defined, and avoiding redefining it. That explains it wasn't a problem in the prior order, because controller-runtime checks for existence of the flag, so kyverno just needs to do the same now based on the updated import paths with flux. @vishal-chdhry Thanks for the tour of this issue 👍

The redefinition is between controller-runtime and kyverno, it's a fun coincidence that fluxcd/pkg causes this, but the indirect dependency explanation appears to be the correct one.

[makkes]

controller-runtime registers the flag in the initialization phase using an init function while Kyverno does not. Therefore, controller-runtime's registration always happens first, according to Go's initialization procedure. There is no way for the Kyverno app to change that because dependencies are always initialized first.