Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Errors with experimental "Control Loop" feature enabled #234

Open
svenfoo opened this issue Feb 24, 2025 · 1 comment
Open

Errors with experimental "Control Loop" feature enabled #234

svenfoo opened this issue Feb 24, 2025 · 1 comment
Labels
defect Suspected defect such as a bug or regression

Comments

@svenfoo
Copy link

svenfoo commented Feb 24, 2025 •
edited
Loading

What version were you using?

  • NATS server v2.10.25
  • nack chart v0.28.0
  • JetStream controller v0.17.1

What environment was the server running in?

nack is configured with jetstream.enabled: true and jetstream.additionalArgs: ["--control-loop"]

Is this defect reproducible?

Yes, the nack pod keeps crashing. It then gets restarted, and it actually does its job of monitoring and creating / updating the configured resources, but eventually it will run into the error and crash.

Given the capability you are leveraging, describe your expectation?

I would expect the nack pod to keep runnning, ideally without any warnings or errors in the logs.

I am aware though that we are using a new feature here that is marked as experimental. Still you might be interested in getting issues reported so that the new feature can be stabilized.

Given the expectation, what is the defect you are observing?

I am observing some errors and warnings in the logs of the nack pod and eventually, after a few minutes, it will run into an error and crash. As explained above this is reproducible. The pod gets restarted, and it actually does what I would expect it to do until it runs into the same error again.

Here's a condensed version of the log, I have removed some excessive warnings that seem unrelated to the problem described here:

W0224 12:10:26.758879       1 main.go:96] Starting JetStream controller in experimental control loop mode
I0224 12:10:26.761520       1 main.go:213] starting manager
I0224 12:10:26.761826       1 server.go:208] "Starting metrics server" logger="controller-runtime.metrics"
I0224 12:10:26.762173       1 controller.go:198] "Starting EventSource" controller="consumer" controllerGroup="jetstream.nats.io" controllerKind="Consumer" source="kind source: *v1beta2.Consumer"
I0224 12:10:26.762193       1 controller.go:198] "Starting EventSource" controller="account" controllerGroup="jetstream.nats.io" controllerKind="Account" source="kind source: *v1beta2.Account"
I0224 12:10:26.762280       1 controller.go:198] "Starting EventSource" controller="keyvalue" controllerGroup="jetstream.nats.io" controllerKind="KeyValue" source="kind source: *v1beta2.KeyValue"
I0224 12:10:26.762685       1 controller.go:198] "Starting EventSource" controller="stream" controllerGroup="jetstream.nats.io" controllerKind="Stream" source="kind source: *v1beta2.Stream"
I0224 12:10:26.763104       1 server.go:247] "Serving metrics server" logger="controller-runtime.metrics" bindAddress=":8080" secure=false
I0224 12:10:26.763207       1 controller.go:198] "Starting EventSource" controller="objectstore" controllerGroup="jetstream.nats.io" controllerKind="ObjectStore" source="kind source: *v1beta2.ObjectStore"
E0224 12:10:26.774482       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:10:26.776383       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
I0224 12:10:26.872004       1 controller.go:233] "Starting Controller" controller="account" controllerGroup="jetstream.nats.io" controllerKind="Account"
I0224 12:10:26.872080       1 controller.go:242] "Starting workers" controller="account" controllerGroup="jetstream.nats.io" controllerKind="Account" worker count=1
I0224 12:10:26.875342       1 controller.go:233] "Starting Controller" controller="consumer" controllerGroup="jetstream.nats.io" controllerKind="Consumer"
I0224 12:10:26.875389       1 controller.go:242] "Starting workers" controller="consumer" controllerGroup="jetstream.nats.io" controllerKind="Consumer" worker count=1
I0224 12:10:26.876646       1 controller.go:233] "Starting Controller" controller="stream" controllerGroup="jetstream.nats.io" controllerKind="Stream"
I0224 12:10:26.876694       1 controller.go:242] "Starting workers" controller="stream" controllerGroup="jetstream.nats.io" controllerKind="Stream" worker count=1
E0224 12:10:36.765364       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:10:36.767580       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:10:46.765533       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:10:46.767107       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:10:56.765504       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:10:56.767372       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:11:06.768629       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:11:06.770937       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:11:16.766167       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:11:16.768575       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:11:26.767386       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:11:26.769942       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:11:36.765867       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:11:36.767514       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:11:46.765441       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:11:46.766830       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:11:56.765295       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:11:56.767610       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:12:06.765979       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:12:06.768490       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:12:16.765967       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"ObjectStore\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="ObjectStore.jetstream.nats.io"
E0224 12:12:16.768021       1 kind.go:71] "if kind is a CRD, it should be installed before calling Start" err="no matches for kind \"KeyValue\" in version \"jetstream.nats.io/v1beta2\"" logger="controller-runtime.source.EventHandler" kind="KeyValue.jetstream.nats.io"
E0224 12:12:26.762580       1 controller.go:210] "Could not wait for Cache to sync" err="failed to wait for keyvalue caches to sync kind source: *v1beta2.KeyValue: timed out waiting for cache to be synced for Kind *v1beta2.KeyValue" controller="keyvalue" controllerGroup="jetstream.nats.io" controllerKind="KeyValue" source="kind source: *v1beta2.KeyValue"
I0224 12:12:26.762684       1 internal.go:538] "Stopping and waiting for non leader election runnables"
I0224 12:12:26.762709       1 internal.go:542] "Stopping and waiting for leader election runnables"
I0224 12:12:26.762738       1 controller.go:262] "Shutdown signal received, waiting for all workers to finish" controller="stream" controllerGroup="jetstream.nats.io" controllerKind="Stream"
I0224 12:12:26.762752       1 controller.go:262] "Shutdown signal received, waiting for all workers to finish" controller="consumer" controllerGroup="jetstream.nats.io" controllerKind="Consumer"
I0224 12:12:26.762777       1 controller.go:262] "Shutdown signal received, waiting for all workers to finish" controller="account" controllerGroup="jetstream.nats.io" controllerKind="Account"
I0224 12:12:26.762810       1 controller.go:264] "All workers finished" controller="account" controllerGroup="jetstream.nats.io" controllerKind="Account"
I0224 12:12:26.762836       1 controller.go:264] "All workers finished" controller="consumer" controllerGroup="jetstream.nats.io" controllerKind="Consumer"
I0224 12:12:26.762838       1 controller.go:264] "All workers finished" controller="stream" controllerGroup="jetstream.nats.io" controllerKind="Stream"
I0224 12:12:26.762888       1 controller.go:233] "Starting Controller" controller="objectstore" controllerGroup="jetstream.nats.io" controllerKind="ObjectStore"
I0224 12:12:26.762936       1 controller.go:242] "Starting workers" controller="objectstore" controllerGroup="jetstream.nats.io" controllerKind="ObjectStore" worker count=1
I0224 12:12:26.762968       1 controller.go:262] "Shutdown signal received, waiting for all workers to finish" controller="objectstore" controllerGroup="jetstream.nats.io" controllerKind="ObjectStore"
I0224 12:12:26.763000       1 controller.go:264] "All workers finished" controller="objectstore" controllerGroup="jetstream.nats.io" controllerKind="ObjectStore"
I0224 12:12:26.763027       1 internal.go:550] "Stopping and waiting for caches"
I0224 12:12:26.763272       1 internal.go:554] "Stopping and waiting for webhooks"
I0224 12:12:26.763309       1 internal.go:557] "Stopping and waiting for HTTP servers"
I0224 12:12:26.763361       1 server.go:254] "Shutting down metrics server with timeout of 1 minute" logger="controller-runtime.metrics"
I0224 12:12:26.763894       1 internal.go:561] "Wait completed, proceeding to shutdown the manager"
Error: failed to wait for keyvalue caches to sync kind source: *v1beta2.KeyValue: timed out waiting for cache to be synced for Kind *v1beta2.KeyValue

The logs seem to indicate that CRDs might be missing. Note that we are using the CRDs that come bundled with the nack v0.28.0 chart.
@svenfoo svenfoo added the defect Suspected defect such as a bug or regression label Feb 24, 2025
@svenfoo
Copy link
Author

svenfoo commented Feb 24, 2025

Okay, I tried again on a fresh installation and could not reproduce this error. So it looks like the upgrade of the nack chart to v0.28.0 did not cause the CRDs to be updated to the ones that come bundled with the chart.

I am leaving the issue open for now so that it gets some attention. There seems to be a problem when upgrading nack that should be addressed. Maybe the apiVersion of the CRDs should be bumped?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@svenfoo