Karpenter released the beta version of its APIs and features in October 2023. The intention behind this beta was that we would be able to determine the final set of changes and feature adds that we wanted to add to Karpenter before we considered Karpenter feature-complete. The list below details the features that Karpenter has on its roadmap before Karpenter becomes feature complete and stable at v1.
This list represents the minimal set of changes that are needed to ensure proper operational excellence, feature completeness, and stability by v1. For a change to make it on this list, it must meet one of the following criteria:
- Breaking: The feature requires changes or removals from the API that would be considered breaking after a bump to v1
- Stability: The feature ensures proper operational excellence for behavior that is leaky or has race conditions in the beta state
- Planned Deprecations: The feature cleans-up deprecations that were previously planned the project
- v1 APIs
- Update
karpenter.sh/disruption=disrupting:NoScheduletaint tokarpenter.sh/disrupted:NoSchedule - Add the
karpenter.sh/unregistered:NoExecutetaint on registration - Wait for Instance Termination on NodeClaim/Node Deletion
- Drift Hash Breaking Change Handling
- Introduce ConsolidateAfter for Consolidation Controls
- Change Expiration from Graceful to Forceful
- Define SemVer Versioning Policy for
kubernetes-sigs/karpenterLibrary - Stabilize Observability (metrics, status, eventing)
- NodeClaim Conceptual Documentation
- Drop Knative References from the Code
- Migrate Knative Webhook away from Karpenter
- Karpenter Global Logging Configuration Changes
- Promoting Drift Feature to Stable
Issue Ref(s): #758, aws/karpenter-provider-aws#5006
Category: Breaking, Stability
For Karpenter to be considered v1, the CustomResources that are shipped with an installation of the project also need to be stable at v1. Changes to Karpenter’s API (including labels, annotations, and tags) in v1 are detailed in Karpenter v1 API. The migration path for these changes will ensure that customers will not have to roll their nodes or manually convert their resources as they did at v1beta1. Instead, we will leverage Kubernetes conversion webhooks to automatically convert their resources to the new schema format in code. The API groups and Kind naming will remain unchanged.
Category: Breaking
Karpenter currently uses the karpenter.sh/disruption key with the disrupting value to signal when nodes are actively being disrupted by the disruption controller. This key was originally added with a value because it was assumed that we would expand upon this taint with additional values over time. As part of the v1 stability, we now realize that this behavior is inconsistent with the usage of taint keys in Kubernetes generally. Rather than keeping the karpenter.sh/disruption key and differentiating based on values, we will be creating different taint keys for each unique action that Karpenter takes (e.g. karpenter.sh/disrupted and karpenter.sh/unregistered) This update also changes the taint key from karpenter.sh/disrupting to karpenter.sh/disrupted.
- Update the
karpenter.sh/disruption=disrupting:NoScheduletaint to bekarpenter.sh/disrupted:NoSchedule
Category: Breaking
Karpenter currently adds taints, labels, and annotations to nodes from the NodeClaim when nodes register with the cluster. This is done without restricting pods from scheduling to the node which means that for any labels or taints that aren't present on the node on startup, this process or registration can break scheduling compatability.
For instance, a pod may not tolerate a specific taint that was configured in the NodeClaim template, but the taint did not appear on the node immediately on startup, meaning that the pod was able to successfully schedule when it shouldn't have been able to.
To mitigate this, Karpenter is introducing the karpenter.sh/unregistered:NoExecute taint at v1. Nodes registering with Karpenter will be required to have this taint present on their nodes when the node first joins. Karpenter will propagate lables, annotations, and taints, and remove this karpenter.sh/unregistered taint simultaneously, allowing Karpenter to ensure that pods will not schedule against the node before it has had a chance to add configuration.
- Add
karpenter.sh/unregistered:NoExecutetaint into the cloudprovider bootstrap config (Kubelet's--register-with-taintsparameter) - Remove the
karpenter.sh/unregistered:NoExecutetaint as part of the NodeClaim lifecycle registration flow - Validate the
karpenter.sh/unregistered:NoExecutetaint exists for nodes owned by Karpenter
Category: Stability
Karpenter currently leaks DaemonSet pods and leases when it is terminating instances. This occurs because Karpenter currently initiates a Delete() operation once on the CloudProvider but does not continually check that the instance is fully terminated. For AWS and Azure, terminating the instance simply starts a shutdown procedure for the instance, meaning that kubelet can continue to reach out to the node until the instance is fully shutdown and terminated.
Because Karpenter is not waiting, the node fails to properly deregister, leaking daemonsets and node leases onto the cluster. By waiting for instance termination before fully deleting the node, we are allowing the node to go through its graceful shutdown process.
- Implement a retry mechanism to ensure that the instance is fully terminated before removing the Node and the NodeClaim from the cluster
- Validate and remove the current lease garbage collection controller from
kubernetes-sigs/karpenter, removing the permission on node leases in thekube-node-leasenamespace
Issue Ref(s): #957
Category: Stability
Karpenter currently relies on a hash to determine whether certain fields in Karpenter’s NodeClaims have drifted from their owning NodePool and owning EC2NodeClass. Today, this is determined by hashing a set of fields on the NodePool or EC2NodeClass and then validating that this hash still matches the NodeClaim’s hash.
This hashing mechanism works well for additive changes to the API, but does not work well when adding fields to the hashing function that already have a set value on a customer’s cluster. In particular, we have a need to make breaking changes to this hash scheme from these two issues: #909 and aws/karpenter-provider-aws#5447.
We need to implement a common way for handing breaking changes to our hashing logic ahead of v1 so we can make the requisite changes that we need to make to the v1 APIs at v1 as well as handle breaking changes through defaults to our drift hashing API moving forward post-v1 e.g. we introduce an alpha field that affects our static hashing and then drop it later.
- Design and implement a hash version-style implementation that allows us to make breaking changes to the hash versioning scheme
Issue Ref(s): #735
Category: Breaking, Stability
Karpenter currently evaluates all forms of disruption in a synchronous loop, starting with expiration, drift, emptiness, and considering consolidation and multi-node consolidation only if the other conditions are not satisfied. Consolidation performs scheduling simulations on the cluster and evaluates if there are any opportunities to save money by removing nodes or replacing nodes on the cluster.
The current consolidateAfter behavior creates a consolidation decision, waits for 15s synchronously inside of the disruption loop, and then re-validates that the same decision is still valid. This was intended to address the concern from users that consolidation was acting too aggressively, and consolidating nodes that, if kept around for a little longer, would be valid for pods to schedule to. Adding the synchronous wait had the desired effect, but it caused us to have to keep this value low since it blocks all forms of disruption while it is waiting.
Users have asked that we make this a configurable field so that they can tweak whether we keep nodes around for longer when they are underutilized before we make a consolidation decision to terminate them. To do this, we will have to change our synchronous waiting mechanism to some other mechanism that will allow us to perform longer waits.
- Design and implement a
spec.consolidateAfterfield for the v1 NodePool API, reworking our synchronous wait to ensure that waiting for nodes that haven’t reached the end of theirconsolidateAftertimeframe doesn’t block other disruption evaluation
Issue/RFC Ref: #1303
Category: Breaking
Users care about the removal of their nodes within a certain timeframe due to security requirements. Budgets, karpenter.sh/do-not-disrupt pods and blocking PDBs should not stop us from starting the drain of the node after it exceeds its expireAfter value. Once a node has begun evicting, it will remain on the cluster up to its terminationGracePeriod duration (assuming there are blocking PDBs or pods that are running with the karpenter.sh/do-not-disrupt annotation) and then be removed.
- Remove expiration from the graceful disruption controller
- Perform a direct NodeClaim deletion when a Node has exceeded its expireAfter lifetime value
Category: Stability
Karpenter currently deeply couples the versioning of the kubernetes-sigs/karpenter library releases with the versioning of the aws/karpenter-provider-aws image and chart releases. This means that when we release a v0.33.2 version of the Karpenter AWS image and chart, we also tag the kubernetes-sigs/karpenter library with the same version. Realistically, this is not sustainable long-term since other contributors and cloudproviders will begin to take a heavier reliance on this project’s libraries.
We need to define a mechanism for communicating breaking changes to the library to projects that rely on it. This library is similar to something like controller-runtime , where the library is versioned independent of the projects that rely on it. Starting in v1, we should adopt a versioning scheme similar to this, that decouples the AWS release version from the neutral library version.
- Create a design doc for defining a versioning strategy for
kubernetes-sigs/karpenterand cloud provider repos. Begin adhering to this strategy starting in v1
Issue Ref: #1051
Category: Breaking, Stability
Karpenter needs to stabalize a set of metrics, status conditions, and events that can be relied-upon for monitoring. The design for these metrics, status conditions and events will be added in a separate RFC.
Issue Ref: aws/karpenter-provider-aws#5144
Category: Stability
Karpenter currently has no conceptual documentation around NodeClaims. NodeClaims have become a fundamental part of how Karpenter launches and manages nodes. There is critical observability information that is stored inside of the NodeClaim that can help users understand when certain disruption conditions are met (Expired, Empty, Drifted) or why the NodeClaim fails to launch.
For Karpenter’s feature completeness at v1, we need to accurately describe to users what the purpose of Karpenter’s NodeClaims are and how to leverage the information that is stored within the NodeClaim to troubleshoot Karpenter’s decision-making.
- Add a NodeClaim doc to the “Concepts” section of the documentation
Issue Ref: #332
Category: Stability
Karpenter has used knative from the beginning of the project. knative’s pkg libraries were only intended for their own use and were not intended to be used widely by the community. Because knative relies on and generates so much of the upstream API, attempting to bump to a newer version of client-go (or any other upstream k8s package) without having knative pkg pinned to that same version causes incompatibilities. Practically, this means that we can be bottlenecked on older versions of k8s libraries while we are waiting on knative to update its own dependencies. Knative has a slower release cycle than Karpenter so we need to avoid these bottlenecks while we have the opportunity to make breaking changes to the API.
- Remove the knative logger and replace with the controller-runtime logger
- Update the status condition schema for Karpenter CustomResources to use the metav1 status condition schema
Issue Ref: #332
Category: Stability
As part of Karpenter completely removing its dependency on Knative, Karpenter needs to remove its coupling on the knative webhook certificate reconciliation logic. Currently, Karpenter leverages knative’s webhook reconciler to reconcile a certificate needed to enable the TLS webhook traffic. As part of removing this dependency that kubernetes-sigs/karpenter has on the knative webhook reconciliation logic, Karpenter will be dropping webhooks entirely.
Because K8s versions 1.25+ have support for CustomResourceValidations driven through Common Expression Language, Karpenter is dropping support for the webhook container entirely.
Azure/karpenter-provider-azure dropped support for Kubernetes 1.25 since AKS dropped support for this version in 2023. aws/karpenter-provider-aws still supports versions all the way back to Kubernetes 1.23 due to extended support. The AWS provider will continue to support versions prior to 1.25 until January 31, 2025. At this point, if not other providers support versions prior to 1.25, we will drop support for webhooks in kubernetes-sigs/karpenter.
- Drop the webhook, removing the
knative/pkgdependency from the Karpenter controller package
Issue Ref(s): aws/karpenter-provider-aws#5352
Category: Planned Deprecations, Breaking
Dropping our global logging configuration was a planned deprecation at v1beta1 and we will continue by fully dropping support for the ConfigMap-based configuration for our logging at v1.
- Remove logging configuration (only allow LOG_LEVEL, potentially LOG_ENCODING if users request it)
Category: Stability
Karpenter supported drift in its alpha state from v0.21-v0.32. During alpha, we worked to build out features and promoted drift to beta after releasing full drift support for all Karpenter configuration. Since releasing drift in beta, we’ve received no feedback that would lead us to believe the feature is unstable or not the right direction for the declarative state of Karpenter.
Since the feature is such a fundamental part to how the declarative state of Karpenter functions, we will promote drift to stable at Karpenter v1.
- Remove
Driftas a feature flag, always enabling it in Karpenter