RFC for Degraded NodePool Status Condition

[jigisha620]

Description

Adding RFC for Degraded NodePool Status Condition.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jigisha620
Once this PR has been reviewed and has the lgtm label, please assign bwagner5 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @jigisha620. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coveralls]

Pull Request Test Coverage Report for Build 13298709565

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 9 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.06%) to 81.387%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/test/expectations/expectations.go	2	94.81%
pkg/controllers/provisioning/scheduling/preferences.go	7	86.52%

Totals
Change from base Build 13292671955:	-0.06%
Covered Lines:	9239
Relevant Lines:	11352

---

💛 - Coveralls

[jmdeal]

Checkpointing

[enxebre]

For instance, if a NodePool is restricted to a specific zone using the topology.kubernetes.io/zone label, but the specified zone is not accessible through the provided subnet configurations, this inconsistency shouldn't trigger a Degraded status.

Can we enumerate different semantics for failures that we'd want to capture as different .Reasons that should trigger degraded == true, e.g. badSecurityGroup

[jonathan-innis]

Major +1 to this -- I think what we need to explore here is how we are going to capture these failure modes -- if we are just relying on the registration timeout being hit, it's going to be tough to know what the reason was that the Node failed to join

[jonathan-innis]

Major +1 to this -- I think what we need to explore here is how we are going to capture these failure modes -- if we are just relying on the registration timeout being hit, it's going to be tough to know what the reason was that the Node failed to join

[jonathan-innis]

It might be worth calling out that cloudproviders should also try to introduce deterministic mechanisms for launch failures -- we can point to the AWS validation controller which does this auth validation and we can point to any other cloudproviders that might have similar mechanisms

[jonathan-innis]

For the example you provide, what should it trigger? Is this a validation error in the NodePool for the lack of instance types that we match against? Is this something that we currently check for?

[jonathan-innis]

Suggested change

2. False - NodePool has configuration issues that require customer investigation and resolution. Since Karpenter cannot automatically detect these specific launch or registration failures, we will document common failure scenarios and possible fixes in our troubleshooting guide to assist customers. The cause for the failure will also be surfaced through the status condition reason and message fields.

2. False - NodePool has configuration issues that require user investigation and resolution. Since Karpenter cannot automatically detect these specific launch or registration failures, we will document common failure scenarios and possible fixes in our troubleshooting guide to assist users. The cause for the failure will also be surfaced through the status condition reason and message fields.

[jonathan-innis]

Specifically, there has been a specific node registration using this unique combination of NodePool and NodeClass spec

[jonathan-innis]

I think we should mention that we should consider if it should be in the future -- like a potential follow-up here might be that we propose a cooldown for attempting this NodePool for some time if we see that we start getting failures

[jonathan-innis]

Put these under a "Goals" section since these are effectively the "goals" of the proposed design

[jonathan-innis]

I think if we really wanted to consider this one as an option, we'd have to implement the time-based consideration and probably something about the size of the cluster too -- like reset this to Unknown or something if we haven't launched after some long amount of time, weight newer launches more highly, etc.

[jonathan-innis]

So, I think this approach needs some downsides mentioned -- it is more complex in general and also doesn't really handle every scenario (it at least needs some more thought if it's actually going to be accurate) -- we effectively have to scale this with cluster size and we have to account for the fact that Falses are going to happen a bunch of times within a time window but Trues are going to happen a single time and then they won't fire again -- you basically have to account for that by weighting successes higher in some way

[jonathan-innis]

I think the observability benefits are really that you can hang monitoring for NodePools on this, you give users more insight into what's going on when NodeClaims are failing and you give us a path in the future for actual hanging NodePool selection logic on this concept