Karpenter is taking longer time to fallback to lower weighted nodepool when ICE errors are hit

[bparamjeet]

Description

Observed Behavior:
Karpenter is taking too long to fallback to a lower-weighted node pool when ICE errors occur. Sudden increase in pod replica count leaves all pods in a pending state for over a period of time.

Expected Behavior:
Karpenter should fallback to lower weighted nodepool immediately when ICE errors occur.

Reproduction Steps (Please include YAML):

• Create multiple nodepools with multiple weights.
• Increase the replica count of a deployment to a larger number.
• Karpenter won't be able to create new nodes due to ICE errors and pods will be accumulated in pending state.
• Karpenter will show ICE errors over a period of time till it fallback

Versions:

• Karpenter Version: 1.0.5
• Kubernetes Version (kubectl version): v1.31

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Vacant2333]

Is a necessary condition for this problem to occur? karpenter will cache InstanceTypes that failed to create and will not try again for a few minutes @bparamjeet
In your tests, how long would it take for karpenter to fall back into the lower-weighted node pool?

[bparamjeet]

In your tests, how long would it take for karpenter to fall back into the lower-weighted node pool ?

• Karpenter did not fallback to the standby nodepools which is of lower weights.
• We intervened in between to mark the lower weights for c5.9x which then helped to create nodes.
• We have multiple nodepools with same weight c7i.8x, c7i.12x, c6i.8x, c6i.12x, c5.9x, c5.12x , After ICE'd error for c7i.8x,c6i.8x karpenter is not fallbacking to c5.9x, c5.12x . why it preferring to choose c7i, c6i ? why karpeter not provisioning instance in c5 when getting ice with c6i and c7i ?

[jonathan-innis]

IIRC, this issue was occurring because Karpenter was thinking that it could create a bunch of NodeClaims on the first couple NodePools that were using c6i and c7i and then just kept looping on those instance types over and over. More or less this is expected with highly constrianed NodePools because we think that we can launch the NodeClaims successfully but then, when we find out that we can't, we have to cleanup all of the NodeClaim decisions that we made due to ICE errors.

Removing these does take a little bit of time for requests to be sent to the apiserver after we get the ICE errors back and during that time we are delaying another scheduling loop. I think you may be only getting through a few of these constrained NodePools before looping back around to the beginning.

[jonathan-innis]

/triage needs-informatino

[k8s-ci-robot]

@jonathan-innis: The label(s) triage/needs-informatino cannot be applied, because the repository doesn't have them.

In response to this:

/triage needs-informatino

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jonathan-innis]

/triage needs-information

[jonathan-innis]

I tried this same configuration and actually didn't see the delays and issues with the ICE timeout that you were hitting -- I actually wonder if this has to do with an interaction with the batchIdleDuration and batchMaxDuration if you are setting custom values for these fields.