Adds timeouts for reconciler contexts #7147

plkokanov · 2022-12-05T08:19:53Z

How to categorize this PR?

/area scalability
/kind enhancement

What this PR does / why we need it:
This PR introduces a timeout for contexts used by reconciler functions.

Which issue(s) this PR fixes:
Part of #4251

Special notes for your reviewer:
As the issue also mentions adding waits for cache syncs - this comes out of the box when using controller runtime with a default timeout of 2 minutes.
The shoot and seed care controllers were already using contexts that timeout for the health check functions:

gardener/pkg/gardenlet/controller/shoot/shoot_care_control.go

Lines 233 to 234 in a60a42f

    
           careCtx, cancel := context.WithTimeout(ctx, r.config.Controllers.ShootCare.SyncPeriod.Duration) 
        
           defer cancel()

and

gardener/pkg/gardenlet/controller/seed/care/reconciler.go

Lines 61 to 62 in a60a42f

    
           ctx, cancel := context.WithTimeout(reconcileCtx, r.Config.SyncPeriod.Duration) 
        
           defer cancel()

I was not sure whether we should keep those as they are, or unify them so that only one time context is used.

Release note:

The `controllermanager` and `gardenlet` controller reconciliations are now limited to a `1m` timeout. Additionally, there is a 1m limit on predicate functions that use contexts.

gardener-prow · 2022-12-05T08:19:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

rfranzke · 2022-12-05T16:33:44Z

/assign
/cc @timebertt

pkg/controllermanager/controller/bastion/reconciler.go

pkg/controllermanager/controller/seed/backupbucketscheck/reconciler.go

pkg/gardenlet/controller/seed/care/reconciler.go

plkokanov · 2022-12-09T09:43:21Z

@timebertt thanks for the review! There was something else that I thought would be good to add - timeout contexts for some of our mapper and predicate functions that also inject the parent context (stop channel) from the manager, as they could potentially block as well, if there are calls to the api server.

plkokanov · 2023-01-16T08:13:58Z

@timebertt @rfranzke, this should be ready for review now. I've still used just a single context for the backupsbucketcheck controller:

gardener/pkg/controllermanager/controller/seed/backupbucketscheck/reconciler.go

Lines 47 to 48 in 9b294c5

    
           ctx, cancel := context.WithTimeout(ctx, r.Config.SyncPeriod.Duration) 
        
           defer cancel()

and the extensionscheck reconciler:

gardener/pkg/controllermanager/controller/seed/extensionscheck/reconciler.go

Lines 54 to 55 in 9b294c5

    
           ctx, cancel := context.WithTimeout(ctx, r.Config.SyncPeriod.Duration) 
        
           defer cancel()

because those checks are actually pretty fast and small, so I think that only one context is enough there. However, if you feel that's incorrect I will apply the initial suggestion to separate them in two contexts: #7147 (comment)

Edit: oops, forgot about also adapting the tests that use mocks. Fixing them now ...
Edit2: unit tests should be fixed now.

…controllers

pkg/controllerutils/miscellaneous.go

pkg/resourcemanager/controller/health/progressing/reconciler.go

… controllers in controllermanager - Make `ShootQuotaControllerConfiguration` a pointer - Make `SyncPeriod` pointer. - Default `SyncPeriod` to 60 minutes.

pkg/controllerutils/miscellaneous.go

pkg/gardenlet/controller/bastion/reconciler.go

pkg/gardenlet/controller/seed/seed/reconciler.go

pkg/gardenlet/controller/shoot/shoot/reconciler.go

… now.

plkokanov · 2023-03-28T12:45:44Z

Thanks for addressing my comments. Leaving it to @timebertt if he wants to take one more look.

timebertt

One remaining comment, otherwise lgtm

pkg/controllerutils/miscellaneous.go

timebertt

/lgtm
/approve

gardener-prow · 2023-03-29T06:28:57Z

LGTM label has been added.

Git tree hash: 68635569718b7c4f98a87cf0e02fa42b179092d2

gardener-prow · 2023-03-29T06:29:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: timebertt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [timebertt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Add timeout for reconciler contexts * Adapts unit tests * Fixes integration tests * Use separate context for garden client and seed client * Address PR review feedback * Fix failing test * Address PR review feedback * Introduce utility function to have consistent contextTimeouts in all controllers * [PR feedback] Revert back the changes. * Address PR review feedback * Streamline resourcemanager controller to use context with timeouts * Streamline ShootQuotaControllerConfiguration struct inline with other controllers in controllermanager - Make `ShootQuotaControllerConfiguration` a pointer - Make `SyncPeriod` pointer. - Default `SyncPeriod` to 60 minutes. * Remove timeout from context in case of seed and shoot reconcilers for now. * Address PR review feedback --------- Co-authored-by: Ashish Ranjan Yadav <[email protected]>

gardener-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 5, 2022

gardener-prow bot added area/scalability Scalability related kind/enhancement Enhancement, improvement, extension cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. labels Dec 5, 2022

gardener-prow bot requested review from shafeeqes and timuthy December 5, 2022 08:20

gardener-prow bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 5, 2022

plkokanov force-pushed the enh/add-context branch 3 times, most recently from 3cffc9a to a8cd66a Compare December 5, 2022 08:21

rfranzke mentioned this pull request Dec 5, 2022

Refactor Gardener components to native controller-runtime components #4251

Closed

76 tasks

plkokanov marked this pull request as ready for review December 5, 2022 09:20

gardener-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 5, 2022

gardener-prow bot requested a review from acumino December 5, 2022 09:20

plkokanov marked this pull request as draft December 5, 2022 09:38

gardener-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 5, 2022

gardener-prow bot assigned rfranzke Dec 5, 2022

gardener-prow bot requested a review from timebertt December 5, 2022 16:33

timebertt reviewed Dec 9, 2022

View reviewed changes

pkg/controllermanager/controller/bastion/reconciler.go Outdated Show resolved Hide resolved

pkg/controllermanager/controller/seed/backupbucketscheck/reconciler.go Outdated Show resolved Hide resolved

pkg/gardenlet/controller/seed/care/reconciler.go Show resolved Hide resolved

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 16, 2022

plkokanov force-pushed the enh/add-context branch from a8cd66a to 9b294c5 Compare January 16, 2023 08:09

gardener-prow bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 16, 2023

plkokanov marked this pull request as ready for review January 16, 2023 08:10

gardener-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 16, 2023

gardener-prow bot requested review from ary1992 and oliver-goetz January 16, 2023 08:10

plkokanov and others added 8 commits March 27, 2023 18:59

Fixes integration tests

2d49b2c

Use separate context for garden client and seed client

9aacc1c

Address PR review feedback

01473b2

Fix failing test

978a24e

Address PR review feedback

1a37d8e

Introduce utility function to have consistent contextTimeouts in all …

fc16b8a

…controllers

[PR feedback] Revert back the changes.

172bc78

Address PR review feedback

4a8d485

ary1992 force-pushed the enh/add-context branch from 0502c77 to 4a8d485 Compare March 27, 2023 13:44

gardener-prow bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 27, 2023

timebertt reviewed Mar 27, 2023

View reviewed changes

pkg/controllerutils/miscellaneous.go Show resolved Hide resolved

pkg/resourcemanager/controller/health/progressing/reconciler.go Outdated Show resolved Hide resolved

Streamline resourcemanager controller to use context with timeouts

71f62ab

gardener-prow bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 28, 2023

Streamline ShootQuotaControllerConfiguration struct inline with other…

cabb357

… controllers in controllermanager - Make `ShootQuotaControllerConfiguration` a pointer - Make `SyncPeriod` pointer. - Default `SyncPeriod` to 60 minutes.

plkokanov commented Mar 28, 2023

View reviewed changes

ary1992 added 2 commits March 28, 2023 16:08

Remove timeout from context in case of seed and shoot reconcilers for…

0ee5b2e

… now.

Address PR review feedback

24db52a

gardener-prow bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 28, 2023

timebertt reviewed Mar 28, 2023

View reviewed changes

pkg/controllerutils/miscellaneous.go Show resolved Hide resolved

timebertt reviewed Mar 29, 2023

View reviewed changes

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Mar 29, 2023

gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 29, 2023

gardener-prow bot merged commit dff43d9 into gardener:master Mar 29, 2023

plkokanov deleted the enh/add-context branch March 30, 2023 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds timeouts for reconciler contexts #7147

Adds timeouts for reconciler contexts #7147

plkokanov commented Dec 5, 2022 •

edited

Loading

gardener-prow bot commented Dec 5, 2022

rfranzke commented Dec 5, 2022

plkokanov commented Dec 9, 2022 •

edited

Loading

plkokanov commented Jan 16, 2023 •

edited

Loading

plkokanov commented Mar 28, 2023

timebertt left a comment

timebertt left a comment

gardener-prow bot commented Mar 29, 2023

gardener-prow bot commented Mar 29, 2023

	careCtx, cancel := context.WithTimeout(ctx, r.config.Controllers.ShootCare.SyncPeriod.Duration)
	defer cancel()

	ctx, cancel := context.WithTimeout(reconcileCtx, r.Config.SyncPeriod.Duration)
	defer cancel()

Adds timeouts for reconciler contexts #7147

Adds timeouts for reconciler contexts #7147

Conversation

plkokanov commented Dec 5, 2022 • edited Loading

gardener-prow bot commented Dec 5, 2022

rfranzke commented Dec 5, 2022

plkokanov commented Dec 9, 2022 • edited Loading

plkokanov commented Jan 16, 2023 • edited Loading

plkokanov commented Mar 28, 2023

timebertt left a comment

Choose a reason for hiding this comment

timebertt left a comment

Choose a reason for hiding this comment

gardener-prow bot commented Mar 29, 2023

gardener-prow bot commented Mar 29, 2023

plkokanov commented Dec 5, 2022 •

edited

Loading

plkokanov commented Dec 9, 2022 •

edited

Loading

plkokanov commented Jan 16, 2023 •

edited

Loading