Provide deadlines or cancelations for all uses of context.Context

[devigned]

/kind feature

Describe the solution you'd like
An iteration of a reconcile loop should be bounded by a default deadline. In other words, a single execution of reconcile in a controller should not be able to run indefinitely.

As of today, there are several code paths where context.TODO() is used for a context. This context provides no ability to cancel or bound execution on those code paths. Some of these code paths will need to have a context piped in through functions up the call stack, and may have more widespread impact to function signatures.

Examples:

• cluster-api-provider-azure/exp/cloud/scope/machinepool.go

  Lines 151 to 152 in 4d682a1

  	// TODO[dj]: any function we are building where we are not adding context to the signature is welcoming unbound execution times. This needs to be addressed.
  	return m.patchHelper.Patch(context.TODO(), m.AzureMachinePool)

• cluster-api-provider-azure/cloud/scope/machine.go

  Line 214 in 4afe3c2

  return m.patchHelper.Patch(context.TODO(), m.AzureMachine)

• cluster-api-provider-azure/cloud/scope/cluster.go

  Line 162 in 4778ca5

  return s.patchHelper.Patch(context.TODO(), s.AzureCluster)

• cluster-api-provider-azure/controllers/azuremachine_controller.go

  Line 75 in 4afe3c2

  ctx := context.TODO()

TODOs:

• determine a default max deadline for a single reconcile
• provide a manager cli argument for default max deadline
• remove usage of context.TODO and replace with bounded contexts

See Also

• Support global timeouts for reconcilers controller-runtime#798
• Reconcile methods should take a context and logger controller-runtime#801

[CecileRobertMichon]

/priority important-soon

[devigned]

/assign