KEP-2170: Implement runtime framework #2248

tenzen-y · 2024-09-04T17:21:35Z

What this PR does / why we need it:
Brief Design: https://docs.google.com/presentation/d/1HyEsBa7hxWpIoBXaX6uECiB48FWB85SG1kx15mO8hug/edit#slide=id.g30596bfee76_0_202

I implemented the runtime framework interfaces.
The responsibilities are the following:

/runtime.v2/core: This contains the actual Kubeflow Job Pipeline like TrainigRuntime (not CRD), which is an internal concept.
These pipelines build objects or create reconcile builders. We will add some pipelines in the future like SingleHostTrainingRuntime.
/runtime.v2/framework: This contains the Kubeflow Job Pipeline Framework, which has some extension points in the following, and we will add some extension points in the future.
- WatchExtensionPlugin
- EnforcePodGroupPolicyPlugin
- EnforceMLPolicyPlugin
- CustomValidationPlugin
- ComponentBuilderPlugin
/runtime.v2/framework/plugins: This contains the Kubeflow Job Pipeline Framework plugins, which implement the Framework extension points. Each of these plugins is performed in Kubeflow Job Pipeline Framework extension points.
- coscheduleing
- jobset (Under development)
- mpi
- plainml
- torch

Additionally, I did not implement all plugins. So, I will open an issue and delegate plugin implementation contributors who are interested in this project.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #
Part-of #2290

Checklist:

Docs included if any changes are user facing

coveralls · 2024-09-06T17:54:11Z

Pull Request Test Coverage Report for Build 11372731768

Details

9 of 9 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 11330381194:	0.0%
Covered Lines:	73
Relevant Lines:	73

💛 - Coveralls

Makefile

andreyvelich · 2024-09-10T17:39:14Z

go.mod

 	sigs.k8s.io/controller-runtime v0.17.3
 	sigs.k8s.io/jobset v0.5.2
+	sigs.k8s.io/kueue v0.6.3


Why do we need kueue dependency ?

This dependency came from

training-operator/pkg/runtime.v2/runtime.go

Line 125 in 22da8af

PodRequests: kueuelr.TotalRequests(&spec.podSpec),

.

This allows us to set the appropriate required resources for PodGroup. If we remove this dependency, we need to just copy Kueue's "TotalRequests" function here. I believe that just coping and pasting is not the ideal way.

It would be great to reduce the dependency. Maybe copy and paste is okay in this case as long as we provide a reference to the original source

That's not simple implementation. That's so complex, multiple files and lines codes.
So, I would propose keeping it here and then (after kube 1.32) switching to the kube library as I mentioned in #2280.

Yes, we discussed with @tenzen-y offline that after k/k separates this utility function, we will remove dependency on Kueue.

Signed-off-by: Yuki Iwai <[email protected]>

…indexes for the TrainJobs Signed-off-by: Yuki Iwai <[email protected]>

Signed-off-by: Yuki Iwai <[email protected]>

andreyvelich · 2024-10-16T20:05:34Z

pkg/runtime.v2/runtime.go

+	options := defaultOptions
+	for _, opt := range opts {
+		opt(&options)
+	}


Do we need default options for Info object ?

Suggested change

options := defaultOptions

for _, opt := range opts {

opt(&options)

}

options := InfoOptions{}

for _, opt := range opts {

opt(&options)

}

Do you recommend this?

Maybe, I am trying to understand how are your planning to use InfoOptions in other parts ?
@tenzen-y What are the differences between Info{} and InfoOptions{} ?

The infoOptions is object to set up the Info object.
This approach allows us to dynamically specify the parameters to the Info.

When we get rid of the InfoOptions, we need to specify all parameters everytime or need to directly pass the Info object.

This approach is well-known to avid the following function:

// Even if all parameters is not used, all parameters should be specified. func Foo(paramA string, paramB int, paramC int32, paramD bool, paramE int64)

// After introduced `infoOptions`. func Foo(params ...InfoOption)

I see, so maybe let's name it as InfoOption, not defaultOptions to make it clearer, and since we are not going to have default values for Info object.

we are not going to have default values for Info object.

Actually, this is the default info Option. Here, this means that the default is an empty struct.
So, the below indicates to initialize options as a default values (currently default has empty fields)

options := defaultOptions

In the future, we need to consider if we should the default parameters to infoOption. For example common labels and annotations.

I see, that makes sense.

andreyvelich · 2024-10-16T20:08:16Z

pkg/runtime.v2/framework/plugins/plainml/plainml.go

+	for rName := range info.TotalRequests {
+		info.TotalRequests[rName] = runtime.TotalResourceRequest{
+			Replicas:    numNodes,
+			PodRequests: info.TotalRequests[rName].PodRequests,
+		}
+	}


How are we using this while enforcing the MLPolicy ?

~~~~ if info == nil || info.MLPolicy != nil { ~~~~

I wanted to implement this in line 43. Maybe I failed to reabase.
Let me fix this.

NVM above comments.

spec: mlPolicy: numNodes: 1

We can imagine this situation.

I see, so we override the value that we set here, right ?

training-operator/pkg/runtime.v2/runtime.go

Lines 120 to 125 in 0c376d3

for _, spec := range options.podSpecReplicas {

info.TotalRequests[spec.name] = TotalResourceRequest{

Replicas: spec.replicas,

// TODO: Need to address LimitRange and RuntimeClass.

PodRequests: kueuelr.TotalRequests(&spec.podSpec),

}

Yes, this plainml codes try to update with the proper one.

andreyvelich · 2024-10-16T20:58:58Z

pkg/runtime.v2/framework/plugins/coscheduling/indexer.go

+)
+
+var (
+	TrainingRuntimeContainerRuntimeClassKey        = ".trainingRuntimeSpec.jobSetTemplateSpec.replicatedJobs.podTemplateSpec.runtimeClassName"


Should the indexer be equal to the Golang struct or json name ?
E.g. the trainingRuntimeSpec is named as .spec

training-operator/pkg/apis/kubeflow.org/v2alpha1/trainingruntime_types.go

Line 77 in 22d5dde

Spec TrainingRuntimeSpec `json:"spec,omitempty"`

We can specify the arbitrary key name. But the key is global within the training-ooerator.

andreyvelich · 2024-10-16T21:03:14Z

We should be ready to merge this.
Thank you for this great work @tenzen-y!
/lgtm
/assign @terrytangyuan @johnugeorge @kannon92

google-oss-prow · 2024-10-16T21:03:17Z

@andreyvelich: GitHub didn't allow me to assign the following users: kannon92.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

We should be ready to merge this.
Thank you for this great work @tenzen-y!
/lgtm
/assign @terrytangyuan @johnugeorge @kannon92

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kannon92

I don't really have context to review this at the moment.

I'll leave that to kubeflow members.

tenzen-y · 2024-10-17T13:49:30Z

We should be ready to merge this. Thank you for this great work @tenzen-y! /lgtm /assign @terrytangyuan @johnugeorge @kannon92

Thanks for the review!

andreyvelich · 2024-10-17T13:51:22Z

/approve

google-oss-prow · 2024-10-17T13:51:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

johnugeorge · 2024-10-17T14:13:53Z

/hold cancel

google-oss-prow bot added do-not-merge/work-in-progress size/XXL labels Sep 4, 2024

google-oss-prow bot requested review from jinchihe and kuizhiqing September 4, 2024 17:21

tenzen-y force-pushed the second-implementation-for-traininig-v2 branch 6 times, most recently from 92b1dd1 to 4195338 Compare September 6, 2024 17:49

tenzen-y force-pushed the second-implementation-for-traininig-v2 branch 18 times, most recently from d220851 to caa8564 Compare September 10, 2024 17:40

andreyvelich reviewed Sep 10, 2024

View reviewed changes

tenzen-y added 4 commits October 15, 2024 08:44

Remove grep dependency

d64cb34

Signed-off-by: Yuki Iwai <[email protected]>

KEP-2170: Implement ValidateObjects interface to the runtime framework

a0cae9a

Signed-off-by: Yuki Iwai <[email protected]>

KEP-2170: Expose the TrainingRuntime and ClusterTrainingRuntime Kind

a527f86

Signed-off-by: Yuki Iwai <[email protected]>

KEP-2170: Remove unneeded scheme field from the internal TrainingRuntime

2aaae2b

Signed-off-by: Yuki Iwai <[email protected]>

tenzen-y force-pushed the second-implementation-for-traininig-v2 branch from fd7aab4 to 2aaae2b Compare October 14, 2024 23:46

tenzen-y added 7 commits October 16, 2024 22:25

Rephrase the error message

1c88c85

Signed-off-by: Yuki Iwai <[email protected]>

Distinguish TrainingRuntime and ClusterTrainingRuntime when creating …

1c94ede

…indexes for the TrainJobs Signed-off-by: Yuki Iwai <[email protected]>

Propagate the TrainJob labels and annotations to the JobSet

338cf6e

Signed-off-by: Yuki Iwai <[email protected]>

Remove PodAnnotations from the runtime info

498ff28

Signed-off-by: Yuki Iwai <[email protected]>

Implement TrainingRuntime ReplicatedJob validation

ba81beb

Signed-off-by: Yuki Iwai <[email protected]>

Add TODO comments

2f616e5

Signed-off-by: Yuki Iwai <[email protected]>

Replace queueSuspendedTrainJob with queueSuspendedTrainJobs

0c376d3

Signed-off-by: Yuki Iwai <[email protected]>

andreyvelich reviewed Oct 16, 2024

View reviewed changes

google-oss-prow bot assigned johnugeorge and terrytangyuan Oct 16, 2024

google-oss-prow bot assigned andreyvelich Oct 16, 2024

google-oss-prow bot added the lgtm label Oct 16, 2024

kannon92 reviewed Oct 16, 2024

View reviewed changes

google-oss-prow bot added the approved label Oct 17, 2024

google-oss-prow bot removed the do-not-merge/hold label Oct 17, 2024

google-oss-prow bot merged commit 62a058d into kubeflow:master Oct 17, 2024
40 checks passed

tenzen-y deleted the second-implementation-for-traininig-v2 branch October 17, 2024 18:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-2170: Implement runtime framework #2248

KEP-2170: Implement runtime framework #2248

tenzen-y commented Sep 4, 2024 •

edited

Loading

coveralls commented Sep 6, 2024 •

edited

Loading

andreyvelich Sep 10, 2024

tenzen-y Sep 10, 2024

terrytangyuan Oct 12, 2024

tenzen-y Oct 14, 2024

andreyvelich Oct 14, 2024

andreyvelich Oct 16, 2024

tenzen-y Oct 16, 2024

andreyvelich Oct 16, 2024

tenzen-y Oct 16, 2024

tenzen-y Oct 16, 2024

andreyvelich Oct 16, 2024

tenzen-y Oct 16, 2024

tenzen-y Oct 16, 2024 •

edited

Loading

andreyvelich Oct 16, 2024

andreyvelich Oct 16, 2024

tenzen-y Oct 16, 2024 •

edited

Loading

tenzen-y Oct 16, 2024

tenzen-y Oct 16, 2024

andreyvelich Oct 16, 2024

tenzen-y Oct 16, 2024

andreyvelich Oct 16, 2024

tenzen-y Oct 16, 2024

andreyvelich commented Oct 16, 2024

google-oss-prow bot commented Oct 16, 2024

kannon92 left a comment

tenzen-y commented Oct 17, 2024

andreyvelich commented Oct 17, 2024

google-oss-prow bot commented Oct 17, 2024

johnugeorge commented Oct 17, 2024

	for _, spec := range options.podSpecReplicas {
	info.TotalRequests[spec.name] = TotalResourceRequest{
	Replicas: spec.replicas,
	// TODO: Need to address LimitRange and RuntimeClass.
	PodRequests: kueuelr.TotalRequests(&spec.podSpec),
	}

KEP-2170: Implement runtime framework #2248

KEP-2170: Implement runtime framework #2248

Conversation

tenzen-y commented Sep 4, 2024 • edited Loading

coveralls commented Sep 6, 2024 • edited Loading

Pull Request Test Coverage Report for Build 11372731768

Details

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich commented Oct 16, 2024

google-oss-prow bot commented Oct 16, 2024

kannon92 left a comment

Choose a reason for hiding this comment

tenzen-y commented Oct 17, 2024

andreyvelich commented Oct 17, 2024

google-oss-prow bot commented Oct 17, 2024

johnugeorge commented Oct 17, 2024

tenzen-y commented Sep 4, 2024 •

edited

Loading

coveralls commented Sep 6, 2024 •

edited

Loading

tenzen-y Oct 16, 2024 •

edited

Loading

tenzen-y Oct 16, 2024 •

edited

Loading