Add tolerations only to specific worker pods

[anxietymonger]

I am currently facing with a scenario, where I need to schedule worker0 pod onto specific nodes. Is there any possibility to configure custom tolerations or labels only to specific worker pods (like worker0). Thank you in advance for any information.

[tenzen-y]

No, the mpi-operator doesn't support configuring parameters for a specific worker.

[tenzen-y]

/kind question

[anxietymonger]

Thank you very much for your information. Closing the issue.

[alculquicondor]

Why is worker 0 special? In addition to the launcher already being "special".

[anxietymonger]

In my case, worker0 is special since rank0 need to access some resources that only exists on specific nodes. It is true that launcher is already special to some extents, but in my understanding, the launcher won't do any computation, right? BTW, is it possible to make rank0 running on the launcher?

[alculquicondor]

That is correct, the launcher just coordinates.

Having the workers in its own pods has the advantage that the resources can be exclusive to the worker computations, as opposed to be shared with launcher tasks. Nothing prohibits the launcher pod to be in the same node as other workers, but you have the isolation of the pod namespaces to have better control.

[alculquicondor]

I wonder how common is the specialization you mention is.

Would we need to add support for an arbitrary number of pod templates?

cc @ahg-g @danielvegamyhre

[alculquicondor]

/reopen

[google-oss-prow]

@alculquicondor: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

This was also discussed in #384