You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I already tested nccl test with four pods then i tried to make component and pipeline with same configurations.
But i faced with some attribute issue and type issue during executing pipeline generated below python code.
I already checked kubeflow document and it said container component support only image, command and args.
If so how to modify below code to apply existing environment of pods?
from kfp import dsl
from kfp.dsl import ContainerSpec, OutputPath
from kubernetes import client as k8s_client
Define your component using dsl.container_component decorator
@dsl.pipeline(
name='nccl test Pipeline',
description='A pipeline that deploys a master node and multiple worker nodes for distributed training'
)
def nccl_test_pipeline():
work_node1 = create_component(
image='docker.io/library/pytorch_nccl_master:local',
node='atc01',
name='work-node1' # Argument for clarity, not used in KFP v2
)
work_node2 = create_component(
image='docker.io/library/pytorch_nccl_work:local',
node='atc01',
name='work-node2' # Argument for clarity, not used in KFP v2
)
work_node3 = create_component(
image='docker.io/library/pytorch_nccl_work:local',
node='atc02',
name='work-node3' # Argument for clarity, not used in KFP v2
)
work_node4 = create_component(
image='docker.io/library/pytorch_nccl_work:local',
node='atc02',
name='work-node4' # Argument for clarity, not used in KFP v2
)
Compile the pipeline
if name == 'main':
import kfp.compiler as compiler
compiler.Compiler().compile(
pipeline_func=nccl_test_pipeline,
package_path='nccl_test_new_local.yaml'
)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I already tested nccl test with four pods then i tried to make component and pipeline with same configurations.
But i faced with some attribute issue and type issue during executing pipeline generated below python code.
I already checked kubeflow document and it said container component support only image, command and args.
If so how to modify below code to apply existing environment of pods?
from kfp import dsl
from kfp.dsl import ContainerSpec, OutputPath
from kubernetes import client as k8s_client
Define your component using dsl.container_component decorator
@dsl.container_component
def create_component(image: str, node: str, name: str): # Keep argument for clarity
Define your pipeline
@dsl.pipeline(
name='nccl test Pipeline',
description='A pipeline that deploys a master node and multiple worker nodes for distributed training'
)
def nccl_test_pipeline():
work_node1 = create_component(
image='docker.io/library/pytorch_nccl_master:local',
node='atc01',
name='work-node1' # Argument for clarity, not used in KFP v2
)
Compile the pipeline
if name == 'main':
import kfp.compiler as compiler
compiler.Compiler().compile(
pipeline_func=nccl_test_pipeline,
package_path='nccl_test_new_local.yaml'
)
Beta Was this translation helpful? Give feedback.
All reactions