[SDK]Support Docker image as objective in the tune API #2338
Conversation
Signed-off-by: akhilsaivenkata <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hi @andreyvelich , I made the changes to katib_client.py. Do we have any test cases that need to be updated or implemented for this change? I also wanted to test these new changes on my local machine so I ran "make check" & "make test" commands on my local machine and there were no failures. I wonder if you could see and suggest any action items that needs to taken up here. |
There was a problem hiding this comment.
Thank you for doing this @akhilsaivenkata!
I left a few comments.
| parameters: Dict[str, Any], | ||
| base_image: str = constants.BASE_IMAGE_TENSORFLOW, | ||
| #base_image: str = constants.BASE_IMAGE_TENSORFLOW, |
There was a problem hiding this comment.
I think, we should keep the base_image, since we use it when user set objective as train function.
| @@ -400,12 +407,12 @@ def tune( | |||
| trial_template = models.V1beta1TrialTemplate( | |||
| primary_container_name=constants.DEFAULT_PRIMARY_CONTAINER_NAME, | |||
| retain=retain_trials, | |||
| trial_parameters=trial_params, | |||
| trial_parameters=trial_params if callable(objective) else [], | |||
There was a problem hiding this comment.
trial_parameters still be required even when user sets Docker image.
You can check example here: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/hp-tuning/random.yaml#L31-L36
There was a problem hiding this comment.
Sure @andreyvelich , I will revert this change and keep the trial_parameters.
There was a problem hiding this comment.
Also, I would like to know if we need to write new unit test cases or change existing ones ?
There was a problem hiding this comment.
@akhilsaivenkata Yes, since we merge this PR: #2325, please add unit test for tune function.
| input_params = {} | ||
| experiment_params = [] | ||
| trial_params = [] | ||
| base_image = constants.BASE_IMAGE_TENSORFLOW, |
There was a problem hiding this comment.
Yes @andreyvelich , i did keep the base_image here at this line, I have added it in the if block so this code change got mixed up with all the other lines
|
@akhilsaivenkata Please rebase your PR. |
| trial_spec=trial_spec, | ||
| ) | ||
|
|
||
| # Add parameters to the Katib Experiment. | ||
| experiment.spec.parameters = experiment_params | ||
| experiment.spec.parameters = experiment_params if callable(objective) else [] |
There was a problem hiding this comment.
I think parameters field is also needed since trial_paramaters is still required.
There was a problem hiding this comment.
You are right @Electronic-Waste , I have just reverted these two code changes and pushed it now.
Signed-off-by: akhilsaivenkata <[email protected]>
There was a problem hiding this comment.
It looks great. Thank you @akhilsaivenkata ! I left a question for you and @andreyvelich .
| command=["bash", "-c"] if callable(objective) else None, | ||
| args=[exec_script] if callable(objective) else None, |
There was a problem hiding this comment.
Also I'm not sure if we can assign None to command and args here when we use Docker image as objective.
As @andreyvelich shows an example for us, we sometimes need to pass command and args to the training container to execute python scripts with some parameters.
Could you explain your idea in details so that I can understand more? WDYT👀 @akhilsaivenkata @andreyvelich
There was a problem hiding this comment.
I think, initially we can just allow user to set image as objective without command and args.
Similar to how we allow create training job using base_image parameter: https://github.com/kubeflow/training-operator/blob/master/sdk/python/kubeflow/training/api/training_client.py#L327C35-L327C45.
|
Hi @akhilsaivenkata, did you get a chance to finish this PR ? |
What this PR does / why we need it: Supporting Docker image as an objective in the tune API.
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #2326
Checklist: