Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ODH] Split up the notebook-e2e-tests to seperated and grouped e2e-tests #51557

Closed
wants to merge 1 commit into from
Closed

[ODH] Split up the notebook-e2e-tests to seperated and grouped e2e-tests #51557

wants to merge 1 commit into from

Conversation

atheo89
Copy link
Contributor

@atheo89 atheo89 commented May 1, 2024 •
edited
Loading

This PR aims to improve testing efficiency and reduce testing time.

To archive that we had to change a bit the ocp-ci configuration.

Currently, all notebook tests are grouped under a single test suite called "notebooks-e2e-tests". However, this approach has several drawbacks:

  • Execution time is prolonged.
  • Failure of a single notebook test necessitates a full restart.
  • Tests often fail due to timeouts and resource shortages.
  • Testing unnecessary notebooks due to lack of selective testing.

To address these issues, we propose breaking down the unified test suite into separate tests, triggered only when relevant changes occur using the run_if_changed: (regex) option, this way will reduce the time and will spot easier problematic notebooks.

So the notebooks-e2e-tests broke into the following:

- notebooks-ubi8-e2e-tests
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi8-python-3.8\/*)|(jupyter\/minimal\/ubi8-python-3.8\/*)|(jupyter\/datascience\/ubi8-python-3.8\/*)|(jupyter\/pytorch\/ubi8-python-3.8\/*)|(jupyter\/tensorflow\/ubi8-python-3.8\/*)|(jupyter\/trustyai\/ubi8-python-3.8\/*)

- notebooks-ubi9-e2e-tests
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi9-python-3.9\/*)|(jupyter\/minimal\/ubi9-python-3.9\/*)|(jupyter\/datascience\/ubi9-python-3.9\/*)|(jupyter\/pytorch\/ubi9-python-3.9\/*)|(jupyter\/tensorflow\/ubi9-python-3.9\/*)|(jupyter\/trustyai\/ubi9-python-3.9\/*)

- codeserver-notebook-e2e-tests
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi9-python-3.9\/*)|(codeserver\/ubi9-python-3.9\/*)

- rstudio-notebook-e2e-tests
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/c9s-python-3.9\/*)|(rstudio\/c9s-python-3.9\/*)

- runtimes-ubi8-e2e-test
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi8-python-3.8\/*)|(runtimes\/datascience\/ubi8-python-3.8\/*)|(runtimes\/pytorch\/ubi8-python-3.8\/*)|(runtimes\/tensorlow\/ubi8-python-3.8\/*)

- runtimes-ubi9-e2e-test
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi9-python-3.9\/*)|(runtimes\/datascience\/ubi9-python-3.9\/*)|(runtimes\/pytorch\/ubi9-python-3.9\/*)|(runtimes\/tensorlow\/ubi9-python-3.9\/*)

- intel-runtimes-e2e-tests
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi9-python-3.9\/*)|(intel\/base\/gpu\/ubi9-python-3.9\/*)|(intel\/runtimes\/tensorflow\/ubi9-python-3.9\/*)|(jupyter\/intel\/tensorflow\/ubi9-python-3.9\/*)||(intel\/runtimes\/pytorch\/ubi9-python-3.9\/*)|(jupyter\/intel\/pytorch\/ubi9-python-3.9\/*)|(intel\/runtimes\/ml\/ubi9-python-3.9\/*)|(jupyter\/intel\/ml\/ubi9-python-3.9\/*)

- intel-notebooks-e2e-tests
Triggered when changes occur in the following repository folders:
run_if_changed: (base\/ubi9-python-3.9\/*)|(intel\/base\/gpu\/ubi9-python-3.9\/*)|(intel\/runtimes\/tensorflow\/ubi9-python-3.9\/*)|(jupyter\/intel\/tensorflow\/ubi9-python-3.9\/*)||(intel\/runtimes\/pytorch\/ubi9-python-3.9\/*)|(jupyter\/intel\/pytorch\/ubi9-python-3.9\/*)|(intel\/runtimes\/ml\/ubi9-python-3.9\/*)|(jupyter\/intel\/ml\/ubi9-python-3.9\/*)

PS: The habana-notebooks-e2e-tests was already separated

@atheo89
Split up the notebook-e2e-tests to seperated and grouped e2e-tests to…
2c69d1f 
… enchance the testing time and avoid of using a lot of resources
@openshift-ci openshift-ci bot requested review from harshad16 and rkpattnaik780 May 1, 2024 15:12
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented May 1, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: atheo89

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 1, 2024
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@atheo89: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-opendatahub-io-notebooks-main-codeserver-notebook-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-intel-notebooks-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-intel-runtimes-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-notebooks-ubi8-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-rstudio-notebook-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-runtimes-ubi8-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed
pull-ci-opendatahub-io-notebooks-main-runtimes-ubi9-e2e-tests opendatahub-io/notebooks presubmit Presubmit changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@atheo89
Copy link
Contributor Author

atheo89 commented May 1, 2024

/pj-rehearse

@openshift-ci-robot
Copy link
Contributor

@atheo89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented May 1, 2024

@atheo89: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/opendatahub-io/notebooks/main/intel-runtimes-e2e-tests 2c69d1f link unknown /pj-rehearse pull-ci-opendatahub-io-notebooks-main-intel-runtimes-e2e-tests
ci/rehearse/opendatahub-io/notebooks/main/codeserver-notebook-e2e-tests 2c69d1f link unknown /pj-rehearse pull-ci-opendatahub-io-notebooks-main-codeserver-notebook-e2e-tests
ci/rehearse/opendatahub-io/notebooks/main/intel-notebooks-e2e-tests 2c69d1f link unknown /pj-rehearse pull-ci-opendatahub-io-notebooks-main-intel-notebooks-e2e-tests
ci/rehearse/opendatahub-io/notebooks/main/notebooks-ubi9-e2e-tests 2c69d1f link unknown /pj-rehearse pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests
ci/rehearse/opendatahub-io/notebooks/main/notebooks-ubi8-e2e-tests 2c69d1f link unknown /pj-rehearse pull-ci-opendatahub-io-notebooks-main-notebooks-ubi8-e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

harshad16
harshad16 reviewed May 1, 2024
View reviewed changes
Copy link
Member

@harshad16 harshad16 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super neat work.
as we are spliting the bits based on the type.
shall we extract anaconda e2e separate ?
WDYT?

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2024
@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@atheo89
Copy link
Contributor Author

atheo89 commented May 9, 2024

/pj-rehearse pull-ci-opendatahub-io-notebooks-main-notebooks-ubi9-e2e-tests

@openshift-ci-robot
Copy link
Contributor

@atheo89: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

@atheo89, pj-rehearse: unable prepare a candidate for rehearsal; rehearsals will not be run. This could be due to a branch that needs to be rebased. ERROR:

couldn't rebase candidate onto master: %!w(<nil>)

@openshift-bot
Copy link
Contributor

Issues in openshift/release go stale after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 15d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 8, 2024
@atheo89
Copy link
Contributor Author

atheo89 commented Jun 10, 2024 •
edited
Loading

as we are spliting the bits based on the type.
shall we extract anaconda e2e separate ?
WDYT?

Thanks, you are right would be better to seperate the anacoda test as well, I have conside the change on the new PR that create to eliminate the mess around! The rest remain the same.

New PR here: #53010

@atheo89 atheo89 closed this Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harshad16 harshad16 harshad16 left review comments

@rkpattnaik780 rkpattnaik780 Awaiting requested review from rkpattnaik780

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@atheo89 @openshift-ci-robot @openshift-merge-robot @openshift-bot @harshad16