Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: Add OpenLineage to system tests in Google provider #46244

Merged
merged 1 commit into from
Feb 1, 2025

Conversation

kacpermuda
Copy link
Contributor

@kacpermuda kacpermuda commented Jan 29, 2025

This PR adds checking OpenLineage events produced by selected Google Provider operators when running system checks. This is an addition to unit tests already checking OpenLineage for specific operators.

Thanks to VariableTransport introduced in #43643, we can easily check if OpenLineage events produced by any operator match the expected values. The logic is simple, for each system tests: adding the OpenLineageTestOperator to the DAG and providing it with json file with what we expect to be in the events. Whenever a system test is run, we will verify OpenLineage events emitted.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jan 29, 2025
@kacpermuda kacpermuda force-pushed the feat-ol-add-google-system-tests branch from 8ea0fd4 to 1cd354c Compare January 30, 2025 12:15
@kacpermuda kacpermuda marked this pull request as ready for review January 30, 2025 12:29
@kacpermuda kacpermuda force-pushed the feat-ol-add-google-system-tests branch from 1cd354c to b583752 Compare January 30, 2025 13:11
@potiuk
Copy link
Member

potiuk commented Jan 31, 2025

cc: @VladaZakharova - is that ok with you ?

@potiuk potiuk merged commit 1fd4b88 into apache:main Feb 1, 2025
60 checks passed
@potiuk
Copy link
Member

potiuk commented Feb 1, 2025

I merge it so that we can make provider move - but we can always remove / update if needed.

amoghrajesh pushed a commit to astronomer/airflow that referenced this pull request Feb 3, 2025
@kacpermuda kacpermuda deleted the feat-ol-add-google-system-tests branch February 3, 2025 07:07
dabla pushed a commit to dabla/airflow that referenced this pull request Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants