Skip to content

Commit

Permalink
Improve CI test that prepares and test python client
Browse files Browse the repository at this point in the history
The test for python client require locally installed airflow with
some providers prepared from sources. We have been preparing the
selected packages and installing them, but sometimes this required
to add new package there.

To make it simpler, we can instead prepare and install all the
packages - this will be pretty fast with `uv` and we will avoid
to manually modify the CI tests every time we add some package
that requires installation.
  • Loading branch information
potiuk committed Nov 14, 2024
1 parent e7a0ddd commit 9138f5f
Showing 1 changed file with 19 additions and 11 deletions.
30 changes: 19 additions & 11 deletions .github/workflows/basic-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,6 @@ jobs:
- run: python -m pytest -n auto --color=yes
working-directory: ./dev/breeze/


tests-ui:
timeout-minutes: 10
name: React UI tests
Expand Down Expand Up @@ -178,6 +177,8 @@ jobs:
path: ./airflow-client-python
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: "Cleanup dist files"
run: rm -rfv ./dist/*
- name: "Generate client with breeze"
run: >
breeze release-management prepare-python-client --package-format both
Expand All @@ -187,25 +188,30 @@ jobs:
working-directory: ./airflow-client-python
- name: Install hatch
run: |
set -x
python -m pip install --upgrade uv
uv tool install hatch
- name: Run tests
run: hatch run run-coverage
env:
HATCH_ENV: "test"
working-directory: ./clients/python
- name: "Install source version of required packages"
- name: "Prepare provider packages"
run: >
breeze release-management prepare-provider-packages --package-format
wheel --skip-tag-check --version-suffix-for-pypi dev0
- name: "Prepare task.sdk package"
run: breeze release-management prepare-task-sdk-package --package-format wheel
- name: "Install all packages"
run: |
breeze release-management prepare-provider-packages fab standard common.sql --package-format \
wheel --skip-tag-check --version-suffix-for-pypi dev0
pip install . dist/apache_airflow_providers_fab-*.whl \
dist/apache_airflow_providers_standard-*.whl dist/apache_airflow_providers_common_sql-*.whl
breeze release-management prepare-task-sdk-package --package-format wheel
pip install ./dist/apache_airflow_task_sdk-*.whl
- name: "Install Python client"
run: pip install ./dist/apache_airflow_client-*.whl
set -x
apt-get update -y && apt-get upgrade -y && apt-get install -y libkrb5-dev gcc
uv venv
uv pip install . dist/*.whl
- name: "Initialize Airflow DB and start webserver"
run: |
. ./venv/bin/activate
set -x
airflow db init
# Let scheduler runs a few loops and get all DAG files from example DAGs serialized to DB
airflow scheduler --num-runs 100
Expand All @@ -225,7 +231,9 @@ jobs:
timeout 30 bash -c 'until nc -z $0 $1; do echo "sleeping"; sleep 1; done' localhost 8080
sleep 5
- name: "Run test python client"
run: python ./clients/python/test_python_client.py
run: |
. ./venv/bin/activate
python ./clients/python/test_python_client.py
env:
FORCE_COLOR: "standard"
- name: "Stop running webserver"
Expand Down

0 comments on commit 9138f5f

Please sign in to comment.