Milestone 1: end of Jan. Github Milestone
Feature Branch: feat/e2e-fabric-dataops-sample

Fabric functionality considerations: Features needs to be in Public Preview by Ignite to be considered in this milestone.

E2E Sample "Business Problem"

• Business Problem: visual parking availability in the city. "Parking analytics".
  • Utilize existing Parking Sensor to keep as is similar to existing Databricks sample
• Persona: Contoso City Council
• Batch only. Land into storage, then shortcut main point of ingestion.
• Decide on the name of the sample. Propose: parking_sensor_fabric

User Stories:

• Create V1 Data Pipeline Architecture Diagram of medallion data pipeline #788
• Update Solution Overview in README #789
• Create CI/CD overview diagram #790

IaC

• TF scripts utilizing the Fabric TF Provider
• Overall Deployment script ./deploy.sh
  • Dev, Test, Prod environment deployment
  • Seed data, bootstrapping
• Follow from IaC for Everything recommendations
• Document any known issues such as non-automatable tasks such as lack of SP support for some APIs

User Stories:

• [Fabric e2e sample] Add Terraform scripts #759
• Environment Bootstrapping (seed data) script #798
• Incorporate Data Generator from Brian #799

Data Pipelines

• Medallion style data pipeline

User Stories:

• Create Fabric notebook for data transform step #837
• Create Fabric notebook for data standardization step #836
• Create required assets for ingestion process #842
• Create Fabric notebook for setup steps #843
• Build Fabric Data Pipeline #801
• Determine final schema for the dataset + CELA approval #848
• Update Lakehouse and Warehouse table creation notebooks to align with schema changes #849
• Update Python .py to align with schema changes #850
• Update PowerBI dashboard to align with schema changes #851

CI/CD

• PR validation (basic linting, unit tests)
• CI pipeline (basic build - whl package generation)
• Release pipeline
• Document any known issues such as non-automatable tasks.

Known issues:

• Forking workspaces doesn't sync Lakehouse. Similar to shortcuts
• Data manage between dev workspaces

User Stories:

• Create Basic AzDO PR Validation pipeline #802
• Create Basic AzDO CI pipeline #803
• Create Baseline Release Pipeline #804
• Add AzDO pipelines to sample deployment script #805
• Create Variable Groups #943

Automated Testing

• Unit Testing (Python packages)

User Stories:

• Include unit testing in CI pipeline #809

Observability

• Data Quality monitoring with great expectations (sink to AppInsights)

User Stories:

• Add Data Quality monitoring in notebooks #947

Dev Tooling, Environment

Common Dev environment, CICD, Developer onboarding guide for the sample.

• Setup common CI/CD environment (Azure DevOps) #796
• Setup common "integration" environment: Azure Subscription, Fabric tenant #797
• Create Devcontainer with appropriate linters and markdown link checkers. #794
• Developer onboarding guide V1 #795