Databricks+ADF E2E Sample (Milestone 1)

[ydaponte]

Milestone 1: mid-Jan.

E2E Sample "Technical Problem"

• Technical Problem: Fix the ability for the sample to be deployed e2e and modernize sample with latest features from Databricks and ADF
• Persona: Data Engineers and Developers

IaC

• Fixing sample devcontainer
• Generate data that is independent from the public REST API (it can be re-used by the other E2E sample) and allowing batch and streaming modes to ingest the data
• Fixing bicep scripts that are failing with the latest versions and newer schemas (Synapse SQL Dedicated pool)
• Fixing broken main ./deploy.sh script when configuring the Databricks workspace for dev, stg and prod
• Upgrade to the latest Databricks CLI as the previous version is no longer working for certain parts of the code
• Incorporate the deployment of a Unity Catalog
• Fixing Key Vault Soft-Delete capability in the clean-up and deployment scripts
• Act on issues identified on the ISE Checklist (Other fundamentals)
• Document all changes, including architecture schemas when applicable
• [27.11.2014]. As we have more people contributing, we added some more improvements to the milestone as for example readme and guidance improvements, upgrading bicep versions used in the sample and replacing sql authentication by only entra id authentication.

CI/CD

• Incorporate Asset Bundles to the CICD Databricks process
• Act on issues identified on the ISE Checklist (Other fundamentals)
• Document all changes

Security

• Start and act on the ISE Checklist on the Security pillar
• Document all changes
• Change SQL authentication by Entra only authentication

Observability

• Act on issues identified on the ISE Checklist (Observability)
• Document all changes, including architecture schemas when applicable

Testing

• Document all changes, including architecture schemas when applicable

Python Package modernization

For Package development, update to latest / best practices. Align with Fabric update team on python package modernization.

VNext Milestone ideas:

• Adding Autoloader to ingestion
• Adding Industry packages to the sample
• Add a dependency map of the versions across the sample

---

Ensure you've read the Contributing Guide (Microsoft Internal only.)

IaC

• Fixing main issues: Fix main deployment issues #817
  • Fix Dev container: Failing devcontainer setup in parking sensor sample due to invalid jdk version #644
  • Fix Dev container: Devcontainer in parking sensors sample changes file ownership in linux/wsl  #646
  • Fix bicep scripts: Fixing Dedicated Synapse pool deployment #778
  • Fix main deployment script (ADB workspace): Parking Sensor (Databricks) deployment fails due to change in behaviour of workspace startup #630
  • Upgrade Databricks CLI: Upgrade Databricks CLI #763
• Fix clean-up script: Fix clean-up script #819
  • Key-vault soft delete: Key vault soft delete handle #596
  • Delete based on deployment_id: Consider deployment_id in clean-up script #821
• Fix pipeline problems
  • Fix ADF pipelines - temporary solution before Create Alternative to public REST API #764 is ready: Temporary workaround for REST API broken links #852
  • Fix release pipelines: Fixing release pipelines #864
• Remove REST API dependency: Create Alternative to public REST API #764
• Incorporate Unity Catalog: Automate deployment of Unity Catalog #985
  Added on 27.11.2024 - as we have more contributors joining:
• Fixing ReadMe, guidance and implementing usage and functionality improvements:
  • Fix IMG folder dependency: Create independent IMG folder for parking sensor sample #831
  • Upgrade bicep templates: Upgrading bicep templates #886
  • Remove CC SKU default reference: Change Confidential Compute reference #885
  • Reduce deployment manual steps: Decrease manual steps for deploy #879
  • Review GitHub Enterprise guidance: Review GitHub Enterprise /environment deployments guidance #898
  • AzDO pipeline exists checks prior deployment: Check if azdo pipeline exists and override #881
  • SQL dedicated pool pause automated script: Start and Stop SQL Synapse script #882
  • Review GitHub Enterprise guidance: Review GitHub Enterprise /environment deployments guidance #898

CICD

• Incorporate Asset Bundles: Modify Databricks CICD approach to use Asset Bundles #766
• ISE Fundamentals Checklist: ISE Engineering Fundamentals Checklist #841 - postponed
• Automate manual publish in ADF: Automate publish improvement in ADF #897

Security

• ISE Security Checklist: ISE Security Checklist #826 - ongoing, will continue on M2
• Adding Keyvault secrets fail when value starts with an "-" hífen: KeyVault Set command fails when pwd starts with "-" #823
• Review complexity in random generated password: Review SQL Server password complexity in random string #824
• Change PAT token guidance to new granular version: Change PAT Token to new granular version #889
• Replace SQL Auth for Entra Auth only: Replace SQL authentication by Entra ID only authentication #887

Observability

• ISE Observability Checklist: ISE Observability Checklist #839 - postponed

Testing

• [POSTPONED to V2 ] Incorporate DFTF on ADF pipelines: Add Unit Testing Framework tests to ADF pipelines #832 - - postponed to the M2 (as it is depending on the UC)