Azure Databricks Professional Services

The province of Ontario is seeking a qualified vendor to provide professional services for Azure Databricks. The selected supplier will be responsible for architecting, planning, implementing, testing, and delivering two specific use cases on the existing Azure Databricks platform.

The scope of work includes conducting pre-assessment and planning tasks, such as reviewing existing documentation, interviewing stakeholders, assessing current requirements, and identifying success factors. The vendor must also document non-functional security requirements, integration constraints, and cost optimization strategies, while outlining the necessary implementation and testing approaches.

Key project activities involve implementing a new Lakehouse architecture utilizing Delta Lake, Unity Catalog, and modern data ingestion pipelines. This includes strategies for managing large volumes of semi-structured data, such as automating the ingestion of millions of small XML files. Additionally, the migration of Python and R-based workloads will require reviewing best practices for Databricks workspace deployment, cluster management, and deep exploration of core Databricks concepts including Delta Lake, Delta Live Tables, Autoloader, and Unity Catalog. The design and deployment should account for ongoing cost optimization, governance, resource allocation, monitoring, and future integration within the Microsoft data ecosystem.