Job Description
Design and implement scalable batch and streaming data pipelines using Azure Databricks, Delta Live Tables, and Apache Spark. Build and maintain ETL/ELT workflows orchestrated through Azure Data Factory and Databricks Workflows. Develop reusable and modular pipeline components following software engineering best practices. Architect and manage Lakehouse solutions using Delta Lake across Bronze, Silver, and Gold layers. Design and enforce data models, schemas, and governance policies using Unity Catalog. Optimize storage, partitioning, and query performance for large-scale datasets on ADLS Gen2. Manage Databricks clusters, compute policies, and job scheduling. Implement Infrastructure as Code (IaC) using Terraform or ARM templates. Integrate Databricks with Azure services including Synapse, Event Hubs, Key Vault, and Azure DevOps. Requirements Strong proficiency in PySpark, Python, and SQL for Big Data processing. Hands-on experience with Delta Lake, Delta Live Tables (DLT), and Medalli...