Job Description
Job Description
- Design and implement scalable batch and streaming data pipelines using Azure Databricks, Delta Live Tables, and Apache Spark.
- Build and maintain ETL/ELT workflows orchestrated through Azure Data Factory and Databricks Workflows.
- Develop reusable and modular pipeline components following software engineering best practices.
- Architect and manage Lakehouse solutions using Delta Lake across Bronze, Silver, and Gold layers.
- Design and enforce data models, schemas, and governance policies using Unity Catalog.
- Optimize storage, partitioning, and query performance for large-scale datasets on ADLS Gen2.