Job Description
About the Role
We are searching for a skilled Data Engineer to join our team. The ideal candidate will have hands‑on experience in development and production implementation with Azure Databricks and Spark, proficiency in Python (PySpark) and SQL coding, and a strong understanding of data lake architecture, ETL processes, and data quality management. Experience with Azure data analytics services and excellent technical writing skills are also required.
Responsibilities
- Develop and implement data processing pipelines using Databricks and Spark.
- Write efficient and optimized code in Python (PySpark) and SQL for TB-scale data processing.
- Design and maintain data lake architecture and ETL processes.
- Ensure data quality and integrity through robust data quality management practices.
- Utilize Azure data analytics services for data storage, processing, and analysis.
- Create comprehensive technical documenta...