Job Description
The Senior Data Engineer is responsible for designing, developing, and maintaining scalable data pipelines and cloud-based solutions that power our internal systems and business intelligence capabilities. This role sits at the intersection of data engineering, backend development, and cloud architecture, working closely with cross-functional teams across Sales, Accounting, and Operations.
Key Responsibilities
- Design, develop, and maintain end-to-end data pipelines on Databricks (Unity Catalog, Delta Lake, PySpark).
- Architect and implement cloud solutions on Azure: Container Apps, Container Registry, Blob Storage, Azure SQL, and Azure Functions.
- Develop APIs and backend services using Python (FastAPI) for integration of internal and external systems.
- Optimize pipeline performance: reduce runtimes, implement checkpoint strategies, and enable parallel processing.
- Model data, design schemas, and manage relational databas...