Job Description
Streaming & Real-Time Data Processing
- Design and develop real-time streaming pipelines using Databricks Structured Streaming.
- Build and maintain Kafka-based ingestion frameworks.
- Handle late-arriving events using watermarks, event-time processing, and stateful streaming concepts.
- Implement exactly-once processing and checkpointing mechanisms.
- Monitor and optimize streaming workloads for performance and reliability.
Spark Declarative Pipelines (SDP) & Delta Live Tables (DLT):
- Design and implement Spark Declarative Pipelines using Databricks.
- Develop Delta Live Table (DLT) pipelines for scalable data transformations.
- Implement data quality expectations, validations, retries, and failure han...