Job Description
Streaming & Real-Time Data Processing
- Design and develop real-time streaming pipelines using Databricks Structured Streaming.
- Build and maintain Kafka-based ingestion frameworks.
- Handle late-arriving events using watermarks, event-time processing, and stateful streaming concepts.
- Implement exactly-once processing and checkpointing mechanisms.
- Monitor and optimize streaming workloads for performance and reliability.
Spark Declarative Pipelines (SDP) & Delta Live Tables (DLT)
- :
Design and implement Spark Declarative Pipelines using Databrick - s.Develop Delta Live Table (DLT) pipelines for scalable data transformation
- s.Implement data quality expectations, validations, retries, and failure h...