Job Description
Responsibilities
- Contribute to building and evolving the platform (infrastructure + reusable abstractions) that standardises data engineering workloads (batch/streaming pipelines, data processing) and traditional ML workflows (feature engineering, training, batch/real-time serving) across teams.
- Implement platform-level IaC, CI/CD, and environment management to support consistent, reproducible workloads across dev/test/prod.
- Build and maintain components using Python and Spark for data processing, shared datasets, and platform services.
- Contribute to shared services for data and ML lifecycle management (data pipelines, experiment tracking, versioning, lineage, permissions), aligned to enterprise governance (e.g. Unity Catalog).
- Support the implementation and operation of a centralised AgentOps capability (LLM gateway, tool integration, prompt and version management).
- Contribute to agent‑specific lifecycle and safet...