FDE - DATA

Develop 20 data table ingestion pipelines end-to-end (source extraction → S3 landing → Glue ETL → Lake Formation curated zone)

Implement ETL transformations per data mapping specifications

Build data quality validation rules (Great Expectations / custom Glue checks) - completeness, schema conformance, referential integrity

Configure error handling and dead-letter patterns for failed ingestion records

Register all datasets in AWS Glue Data Catalog with standardised metadata tags (owner, classification, freshness SLA)

Write and maintain IaC (CDK/Terraform) for pipeline resources - Glue jobs, crawlers, S3 buckets, IAM roles

Execute unit testing (per-transform logic) and integration testing (end-to-end flow with sample data)

Support UAT with Agency A data owners - validate output tables match expected schema and row counts

Document pipeline configurations, runbooks, and data flow diagrams for handover

Part...