Applied AI Engineer — Evaluation & Reliability

Applied AI Engineer — Evaluation & Reliability

The problem you'd own

15Rock is a financial intelligence firm. Our research goes in front of institutional investors and gets read line by line — by people looking for the one number that doesn't hold up. Every figure has to reconcile across every table and sentence; every claim has to trace to a live source.

We build that research with a multi-agent pipeline on large language models. LLMs write well and count badly: left alone, they produce clean prose that contradicts the table three paragraphs up. Our answer isn't prompt harder. It's to route every number through deterministic computation and gate every report behind checks that pass or fail — never a model grading its own work. You'd build that layer.

What you'll build

• Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel.
• A fact-rec...