AI Quality Analyst

This is a remote position.

Reporting to Manager, Quality Engineering & AI Validation, focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. Builds benchmark scenarios, defines scoring rubrics, evaluates business usefulness, and identifies failure patterns that conventional pass or fail software testing would not catch.

Key Responsibilities

AI Output Evaluation

·        Design and execute structured evaluations for AI-enabled features and workflows.

·        Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk.

·        Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations.

Benchmark & Rubric Development

·        Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards.

·        Develop rubrics that allow quality to be measured consistently across releases and changes.

Workflow & Model Change Validation

·        Compare performance across prompt versions, workflow revisions, tools, and models.

·        Support release decisions with evidence on quality regression or improvement.

Business & Domain Partnership

·        Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts.

·        Help define human-review thresholds and escalation patterns for higher-risk use cases.

Production Feedback

·        Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time.

Requirements

Required Qualifications

·        4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles.

·        Strong writing, analysis, and pattern-recognition skills.

·        Experience evaluating outputs against nuanced criteria rather than only binary correctness.

·        Ability to work with structured rubrics, scenario libraries, and evidence-based reviews.

·        Comfort collaborating across Engineering and business teams.

·        Experience with finance, accounting, FP&A, transaction services, or business process design preferred.

·        Bachelor's degree preferred.

You Are

·        Thoughtful, precise, and highly discerning.

·        Strong at spotting subtle output problems others miss.

·        Comfortable with ambiguity but disciplined in scoring and documentation.

·        Focused on trust, usefulness, and business reality.

Benefits