Back to search
Description
The Machine Learning Data Ops QA team ensures that Research and Development teams receive complete, accurate, and consistent datasets to train the models powering continuous feature development. We support our data collection, annotation and synthesis partners with defining quality standards and verifying that data deliverables meet this high quality bar before they are consumed by R&D teams. As the Data Quality Lead, you own the quality of the datasets in your portfolio and the standards they are measured against. The role spans the full data request life cycle: defining what good looks like with R&D before collection begins, designing checks that catch problems during collection rather than after delivery, leading the analysts who carry out review, and reporting findings to project teams, partner organizations, and vendors. You will also build and extend the team's QA tooling, including review interfaces, analysis pipelines, and reporting, using agentic AI tools to add new capabilities and to find more efficient ways of delivering high quality data.
Minimum Qualifications
Bachelor's degree, or equivalent practical experience. 4+ years of experience in ML data operations, data quality, or a comparable data-centric quality function. Working proficiency in Python for data manipulation and reporting. Hands-on experience using AI coding assistants to build working QA tools or analysis. Strong written and verbal communication skills.
Preferred Qualifications
Experience designing labeling taxonomies or annotation guidelines and adjudicating ambiguous cases with vendors. Experience leading internal or external quality analysts and designing or running human rating and evaluation programs, including rater calibration, gold sets, and ongoing quality monitoring. Familiarity with statistical quality methods, including sampling strategy, inter-rater agreement, acceptance rates, and error magnitude and confidence analysis. Experience designing and iterating on prompts for quality checks assisted by large language models (LLMs) or vision language models (VLMs). Experience building internal QA tooling end to end, such as a review interface, a data pipeline, or a browser-based dashboard (HTML, CSS, JavaScript). Excellent attention to detail with a passion for problem solving, investigation, and root cause analysis. Strong critical thinking, with the judgment to question assumptions and validate a quality signal before relying on it. Excellent project management, analytical, and organizational skills, with the ability to manage several projects in parallel in a dynamic environment with shifting priorities.
Learn more about this Employer on their Career Site
