Back to search
Job Description:
Serve as the technical authority for AI quality, evaluation, and risk controls within the Commercial AI Center of Excellence (CAI CoE).
Define and operationalize technical mechanisms that ensure Vertex AI systems meet tax-grade standards for accuracy, defensibility, and compliance.
Design and implement AI evaluation frameworks, including gold datasets, regression testing, and CI gates; define metrics across accuracy, confidence calibration, citation coverage, and abstention correctness.
Lead model validation, error analysis, telemetry, and drift detection strategies; establish best practices for GenAI and ML hybrid systems.
Partner with Legal, Security, and Risk teams to translate policy into enforceable technical mechanisms; review high-risk AI use cases and guide product teams on mitigation strategies.
Influence standards and practices across large engineering organizations while partnering with product teams to improve model performance, reliability, and adoption.
Required Qualifications
15+ years of experience in AI/ML engineering, with demonstrated impact at Senior Principal or equivalent level.
Deep hands-on experience designing and implementing AI/ML evaluation frameworks in production.
Expertise in model validation, drift detection, telemetry, and automated testing pipelines.
Strong understanding of AI governance, auditability, and risk controls.
Experience building evaluation harnesses, gold datasets, and CI-integrated regression testing.
Proven ability to partner cross-functionally with Product, Legal, Security, and Risk teams.
Strong system-level thinking in regulated SaaS environments.
Preferred Qualifications
Bachelor's degree in Computer Science, Engineering, or a related discipline; advanced degree preferred.
Industry experience in tax or other highly regulated domains (insurance, fintech, or healthcare).
Experience with vector databases and retrieval optimization.
FinOps for GenAI: experience modeling and optimizing LLM token and inference costs.
Data science or classical AI/ML background (beyond prompt engineering).
Strong executive communication and technical storytelling skills.
Equivalent combination of education, training, and relevant professional experience accepted in lieu of a formal degree.
Disclaimer:
The above statements describe the general nature and level of work performed in this role. Other duties may be assigned.
Other Qualifications:
The Winning Way behaviors that all Vertex employees need in order to meet the expectations of each other, our customers, and our partners.
• Communicate with Clarity - Be clear, concise and actionable. Be relentlessly constructive. Seek and provide meaningful feedback.
• Act with Urgency - Adopt an agile mentality - frequent iterations, improved speed, resilience. 80/20 rule – better is the enemy of done. Don’t spend hours when minutes are enough.
• Work with Purpose - Exhibit a “We Can” mindset. Results outweigh effort. Everyone understands how their role contributes. Set aside personal objectives for team results.
• Drive to Decision - Cut the swirl with defined deadlines and decision points. Be clear on individual accountability and decision authority. Guided by a commitment to and accountability for customer outcomes.
• Own the Outcome - Defined milestones, commitments and intended results. Assess your work in context, if you’re unsure, ask. Demonstrate unwavering support for decisions.
Comments:
The above statements are intended to describe the general nature and level of work being performed by individuals in this position. Other functions may be assigned, and management retains the right to add or change the duties at any time.
Pay Transparency Statement:
Base pay offered to new hires may vary based upon factors including relevant industry and job-related skills and experience, geographic location, and business needs.* The range displayed does not encompass the full potential of the role, which allows for further growth and career progression.
In addition, as a part of our total compensation package, this role may be eligible for the Vertex Bonus Plan (VOB), a role-specific sales commission/bonus, and/or equity grants.
Learn more about Life at Vertex and connect with your recruiter for more details regarding Vertex's compensation and benefit programs.
*In no case will your pay fall below applicable local minimum wage requirements.
Learn more about this Employer on their Career Site
