Senior Software Engineer in Test (AI Agentic Systems)

Benefits

Parental leave

Not verified

Non-birth-parent leave

Not verified

Family-building benefits

• Fertility benefits: Not verified
• Adoption assistance: Not verified
• Surrogacy assistance: Not verified

Mental health support

Not verified

Relocation assistance

Not verified

Childcare support

Not verified

Learning budget

Not verified

Verification

Not verified

Salary

Not verified not verified - source not recorded; timestamp not recorded

401(k) match

Not verified

Was this benefit information wrong? Tell us.

Market context

Median wage (BLS OEWS)

$116,543 national median

Projected growth (BLS Employment Projections)

+9.8% - Much faster than average

4% below the BLS national median for software engineering aggregate.

Matched to SOC 15-1252 - Software Engineering aggregate by role bucket.

Source: U.S. Bureau of Labor Statistics, OEWS, May 2024 and Employment Projections, 2024-2034.

Schedule

Shift type

Not verified

Weekend work

Not verified

Application

Cover letter

Not verified

Assessment

Not verified

Deadline

Not stated

Where they hire

State eligibility is not yet verified.

About this role

Senior Software Engineer in Test (AI Agentic Systems) Lehi, UT At Collective Health, we're transforming how employers and their people engage with their health benefits by seamlessly integrating cutting-edge technology, compassionate service, and world-class user experience design. This is not a traditional QA role . You will be the quality owner for an LLM-based multi-agent pipeline that autonomously adjudicates health insurance claims for self-funded plan sponsors. You are building a Three-Tier Evaluation Framework to ensure our Gemini-powered agents reason correctly, call tools accurately, and produce DOL-ready outcomes. You will work at the intersection of Vertex AI, healthcare compliance, and high-scale data engineering. Your work directly determines whether claims are paid correctly and whether the company can withstand a Department of Labor (DOL) or state DOI audit. The stakes are real, the domain is hard, and the problems are genuinely novel. What you'll do: - Outcome Evaluation (The "What") - Golden Set Governance: Build and maintain a versioned library of "Grounding Data" results by working with senior claims examiners to define "Ground Truth." - Model-as-a-Judge Automation: Design automated "LLM-grading-LLM" workflows using custom rubrics to score factual grounding and policy compliance. - Semantic Assertion Framework: Develop testing libraries that move beyond string matching to validate semantic equivalence and numerical accuracy in agent outputs. - Trajectory Evaluation (The "How") - Function-Call Auditing: Use Vertex AI traces to programmatically verify that mandatory tools (via MCP) were invoked with correct arguments. - Orchestration Logic Validation: Assert that agents respect defined priorities across the four architectural

Read the full description at jobs.collectivehealth.com. FewerJobs shows a source-linked preview and links to the original posting.