Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Senior Software Engineer in Test

Home > Python programming jobs

Senior Software Engineer in Test in USA new

  • Collective Health
  • Full time
  • Email
  • Plano, TX

Responsibilities

  • Build and maintain a versioned golden-set library of grounding data and ground-truth results with senior claims examiners.
  • Design automated model-as-a-judge workflows and custom rubrics for factual grounding and policy compliance.
  • Develop semantic assertion frameworks that validate semantic equivalence and numerical accuracy in agent outputs.
  • Audit Vertex AI traces to verify that mandatory MCP tools are invoked with correct arguments.
  • Validate orchestration priorities across data and knowledge, orchestration, agentic reasoning, and tooling layers.
  • Ensure autonomous decisions are traceable to specific SOP sentences and live API data points.
  • Integrate automated regression runs with Vertex AI Prompt Management for every prompt or model update.
  • Own automated side-by-side comparisons between new and production agent versions to detect logic drift.
  • Build Vertex AI and ADK mocking infrastructure for high-volume, low-cost logic testing.

Requirements

  • 5+ years of experience in software testing and development.
  • Expert Python and pytest skills, including custom mocking frameworks for external APIs such as Vertex AI and ADK.
  • Hands-on experience with Vertex AI Experiments, Auto-SxS, and Cloud Logging for trace analysis.
  • Expert-level SQL with BigQuery and Pandas skills for comparing large datasets and identifying adjudication discrepancies.
  • Ability to analyze system instructions and refine prompts based on failed test cases.
  • Experience testing multi-layer systems involving RAG with Vertex AI Search, state management with LangGraph, and function calling.
  • Familiarity with healthcare claims adjudication concepts such as pend reason codes, COB, eligibility, and stop-loss is preferred.
  • Understanding of HIPAA/PHI handling and preparation of test evidence for DOL and DOI regulators is preferred.
  • Experience with shadow-mode monitoring and comparing agent decisions against human expert baselines is preferred.

Benefits

  • Hybrid work based in Lehi, Utah or Plano, Texas, with at least two weekdays in the office.
  • Health insurance, 401(k), and paid time off.
  • Professional development through internal mobility, mentorship programs, and tailored courses.
  • Flexible work arrangements and work-life balance support.

Salary: $99k - $136k/yr

Collective Health

Collective Health is the leading independent third party administrator (TPA) that brings together plan administration, cost management, and taking great care of your people—all in one place. Conventional wisdom in health benefits says you must choose: better or safe. Collective Health allows you ...

Similar positions

Machine Learning Engineer III

  • Mapbox
  • Full time
  • Remote
  • 08/16/2026
  • Germany

Software Eng, Sr Staff

  • Realtor.com Careers
  • Full time
  • USA
  • 08/16/2026
  • Austin, TX

Engineer III

  • CrowdStrike, Inc.
  • Full time
  • Remote
  • 08/16/2026
  • Salary: $120,000 – $180,000
  • Vermont, U.S.

AI Operations Engineer, Partnerships

  • Anthropic
  • Full time
  • USA
  • 08/16/2026
  • Salary: $215k - $300k/yr
  • San Francisco, CA

Customer Reliability Engineer - Airflow

  • Astronomer
  • Full time
  • USA
  • 08/16/2026
  • Remote

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026