Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Member of Technical Staff, Evals

Home > Python programming jobs

Member of Technical Staff, Evals in USA

  • Profound
  • Full time
  • Email
  • New York, NY

Responsibilities

  • Build Profound’s evaluation system with golden datasets, scorers, LLM judges, programmatic verifiers, regression alerts, and dashboards.
  • Create evaluations for brand extraction, visibility scoring, sentiment, FactCheck, and citation classification pipelines.
  • Define and measure AI Marketer quality across multi-step tool use, analytical correctness, and content quality.
  • Study answer-engine behavior across runs, platforms, regions, and personas using sound sampling and statistical methods.
  • Collect customer signals, cluster failure modes, and convert regressions into permanent guardrails.
  • Build debugging tools for model and agent behavior and create labeling workflows and annotation guidelines.
  • Integrate evaluations into CI, launch reviews, and model-selection processes.

Requirements

  • At least 3 years of experience building evaluation or measurement systems, including AI evaluations, experimentation, ranking, or search quality.
  • Strong Python and data skills with a record of shipping production systems.
  • Deep familiarity with LLM-as-judge evaluation, benchmark design, agent harnesses, calibration, and significance testing.
  • Strong experimental-design skills and the ability to separate signal from noise in non-deterministic systems.
  • Current understanding of model and agent behavior and related research.
  • Evidence of shipped benchmarks, datasets, open-source tools, or publications.
  • Ability to work in a fast-paced startup environment with rapid iteration.
  • Preferred: experience with search, retrieval, citation, or fact-verification systems.
  • Preferred: publications at NeurIPS, ICML, ICLR, ACL, or COLM.

Benefits

  • On-site work in the New York City or San Francisco office.
  • Base salary range of $215,000 to $300,000, plus equity and a full range of benefits and perks.

Salary: $215k - $300k/yr

Profound

Profound builds a SaaS marketing intelligence platform that helps enterprise brands measure and improve how they appear in AI-driven discovery across tools like ChatGPT and Perplexity. The product delivers real-time visibility analytics, competitive intelligence, and optimization insights for sea...

Similar positions

Senior Software Engineer, Data Platform

  • Doximity
  • Full time
  • USA
  • 11/20/2026
  • Remote

Software Engineer 5, AI Observability

  • Netflix
  • Full time
  • USA
  • 11/20/2026
  • Salary: $500k-$860k
  • Remote

Software Engineer - Agentic Development

  • FICO
  • Full time
  • Remote
  • 11/20/2026
  • Salary: Competitive
  • Bulgaria

Senior/Staff Software Engineer

  • Cohere
  • Full time
  • Remote
  • 11/20/2026
  • Salary: USD 185,000 - 300,000
  • Worldwide

Staff Software Engineer, Data Infrastructure

  • Cohere
  • Full time
  • Remote
  • 11/20/2026
  • Salary: USD 245,000 - 335,000
  • Worldwide