Your browser does not support javascript! Please enable it, otherwise web will not work for you.

Senior AI Evals Engineer

Home > Python programming jobs

Senior AI Evals Engineer in Others new

  • Bengaluru, India

Responsibilities

  • Design and build end-to-end evaluation frameworks for LLM-based agents across correctness, latency, cost, safety, and task completion.
  • Create synthetic testing environments, simulation strategies, golden datasets, test harnesses, and centralized eval-as-a-service capabilities.
  • Automate regression and drift detection through CI/CD-integrated evaluation pipelines.
  • Define evaluation taxonomies, scoring rubrics, QA strategies, test plans, test cases, and automated test suites for agentic workflows.
  • Conduct responsible AI and safety evaluations including adversarial testing, red-teaming, bias detection, toxicity measurement, prompt-injection testing, and alignment assessment.
  • Instrument production observability and analyze quality, safety, performance, and risk metrics to recommend mitigations.
  • Benchmark models and evaluate cost, quality, and latency trade-offs for model-selection decisions.
  • Identify, document, prioritize, and track defects and regressions in Jira while validating fixes with Applied AI Engineers.
  • Evaluate new evaluation technologies, run proofs of concept, and influence the platform roadmap.
  • Partner with policy and legal teams on regulatory compliance, fairness, and AI governance requirements.
  • Provide technical leadership through solutioning, design reviews, code reviews, pair programming, and mentoring.
  • Collaborate with ML, product, platform, security, and engineering teams to embed evaluation quality gates into the agent lifecycle.

Requirements

  • Bachelor's or master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.
  • 8+ years of overall software test engineering experience, including 5+ years in test engineering, QA automation, or developer tooling.
  • 2+ years working with ML/AI systems and hands-on experience with LLM applications, prompt engineering, RAG pipelines, or agentic workflows.
  • 2+ years in a senior or lead role owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems.
  • At least 1 year designing or operating generative AI or LLM-agent evaluation frameworks, including benchmarking, red-teaming, and production quality monitoring.
  • Hands-on experience with LLM evaluation frameworks and tooling such as DeepEval, LLM as a Judge, Promptfoo, RAGAS, LangSmith, Hugging Face evaluation tooling, or custom evaluation harnesses.
  • Strong understanding of agentic AI patterns including tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, and retrieval-augmented generation.
  • Production experience building automated, CI-integrated evaluation infrastructure rather than notebook-only workflows.
  • Strong Python skills with pytest, asynchronous programming patterns, and large-scale data processing.
  • Familiarity with GCP or another cloud service relevant to AI/ML and integration of testing into MLOps workflows.
  • Statistical literacy for experimental design, evaluation-metric interpretation, and avoiding benchmark overfitting or metric gaming.
  • Experience with CI/CD pipelines, Git-based workflows, automated testing, static analysis, deployment automation, and environment configuration.
  • Experience with Python, TypeScript, Selenium, Playwright, Kubernetes, Jenkins or GitHub Actions, Datadog or Grafana, and LLM providers or models including Anthropic Claude, OpenAI, Azure OpenAI, and open-source models.
  • Preferred qualifications include deep learning and NLP knowledge, Pandas and SQL data analysis, AI ethics and industry standards familiarity, complex API testing, prompt-injection testing, data poisoning knowledge, and relevant AI/ML, cloud, or software-testing certifications.
  • Nice-to-have experience includes cybersecurity or data-protection agents, human-in-the-loop evaluation, model fine-tuning evaluation, internal developer platforms, open-source evaluation frameworks, published evaluation methodology, and adversarial testing for jailbreak resistance and data leakage prevention.

Rubrik

Rubrik (NYSE: RBRK), the Security and AI company, operates at the intersection of data protection, cyber resilience and enterprise AI acceleration. The Rubrik Security Cloud platform is designed to deliver robust cyber resilience and recovery including identity resilience to ensure continuous bus...

Similar positions

Senior Threat Intelligence Engineer

  • Coalition, Inc.
  • Full time
  • Canada
  • 10/16/2026
  • Salary: C$107k-C$190k
  • Remote

Client Platform Engineer

  • OpenAI
  • Full time
  • USA
  • 10/16/2026
  • Salary: $347k - $445k/yr
  • San Francisco, CA

Senior Staff Inference Performance Engineer

  • d-Matrix
  • Full time
  • USA
  • 10/16/2026
  • Salary: $175k - $265k/yr
  • Remote

Machine Learning Engineer

  • Reddit
  • Full time
  • USA
  • 10/16/2026
  • Salary: $186k-$260k
  • Remote

Software Engineer 5

  • Netflix
  • Full time
  • USA
  • 10/16/2026
  • Salary: $466k-$750k
  • Remote

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026