Your browser does not support javascript! Please enable it, otherwise web will not work for you.

AI Infrastructure Engineer

Home > Python programming jobs

AI Infrastructure Engineer in USA

  • Thinking Machines Lab
  • Full time
  • Email
  • San Francisco, CA

Responsibilities

  • Own the reliability, performance, and uptime of large-scale post-training and reinforcement learning jobs.
  • Partner with research teams during active model runs to debug failures and accelerate iteration.
  • Debug distributed-system failures across accelerators, networking, storage, schedulers, and training frameworks.
  • Build monitoring, alerting, automated recovery, checkpointing, fault tolerance, and scheduling improvements.
  • Develop internal tools that reduce operational toil and improve cluster utilization.
  • Participate in an on-call rotation and write postmortems that lead to permanent infrastructure fixes.

Requirements

  • At least 4 years of experience as a production, site reliability, or infrastructure engineer operating large-scale distributed systems in production.
  • Experience debugging complex networking, hardware, kernel, or scheduler failures.
  • Strong software engineering skills in Python and/or Go/C++.
  • Solid Linux systems-internals and networking knowledge.
  • Experience owning production systems and participating in on-call rotations.
  • Preferred experience operating GPU or TPU training clusters at scale.
  • Preferred familiarity with RLHF, PPO, DPO, reward model serving, rollout generation, and mixed training/inference workloads.
  • Preferred experience with PyTorch, Ray, Slurm, Kubernetes, InfiniBand, RDMA, NCCL, and ML-training observability tooling.

Benefits

  • Health, dental, and vision benefits.
  • Unlimited paid time off.
  • Paid parental leave.
  • Relocation support as needed.
  • Visa sponsorship is available.
  • The role is based in San Francisco, California.
  • Participation in an on-call rotation is required.

Salary: $350k - $475k/yr

Thinking Machines Lab

Thinking Machines Lab pioneers advanced AI research, blending cutting‑edge theory with practical applications to solve complex problems across industries. Our interdisciplinary team pushes the boundaries of machine learning, neural architecture, and cognitive modeling to deliver innovative soluti...

Similar positions

Software Engineer in Test

  • Roadie
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Remote

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: €73k-€116k
  • Greece

Principal Software Engineer - Builder Experience

  • Elastic
  • Full time
  • Remote
  • 09/13/2026
  • Salary: zł 369k-zł 584k
  • Poland

Senior Devops Engineer

  • Woliba
  • Full time
  • USA
  • 09/13/2026
  • Remote

Senior Core Infrastructure Engineer

  • Oracle
  • Full time
  • USA
  • 09/13/2026
  • Salary: Competitive
  • Nashville, TN

Listings

by


Job Type

Language

Country

CodingJobBoard.com © 2026