Lead Machine Learning Operations Engineer

Job Description:

  • Own ML production reliability strategy
  • Define and lead the operational strategy for production ML systems, including monitoring, traceability, deployment safety, incident response, and post-deployment validation.
  • Set the standards ML teams use to assess model health, performance, and trustworthiness in production.
  • Own model traceability and governance
  • Ensure every production model has clear lineage (data, features, code, artifacts, validation, deployment history) and drive adoption of model registry and metadata tooling across ML teams.
  • Build end-to-end ML observability
  • Design and implement monitoring across the full ML signal path: data arrival, feature freshness, distribution stability, candidate generation, ranking behavior, model metrics, serving latency, and SLA performance.
  • Define production health metrics
  • Partner with ML, data, product, and business stakeholders to define post-deployment metrics covering model quality, system reliability, business guardrails, and degradation indicators.
  • Detect drift and degradation proactively
  • Detect data drift, feature drift, model behavior changes, and silent failures before they impact customers via thresholding, alerting, anomaly detection, and release-over-release monitoring.
  • Lead diagnostic tooling and root-cause analysis
  • Build dashboards, logs, and diagnostic workflows that progress quickly from 'recommendations look off' to root cause, with context captured across candidates, features, scores, ranking decisions, and downstream outcomes.
  • Own ML deployment safety
  • Define and operate automated gates that prevent bad models or bad data from being promoted to production.
  • Partner with MLEs to establish validation checks, rollback criteria, canary strategies, shadow testing, and release health reviews.
  • Lead ML incident response
  • Own incident response practices for ML systems, including rollback playbooks, hotfix strategies, severity definitions, tradeoff frameworks, communications, and post-mortems.
  • Drive closure of systemic gaps after incidents rather than only resolving the immediate issue.
  • Partner across ML Platform, Data, and ML Partner with DevOps/Platform on infrastructure and observability needs; with Data Engineering on data quality, drift, and freshness; and with ML Engineering to embed operational requirements into development and deployment workflows.
  • Set standards and mentor others Act as the technical lead for ML operations: establish reusable patterns, playbooks, and standards, and mentor engineers on reliability, observability, and operational rigor.

Requirements:

  • 5+ years of experience in machine learning engineering, ML platform, applied ML, MLOps, data platform, reliability engineering, or a related technical role.
  • Demonstrated experience operating production ML systems, including monitoring, deployment, incident response, model validation, data quality, or reliability ownership.
  • Experience leading technical initiatives across multiple engineering teams, especially where success required influencing architecture, tooling, standards, or adoption.
  • Hands-on experience with model registries, feature stores, ML metadata systems, production monitoring, model deployment pipelines, or ML observability platforms.
  • Solid knowledge of end-to-end ML systems, including training data, features, model artifacts, offline validation, online serving, post-deployment metrics, and business outcome measurement.
  • Ability to reason about ML operational failure modes: stale features, distribution shift, training-serving skew, delayed labels, and offline-online metric gaps.
  • Solid SQL skills and comfort investigating data quality, feature distributions, model outputs, pipeline behavior, and production anomalies.
  • Track record of cross-functional collaboration with Platform, Data, and ML Engineering to deliver production-grade operational capabilities.
  • Solid written and verbal communication skills, including the ability to explain ML system health, risks, incidents, and tradeoffs to both technical and non-technical stakeholders.

Benefits:

  • medical
  • dental
  • vision
  • 401(k) plan
  • life insurance coverage
  • disability benefits
  • tuition assistance program
  • PTO
Back to blog