Back to search
Responsibilities
- Serve as technical owner and domain expert for a key GenAI inference framework area: serving runtime, distributed inference, graph-mode execution and compilation, or core PyTorch integration
- Lead ambiguous, multi-quarter technical programs end to end: technical design, execution, test strategy and CI, rollout, and production hardening across teams and org boundaries
- Design, implement, and ship features in generative AI inference frameworks and the MTIA integration layers beneath them, from prototype through production deployment
- Own performance for production inference workloads end to end: profile across frameworks, runtime, compiler, and kernel boundaries, identify where time actually goes, and drive the fix to the correct layer rather than the convenient one
- Build and extend the distributed inference substrate: hierarchical KV caching, cross-host transfer paths, disaggregated prefetch and decode, and parallelism strategies for long-context and mixture-of-experts serving
- Optimize the runtime and execution path: graph capture and replay, host-side latency, memory allocation and placement, and throughput under real traffic conditions
- Enable frontier models on MTIA and validate accuracy against GPU baselines, closing correctness gaps and distinguishing real regressions from stale references
- Own the quality bar for your area, ensuring accuracy, stability, latency, and throughput, and test coverage with regression detection that keeps a win from quietly eroding
- Partner with kernel, compiler, and silicon teams on hardware/software co-design, producing framework-level evidence that shapes the future silicon while the design can still change
- Provide technical leadership: mentor other engineers, raise the design quality through reviews, and communicate decisions clearly through design documents and cross-team reviews
Minimum Qualifications
- Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
- Bachelor's degree in Computer Science, Computer Engineering, a relevant technical field, or equivalent practical experience
- 8+ years of professional experience in systems software, ML infrastructure, performance engineering, or compiler/framework development
- 5+ years of hands-on experience with generative AI inference or training frameworks, or production model-serving systems
- Proficiency in Python, C++, or Rust, including low-level systems code and performance-critical paths
- Experience with GenAI inference optimization: prefill and decode optimization, KV-cache management and compression, batching and scheduling, and reasoning about latency and throughput tradeoffs
- Hands-on experience with deep learning framework internals beyond model authoring — operator registration and dispatch, eager execution, autograd, or graph capture and compilation paths
- Experience with runtime-level optimization: graph-mode execution, host-side latency reduction, memory allocation and placement
- Demonstrated experience leading technical design and end-to-end delivery of frameworks or infrastructure projects, including test strategies and CI, across team boundaries
- Experience working across multiple layers of a system stack (framework, compiler, runtime, hardware) and root-causing issues across component boundaries
- Experience debugging numerical accuracy issues and defending accuracy and stability bars, not just performance
Preferred Qualifications
- Experience with LLM inference serving stacks, including continuous batching, paged or chunked attention, speculative decoding, prefix caching, disaggregated prefill and decode, or mixture-of-experts routing and load balancing
- Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
- Experience with long-context inference techniques: context and sequence parallelism, ring or streaming attention, hierarchical or offloaded KV cache
- Experience with AI accelerator platforms (GPU, TPU, or custom ASICs) and bringing up a new hardware backend in a major machine learning framework
- Experience with performance optimization in production environments, with a track record of using profiling and tracing to understand and resolve performance bottlenecks
- Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
- Experience with distributed inference or training at scale: tensor, pipeline, and expert parallelism, collective communication primitives, RDMA transports, and multi-host serving, scale-up and scale-out design
- Experience with low-precision numerics and quantization (FP8, block-scaled formats, INT8/INT4), including calibration and accuracy error analysis, debugging tools for distributed systems
- Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
- Familiarity with the PyTorch compilation stack (TorchDynamo, FX IR, Torch Inductor, dynamic shapes, Triton, AOT I) and custom operator integration
$183,997/year to $257,000/year + bonus + equity + benefits
Learn more about this Employer on their Career Site
