Back to search
Responsibilities
- Profile and benchmark AI training and inference workloads across large-scale HPC clusters to identify network, compute, and memory bottlenecks
- Develop and maintain performance analysis frameworks and dashboards to track system-level metrics including GPU utilization, network bandwidth, latency, and collective communication efficiency
- Investigate and resolve performance regressions in distributed AI training environments, including issues related to RDMA fabrics, collective communication libraries, and job scheduling
- Collaborate with network infrastructure, hardware, and AI research teams to define performance requirements and validate new HPC cluster configurations
- Design and execute capacity and scalability experiments to inform network topology decisions for AI supercomputing infrastructure
- Build tooling and automation to continuously monitor HPC system health, detect anomalies, and reduce mean time to mitigation during performance incidents
- Establish service level objectives for AI cluster network performance and drive cross-functional alignment on reliability and efficiency targets
- Lead technical design reviews for network and system architecture changes affecting AI workload performance, communicating trade-offs clearly to engineering and product stakeholders
- Mentor other engineers on HPC performance methodologies, debugging techniques, and instrumentation best practices
- Leverage AI-assisted workflows to accelerate root cause analysis, automate routine performance reporting, and expand coverage across the HPC stack
Minimum Qualifications
- 5+ years of coding experience in C, C++, Python, or similar programming languages. Flexible to learn new programming languages
- Experience profiling and optimizing distributed AI or HPC workloads, including familiarity with GPU interconnects, RDMA networking, and collective communication frameworks such as NCCL or MPI
- Experience debugging complex, non-reproducible performance issues across multi-layer systems including network fabric, operating system, and application layers
- Experience designing and implementing performance monitoring systems, including instrumentation, telemetry pipelines, and alerting for large-scale infrastructure
- Experience driving cross-functional technical projects from requirements definition through production deployment, including communicating performance findings and trade-offs to diverse stakeholders
- Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
- 6+ years of experience in system performance engineering, network infrastructure engineering, or a related field within large-scale distributed computing or HPC environments
Preferred Qualifications
- Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
- Understanding of RDMA congestion control mechanisms on IB and RoCE Networks
- Understanding of AI training workloads and demands they exert on networks
- Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
- Understanding of the latest artificial intelligence (AI) technologies
- Experience in developing systems software in languages like C++
- Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
- Experience with machine learning frameworks such as PyTorch and TensorFlow
$154,000/year to $217,000/year + bonus + equity + benefits
Learn more about this Employer on their Career Site
