Verified Job
IT / Software / Data Analyst
ML Platform Engineer
IT / Software / Data Analyst
#746754
Remote / WFH
eBay · Bengaluru, Karnataka ·
Job Description
Job description
As an LLM Inference Engineer on our AI Platform team, you'll remove the compute-scaling bottleneck for production LLMs. Your job is to make frontier-model inference fast, efficient, reliable, and observable, the last mile from GPUs to APIs that products depend on. This role sits at the intersection of HPC, GPU systems, and MLOps, and requires strong intuition for how model architecture, runtimes, and hardware interact.
Responsibilities:
• Own production inference: Take models from handoff to production-grade serving, including release engineering, capacity planning, cost optimisation, and incident response.
• Tune inference performance: reduce end-to-end latency and increase throughput across real production traffic patterns.
• Optimise runtimes and servers: Scale inference across heterogeneous GPU fleets; optimise stacks such as vLLM, Triton, and related components (e. g., schedulers, KV cache, batching, memory).
• Benchmark and measure: Build benchmarking suites, metrics, and tooling to quantify latency, throughput, GPU utilisation, memory, and cost.
• Reliability and observability: Improve monitoring, tracing, and alerting; participate in incident response and postmortems to harden systems.
• Apply and ship new optimisations: Evaluate research and implement pragmatic inference optimisations (e. g., quantisation, paging, kernel/runtimes improvements).
• Partner cross-functionally: Work with data science and product teams to translate business requirements into performance and availability SLOs.
Requirements:
• Experience deploying and operating LLM inference services in production.
• Strong production coding skills in Python plus Go or Rust (systems-level implementation and debugging).
• Experience with ML frameworks and runtimes: PyTorch, vLLM, SGLang (and/or TensorRT).
• Knowledge of GPU architecture and performance (profiling, memory bandwidth/latency tradeoffs); CUDA/kernel programming is a strong plus.
• Solid understanding of LLM inference and optimisation techniques: continuous batching, KV cache management, quantisation, speculative decoding (nice-to-have), etc.
• 3+ years of hands-on experience in performance optimisation and systems programming for AI/ML workloads.
• Demonstrated ability to deliver measurable production improvements (e. g., 2X throughput, lower p95/p99 latency, reduced GPU cost).
• Proven skill in root-cause analysis: finding bottlenecks across model, runtime, networking, and infrastructure.
As an LLM Inference Engineer on our AI Platform team, you'll remove the compute-scaling bottleneck for production LLMs. Your job is to make frontier-model inference fast, efficient, reliable, and observable, the last mile from GPUs to APIs that products depend on. This role sits at the intersection of HPC, GPU systems, and MLOps, and requires strong intuition for how model architecture, runtimes, and hardware interact.
Responsibilities:
• Own production inference: Take models from handoff to production-grade serving, including release engineering, capacity planning, cost optimisation, and incident response.
• Tune inference performance: reduce end-to-end latency and increase throughput across real production traffic patterns.
• Optimise runtimes and servers: Scale inference across heterogeneous GPU fleets; optimise stacks such as vLLM, Triton, and related components (e. g., schedulers, KV cache, batching, memory).
• Benchmark and measure: Build benchmarking suites, metrics, and tooling to quantify latency, throughput, GPU utilisation, memory, and cost.
• Reliability and observability: Improve monitoring, tracing, and alerting; participate in incident response and postmortems to harden systems.
• Apply and ship new optimisations: Evaluate research and implement pragmatic inference optimisations (e. g., quantisation, paging, kernel/runtimes improvements).
• Partner cross-functionally: Work with data science and product teams to translate business requirements into performance and availability SLOs.
Requirements:
• Experience deploying and operating LLM inference services in production.
• Strong production coding skills in Python plus Go or Rust (systems-level implementation and debugging).
• Experience with ML frameworks and runtimes: PyTorch, vLLM, SGLang (and/or TensorRT).
• Knowledge of GPU architecture and performance (profiling, memory bandwidth/latency tradeoffs); CUDA/kernel programming is a strong plus.
• Solid understanding of LLM inference and optimisation techniques: continuous batching, KV cache management, quantisation, speculative decoding (nice-to-have), etc.
• 3+ years of hands-on experience in performance optimisation and systems programming for AI/ML workloads.
• Demonstrated ability to deliver measurable production improvements (e. g., 2X throughput, lower p95/p99 latency, reduced GPU cost).
• Proven skill in root-cause analysis: finding bottlenecks across model, runtime, networking, and infrastructure.
