FewerJobs.
All jobs

Inference Performance Engineer

Material Depot - New York, NY | Hybrid | Remote

Posted Jun 10, 2026

Benefits

Parental leave
Not verified
Non-birth-parent leave
Not verified
Family-building benefits
  • Fertility benefits: Not verified
  • Adoption assistance: Not verified
  • Surrogacy assistance: Not verified
Mental health support
Not verified
Relocation assistance
Not verified
Childcare support
Not verified
Learning budget
Not verified
Verification
Not verified
Salary
Not verified
401(k) match
Not verified

Was this benefit information wrong? Tell us.

Market context

U.S. role benchmark (BLS OEWS)
$116,543 U.S. median for this role
Projected growth (BLS Employment Projections)
+9.8% - Much faster than average

Matched to SOC 15-1252 - Software Engineering aggregate by role bucket.

Source: U.S. Bureau of Labor Statistics, OEWS, May 2024 and Employment Projections, 2024-2034.

Role

Role function
Engineering From the posting source checked Jun 20, 2026
Seniority
Mid From the posting source checked Jun 20, 2026
Work mode
Hybrid From the posting source checked Jun 20, 2026
In-office days
2 days From the posting source checked Jun 20, 2026

Schedule

Shift type
Not verified
Weekend work
Not verified

Company

Equity
Offered From the posting source checked Jun 20, 2026

Application

Cover letter
Not verified
Assessment
Not verified
Deadline
Not stated

Where they hire

State eligibility is not yet verified.

About this role

Inference Performance Engineer New York, NY | Hybrid | Remote About the role Serving frontier models at scale requires solving novel systems problems at every layer of the stack. As an Inference Performance Engineer, you'll own the runtime that turns accelerators into a production serving system, optimizing throughput, latency, and cost across thousands of nodes. You'll work alongside hardware and compiler teams operating at the frontier of AI silicon design. What you'll do - Build and improve the inference runtime - Design scheduling, continuous batching, KV cache, and prefill/decode disaggregation - Implement low-precision kernels and speculative decoding - Drive throughput, latency, and cost per token - Collaborate with hardware teams on kernels, operators, and graph optimizations - Own the OpenAI-compatible API surface and serving protocol - Build benchmarking, profiling, and regression infrastructure What you'll need - BS in CS, EE, or related field, or equivalent experience - Software engineering experience: Rust, Go, Python, or C++ - Understanding of concurrency, memory, and tail latency - Understanding of modern inference: transformers, attention, KV cache, batching, speculative decoding, quantization - Experience with model serving frameworks: vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp, or custom runtimes - GPU or ASIC programming experience: CUDA, ROCm, Triton, or vendor-native toolchains - Experience with low-precision inference (FP8, FP4, INT4) - Profiling and benchmarking experience: Nsight, perf, custom harnesses What we offer - Top-tier compensation structured to recognize and retain the best talent - Meaningful equity - Comprehensive medical, dental, vision, life, and disability insurance - Parental leave for all

Read the full description at jobs.ashbyhq.com. FewerJobs shows a preview and links to the original posting.

Apply at jobs.ashbyhq.com

Apply link not verified; last-live date unavailable.

What verified means

Verified means a displayed claim has field-level provenance to a source FewerJobs pulled: a government or employer source, or the original job posting. Posting-sourced facts are employer-stated and are labeled separately from government records.

Related jobs