Amazon SageMaker AI adds KV-aware routing, cutting p50 time to first token by up to 97 percent
AWS says the new RoutingStrategy sends each LLM request to the instance holding the longest cached prompt match, using cache events the vLLM container reports.
What happened
Amazon SageMaker AI now has a routing option that sends each chatbot request to the server that already did the work on the same text. Language models store that work in GPU memory, called the KV cache. Reusing it means the server skips repeated effort.
Why it matters to you
Chat assistants, document Q&A tools and agents with long instructions send many requests that start the same way. Faster routing can mean quicker replies and more requests handled on the same hardware.
The takeaway
If you run LLM chat or document tools on SageMaker, this setting can make replies faster without changing how clients call the endpoint.
What it means for you
Set one environment variable on the model and RoutingStrategy to KV_AWARE, with at least two instances or two inference component copies, and no PrefixLength to tune.
Teams serving long multi-turn chats or RAG under heavy load may get faster responses and more throughput from the same instances, though AWS says results depend on workload.
The benchmark gains are AWS's own figures on p5 instances, and they show AWS adding inference efficiency features to SageMaker.