AWS adds NVIDIA Dynamo to SageMaker HyperPod Inference, with 2.05x throughput in disaggregated test
The managed add-on supports multi-node serving, split prefill and decode, and KV-aware routing with vLLM, SGLang or TensorRT-LLM, with vLLM as the default engine.
What happened
Amazon now lets customers run NVIDIA Dynamo, an open source tool for serving large AI models, inside its SageMaker HyperPod service. Some models are too big for one server's GPU memory, so Dynamo spreads them across several servers. It can also split the work of reading a prompt from the work of writing the answer.
Why it matters to you
Companies that run big AI models need them to answer quickly without buying more hardware than necessary. This setup aims to cut the manual infrastructure work involved.
The takeaway
AWS now offers a managed way to run very large AI models across multiple servers.
What it means for you
You can deploy via an InferenceEndpointConfig with dynamoSpec or bring your own DynamoGraphDeployment, and choose vLLM, SGLang or TensorRT-LLM.
The managed add-on handles model weight staging, node health monitoring and node replacement, which reduces infrastructure work for teams running large models.