AIAIAI.FYI LIVE

AWS adds NVIDIA Dynamo to SageMaker HyperPod Inference, with 2.05x throughput in disaggregated test

The managed add-on supports multi-node serving, split prefill and decode, and KV-aware routing with vLLM, SGLang or TensorRT-LLM, with vLLM as the default engine.

What happened

Amazon now lets customers run NVIDIA Dynamo, an open source tool for serving large AI models, inside its SageMaker HyperPod service. Some models are too big for one server's GPU memory, so Dynamo spreads them across several servers. It can also split the work of reading a prompt from the work of writing the answer.

Why it matters to you

Companies that run big AI models need them to answer quickly without buying more hardware than necessary. This setup aims to cut the manual infrastructure work involved.

The takeaway

AWS now offers a managed way to run very large AI models across multiple servers.

What it means for you

If you build software

You can deploy via an InferenceEndpointConfig with dynamoSpec or bring your own DynamoGraphDeployment, and choose vLLM, SGLang or TensorRT-LLM.

If you run a business

The managed add-on handles model weight staging, node health monitoring and node replacement, which reduces infrastructure work for teams running large models.

Get stories like this every morning

AWS adds NVIDIA Dynamo to SageMaker HyperPod Inference, with 2.05x throughput in disaggregated test · AIAIAI.FYI