AWS adds Global Inference Gateway to SageMaker HyperPod for routing across clusters and Regions
The managed gateway scores clusters on four signals, fails over automatically and sends traffic over AWS PrivateLink, with SigV4 and IAM authentication on every request.
What happened
AWS launched a tool that gives AI teams one address for models running on many GPU clusters. It picks the best cluster for each request. If a cluster fails, it moves traffic to another one.
Why it matters to you
Companies that rent scarce GPU capacity in several places can use idle chips instead of letting requests wait in a queue. They also skip building their own networking and failover.
The takeaway
AWS now routes AI requests across its clusters and Regions for customers, so one busy cluster no longer leaves others idle.
What it means for you
You publish each cluster through spec.globalRouting, create one gateway, and send OpenAI-compatible SigV4-signed requests, with no custom load balancers, DNS or cross-Region networking.
The gateway aims to use GPU capacity you already pay for, and AWS says it gives small platform teams cross-Region failover without a custom routing layer.