THE ESSENTIALS IN 30 SECONDS

  • SageMaker AI real-time endpoints that use inference components can carve P4d, P5, G7 and G7e GPUs into MIG partitions, up to seven per A100 or H100.
  • Customers still pay the full instance price; AWS says the gain comes from packing more models onto hardware they already rent.
  • In AWS's own test, one H100 partition outperformed a whole A100-40GB on Llama-3.1-8B-Instruct, but was only cheaper per token above a workload-dependent concurrency level.

Amazon SageMaker AI now lets real-time inference endpoints divide a GPU into several hardware-isolated slices using NVIDIA Multi-Instance GPU (MIG). AWS listed the feature on its What's New page on 30 October and detailed it in a technical blog post on 6 November, with benchmarks, limits and configuration steps. Until now, an instance could host no more models than it had GPUs, which often left paid capacity idle when models were small.

How it works

MIG, according to NVIDIA's documentation, gives each GPU instance its own share of memory, compute units and L2 cache. On SageMaker, the option covers four instance families: P4d (A100, 40 GB) and P5 (H100, 80 GB), each with up to seven partitions per GPU, plus G7 (RTX PRO 4500 Blackwell, up to two) and G7e (RTX PRO 6000 Blackwell, up to four). It only applies to endpoints built on inference components. Two optional fields drive it: one defines the partition layout on a production variant, the other assigns each model to a partition type. AWS says no new API operations or IAM actions are needed, and applications keep calling models by name.

The feature is offered in US East (N. Virginia and Ohio), US West (Oregon), Asia Pacific (Singapore and Tokyo) and Europe (Frankfurt and Ireland). Billing is unchanged: customers pay the full instance's hourly rate.

What AWS measured

AWS compared one H100 partition of the 3g.40gb profile with an entire A100-40GB, both serving Llama-3.1-8B-Instruct. The authors report 53% to 89% more output tokens per second and lower latency in every scenario, and an 18% to 34% lower cost per million output tokens, but only above a break-even concurrency. For 100-token prompts, that threshold was 64 concurrent requests; below it, the whole A100 was cheaper. AWS stresses that results depend on the workload. Salesforce helped define requirements and is preparing tests on production-like workloads.

Limits

The partition layout is fixed when an endpoint is created; changing it means a new endpoint and a traffic migration. Single-model, serverless and asynchronous endpoints are excluded, and no partition spans more than one GPU. Isolation covers GPU compute and memory only, so models still share host CPU, memory, storage and networking. Partitioned endpoints also do not report GPU, CPU or memory utilisation metrics, which Unite.AI notes can complicate capacity planning.