Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components
Salesforce achieved an 8x reduction in GPU infrastructure costs by co-hosting multiple models on shared GPUs using SageMaker Inference Components (ICs) Default IC placement optimized for cost alone did not guarantee Multi-AZ resilience, creating single points of failure at both instance and AZ levels The new `SchedulingConfig` parameter in the `CreateInferenceComponent` API provides fine-grained control over IC copy placement across instances and AZs Two key sub-parameters drive HA behavior: `Av
Analysis
TL;DR
- Salesforce achieved an 8x reduction in GPU infrastructure costs by co-hosting multiple models on shared GPUs using SageMaker Inference Components (ICs)
- Default IC placement optimized for cost alone did not guarantee Multi-AZ resilience, creating single points of failure at both instance and AZ levels
- The new
SchedulingConfigparameter in theCreateInferenceComponentAPI provides fine-grained control over IC copy placement across instances and AZs - Two key sub-parameters drive HA behavior:
AvailabilityZoneBalance(cross-AZ distribution with configurable imbalance tolerance) andPlacementStrategy(SPREADfor fault isolation vs.BINPACKfor utilization efficiency) - For ongoing rebalancing after repeated scale operations, configuring the endpoint's
ScaleInPolicywith theCONSOLIDATIONstrategy enables a background sweeper to periodically consolidate IC copies while honoring AZ balance constraints
Why It Matters
This case study demonstrates a critical gap between cost-optimized ML infrastructure and enterprise compliance requirements for high availability. For AI practitioners running production inference workloads, it highlights that default cloud platform behaviors may not satisfy Multi-AZ resilience mandates, requiring explicit configuration to meet internal compliance bars. The solution pattern is broadly applicable to any organization using SageMaker ICs for cost-efficient multi-model GPU sharing who also need production-grade HA guarantees.
Technical Details
- SchedulingConfig parameter: Introduced in the
CreateInferenceComponentAPI, enabling fine-grained control over IC copy placement across instances and Availability Zones - AvailabilityZoneBalance: Controls cross-AZ distribution with
EnforcementMode(e.g.,PERMISSIVE) andMaxImbalancesettings;MaxImbalance: 0enforces strict balance (exactly 1 copy per AZ for 2-copy deployments), whileMaxImbalance: 1tolerates at most a 1-copy difference between AZs - PlacementStrategy:
SPREADdistributes copies across as many instances as possible for fault isolation;BINPACKpacks copies onto fewer instances for utilization efficiency - Scaling behavior: SageMaker maintains AZ balance during scale-out (placing new copies evenly) and scale-in (symmetrically removing copies across AZs);
CopyCountmust never be set to 1 for HA-critical models as a single copy resides in only one AZ - CONSOLIDATION ScaleInPolicy: Configured at the endpoint level via
ManagedInstanceScaling, a background sweeper periodically consolidates IC copies and releases idle instances while honoring AZ balance constraints, addressing drift from repeated scale operations
Industry Insight
- Cloud ML platforms are increasingly offering cost-optimization features (like multi-model GPU co-hosting) that outpace their HA and compliance guarantees; practitioners should audit default placement behaviors against their resilience requirements before production deployment
- The pattern of combining per-operation placement controls (
SchedulingConfig) with background rebalancing (CONSOLIDATIONstrategy) provides a reusable blueprint for maintaining Multi-AZ compliance in dynamically scaled inference workloads - Organizations with strict compliance mandates (like Salesforce's 2-AZ requirement) should treat
CopyCount: 1as a hard anti-pattern for production models and implement automated guardrails to prevent accidental single-AZ deployments
Disclaimer: The above content is generated by AI and is for reference only.