Announcing OpenAI-compatible API support for Amazon SageMaker AI endpoints

TL;DR

• Amazon SageMaker AI now supports OpenAI-compatible API endpoints, allowing users to invoke models using the standard OpenAI SDK by simply changing the endpoint URL.
• No custom client libraries or SigV4 wrappers are required; standard tools like LangChain and Strands Agents work out of the box.
• Authentication is handled via time-limited bearer tokens generated from existing AWS credentials, valid for up to 12 hours.
• The feature supports multi-model hosting through inference components, enabling different models (e.g., Llama, Mistral) to be served under a single OpenAI-compatible interface.
• Streaming responses are supported, and the system routes requests based on the endpoint name in the URL.

Why It Matters

This development significantly lowers the barrier to entry for AWS users who rely on the OpenAI SDK ecosystem, eliminating the need for code rewrites or complex authentication setups. It allows enterprises to deploy fine-tuned or open-source models on dedicated infrastructure while maintaining the same application logic used for OpenAI-hosted models. This streamlines development workflows for agentic systems and multi-model architectures within the AWS ecosystem.

Key Data

• Token Validity: Bearer tokens are valid for up to 12 hours by default, with a minimum expiry of 1 second.
• API Path: The OpenAI-compatible endpoint exposes the /openai/v1 path for Chat Completions requests.
• Example Model: The walkthrough uses Qwen3-4B downloaded from Hugging Face as the example model.
• Required Permissions: IAM roles need sagemaker:CallWithBearerToken and sagemaker:InvokeEndpoint permissions to invoke the endpoint.
• SDK Requirement: The SageMaker Python SDK is required to generate tokens using the generate_token function.

Technical Details

• Bearer Token Authentication: The SageMaker Python SDK includes a token generator that creates time-limited tokens from AWS credentials (IAM user, EC2 instance profile, or SSO session). These tokens are passed in the Authorization header as Bearer <token>.
• Endpoint Routing: SageMaker AI routes requests based on the endpoint name in the URL. Any OpenAI-compatible client can access the endpoint without custom routing logic.
• Multi-Model Support: Using inference components, a single SageMaker endpoint can host multiple models (e.g., a large LLM for general tasks and a smaller model for classification). Each model has its own resource allocation but is accessible via the same OpenAI SDK interface.
• Integration: The feature supports integration with frameworks like Strands Agents and LangChain. Users can run agentic workflows on owned infrastructure by pointing these frameworks to the SageMaker endpoint URL instead of OpenAI's API.
• Code Example: A Python class SageMakerAuth can be implemented to auto-refresh tokens for each request by generating a new token via generate_token(region=self.region) inside an auth_flow method.

Industry Insight

• Vendor Neutrality in Cloud AI: This move allows companies to standardize on the OpenAI SDK interface while retaining the data sovereignty and cost control benefits of AWS infrastructure. It decouples application code from the underlying model provider.
• Simplification of Multi-Model Orchestration: Hosting diverse models on a single endpoint with a unified interface reduces application complexity, enabling easier switching or combination of models for different tasks without modifying client code.
• Agentic Infrastructure: By supporting frameworks like Strands Agents natively, AWS is positioning its cloud services as a first-class platform for building and deploying autonomous AI agents, not just static model inference.

FAQ

Q: Do I need to rewrite my existing OpenAI SDK code to use SageMaker?
A: No. You only need to change the endpoint URL to your SageMaker endpoint. The SDK calls, streaming logic, and prompt formatting remain unchanged.

Q: How do I authenticate requests to the new OpenAI-compatible endpoints?
A: You use time-limited bearer tokens generated from your AWS credentials via the SageMaker Python SDK. These tokens are valid for up to 12 hours and require specific IAM permissions.

Q: Can I host multiple different models on the same endpoint?
A: Yes. You can use inference components to host multiple models (e.g., Llama, Mistral) on a single endpoint, each with its own resource allocation, all accessible through the same OpenAI-compatible interface.