Deepgram deepens Amazon SageMaker AI observability with Enhanced Metrics
Deepgram introduces Enhanced Metrics that publish billing and usage data directly into Amazon CloudWatch without requiring agents, sidecars, or additional IAM permissions Prometheus and OpenTelemetry support enables engine-level metrics and per-GPU accelerator monitoring through SageMaker AI's detailed observability Metrics are published via CloudWatch Embedded Metric Format (EMF) written to container stdout, working within AWS Marketplace network isolation constraints The ConsumedUnits metric a
Analysis
TL;DR
- Deepgram introduces Enhanced Metrics that publish billing and usage data directly into Amazon CloudWatch without requiring agents, sidecars, or additional IAM permissions
- Prometheus and OpenTelemetry support enables engine-level metrics and per-GPU accelerator monitoring through SageMaker AI's detailed observability
- Metrics are published via CloudWatch Embedded Metric Format (EMF) written to container stdout, working within AWS Marketplace network isolation constraints
- The ConsumedUnits metric aligns with AWS Marketplace metered billing, allowing customers to reconcile bills against actual traffic by model and transport
- Dimensions are low-cardinality with no PII, covering category, model, and transport but not endpoint name or instance ID
Why It Matters
This advancement addresses a critical gap in AI observability for self-hosted speech models, where vendors typically lock billing and feature usage data inside their containers. By enabling transparent, agentless metrics collection within AWS Marketplace's network-isolated environment, Deepgram allows organizations to perform accurate cost reconciliation and capacity planning without compromising security compliance.
Technical Details
- EMF-based metric publishing: The Deepgram container writes CloudWatch Embedded Metric Format records to stdout, which SageMaker AI forwards to CloudWatch Logs. CloudWatch Logs then automatically extracts EMF records into metrics, eliminating the need for external collectors or agents.
- Billing namespace (Deepgram/SageMakerInference): Publishes ConsumedUnits (billable inference units matching AWS Marketplace metered billing), AudioDurationSeconds, and CharCount metrics per completed request, covering streaming sessions, pre-recorded requests, and TTS requests.
- Prometheus and OpenTelemetry integration: Engine-level Prometheus metrics are scraped directly from the Deepgram container, with per-GPU accelerator and host metrics queryable via PromQL from CloudWatch, Grafana, or any Prometheus-compatible tool.
- Network isolation compatibility: Both capabilities operate within AWS Marketplace model package constraints where containers cannot make outbound connections, using the existing SageMaker-to-CloudWatch logging path instead.
- Dimension limitations: Enhanced Metrics aggregate across endpoints in an account and Region with dimensions for category, model, and transport only; per-endpoint or per-instance breakdowns require the Prometheus/OpenTelemetry metrics.
Industry Insight
- AI observability is becoming a key differentiator for enterprise AI deployments, particularly for regulated industries where data residency and compliance are paramount; vendors that provide transparent, agentless monitoring will gain competitive advantage.
- The convergence of billing transparency with technical observability (GPU utilization, engine metrics) enables more accurate FinOps practices for AI workloads, allowing organizations to optimize costs at the model and feature level.
- AWS Marketplace's network isolation model creates unique telemetry challenges that require innovative solutions like EMF-based publishing, setting a precedent for how vendor-specific metrics can be exposed in secure, isolated cloud environments.
Disclaimer: The above content is generated by AI and is for reference only.