How Jumio built a real-time feature store on AWS
Jumio built a streaming-first real-time feature store on AWS to solve data duplication, manual deployment, latency, and delayed event handling challenges in their fraud detection pipeline The architecture uses Amazon Kinesis Data Streams for ingestion, Amazon Managed Service for Apache Flink for in-flight feature engineering, and Amazon SageMaker Feature Store for low-latency feature serving A dual hot/cold storage strategy leverages Amazon ElastiCache for Valkey for hot in-memory data and SageM
Analysis
TL;DR
- Jumio built a streaming-first real-time feature store on AWS to solve data duplication, manual deployment, latency, and delayed event handling challenges in their fraud detection pipeline
- The architecture uses Amazon Kinesis Data Streams for ingestion, Amazon Managed Service for Apache Flink for in-flight feature engineering, and Amazon SageMaker Feature Store for low-latency feature serving
- A dual hot/cold storage strategy leverages Amazon ElastiCache for Valkey for hot in-memory data and SageMaker Feature Store's standard tier for cold data, achieving sub-100ms latency for fraud detection
- A parallel offline pipeline routes Flink output through Amazon Data Firehose to Amazon S3 (Iceberg format), enabling near real-time model retraining, debugging, and evaluation via Amazon EMR Serverless and Athena
- The system is deployed across three AWS Regions (us-east-1, eu-central-1, ap-southeast-1) and supports schema evolution, conditional feature creation, and agile cross-functional feature development
Why It Matters
This case study provides a practical blueprint for any organization building real-time ML systems that require sub-100ms feature serving, particularly in fraud detection, identity verification, or other latency-sensitive domains. It demonstrates how to unify fragmented offline and online feature engineering workflows into a single centralized platform, eliminating data duplication and manual re-implementation risks that commonly plague ML production pipelines.
Technical Details
- Real-time ingestion and processing: Events enter via Amazon Kinesis Data Streams, where Apache Flink applications perform in-flight feature engineering (conditional feature creation, event-time-based selection) and write directly to Amazon SageMaker Feature Store
- Hot/cold data tiering: Hot features are served from Amazon ElastiCache for Valkey (in-memory, low-latency reads), while cold/less-frequently-accessed features reside in SageMaker Feature Store's standard tier for scalability and durability
- Offline feature store pipeline: Flink sinks write to Amazon Data Firehose → Amazon S3 (Iceberg format); S3 events trigger AWS Lambda → Amazon EMR Serverless, which updates Iceberg tables for model training, debugging, and evaluation
- Multi-region deployment: The architecture is deployed across us-east-1, eu-central-1, and ap-southeast-1 to support global identity verification workloads with regional data residency
- Key AWS services: Amazon SageMaker Feature Store, Amazon Managed Service for Apache Flink, Amazon Kinesis Data Streams, Amazon ElastiCache for Valkey, Amazon Data Firehose, Amazon EMR Serverless, Amazon Athena, and Apache Iceberg
Industry Insight
- Organizations with fragmented feature engineering workflows should prioritize a centralized feature store to eliminate data duplication and reduce the risk of training-serving skew caused by manual re-implementation in production code
- The hot/cold data tiering pattern (ElastiCache + SageMaker Feature Store) offers a cost-effective approach to meeting strict latency SLAs without over-provisioning expensive in-memory infrastructure for all features
- The near real-time offline pipeline (Flink → Firehose → S3/Iceberg → EMR Serverless) demonstrates how streaming and batch can be unified, enabling faster model retraining cycles and improving ML operational maturity
Disclaimer: The above content is generated by AI and is for reference only.