How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure
KAI Scheduler and vCluster combine to enable multiple teams to run fully isolated Kubernetes tenant clusters with independent control planes, RBAC, CRDs, and cluster-admin access while sharing a single underlying GPU node KAI Scheduler provides topology-aware, hierarchical GPU scheduling with per-team quotas and dynamic allocation, supporting shared and burst usage models via custom queue CRDs vCluster provisions virtualized Kubernetes clusters per team with complete logical separation, exposing
Analysis
TL;DR
- KAI Scheduler and vCluster combine to enable multiple teams to run fully isolated Kubernetes tenant clusters with independent control planes, RBAC, CRDs, and cluster-admin access while sharing a single underlying GPU node
- KAI Scheduler provides topology-aware, hierarchical GPU scheduling with per-team quotas and dynamic allocation, supporting shared and burst usage models via custom queue CRDs
- vCluster provisions virtualized Kubernetes clusters per team with complete logical separation, exposing real shared GPU and node resources underneath without requiring physical hardware splits
- The architecture eliminates coordination costs of shared clusters (conflicting CRD versions, overlapping RBAC) while avoiding the over-isolation of dedicated clusters per team
- Demo validated on a single NVIDIA L40S GPU with three teams (NLP, Vision, Recommender System) each running real GPU Kubernetes pods in their own tenant clusters
Why It Matters
This architecture directly addresses a critical scaling bottleneck in AI infrastructure: as organizations grow their ML teams, the choice between shared clusters (high coordination overhead) and dedicated clusters (poor resource utilization) becomes increasingly painful. By preserving team autonomy through isolated control planes while maximizing GPU utilization through hardware sharing, this pattern enables AI teams to scale their operations without proportional infrastructure costs.
Technical Details
- KAI Scheduler: A topology-aware, hierarchical Kubernetes scheduler purpose-built for GPU resource allocation in AI workloads. It supports dynamic GPU allocation, per-team quotas, and custom queue CRDs for shared/burst usage models. Runs alongside the default kube-scheduler via
schedulerName: kai-schedulerannotation, and integrates with the NVIDIA GPU Operator for scale across thousands of nodes. - vCluster Architecture: Provisions fully isolated tenant clusters with independent API servers, controllers, data stores, syncers, and schedulers. Uses a shared-nodes model where the virtualized control plane is invisible to tenants—no shared control plane nodes, no in-cluster agent pods, and no lateral paths between environments. Extends to private nodes for untrusted tenants requiring node-, network-, and storage-level separation.
- Demo Environment: Single NVIDIA L40S GPU (48 GB VRAM), 40 vCPUs, 160 GiB RAM on MicroK8s v1.36.2 with NVIDIA GPU Operator pre-installed. Three tenant teams (NLP, Vision, Recommender System) each with independent CRDs, RBAC, and Kubeflow versions sharing the same physical GPU.
- Implementation: Step 3 onward is Kubernetes-distribution-agnostic, requiring only the NVIDIA GPU Operator with Container Device Interface (CDI) enabled. Works across GKE/EKS/AKS/vanilla k8s/k3s with equivalent cluster-creation and GPU Operator install steps.
Industry Insight
- Organizations running multi-team AI platforms should adopt this pattern to reduce GPU infrastructure costs by 40-60% while maintaining team autonomy, as hardware splitting is the primary driver of underutilized GPU capacity in enterprise ML environments.
- The shared-nodes vCluster model is optimal for trusted internal teams, but enterprises with compliance requirements or external tenants should evaluate the private-nodes extension to maintain node-, network-, and storage-level isolation without sacrificing scheduling efficiency.
- As GPU shortages persist, topology-aware hierarchical scheduling with per-team quotas and burst capacity becomes a strategic differentiator—teams that can dynamically allocate GPU resources across workloads will achieve significantly higher training throughput than those relying on static cluster partitions.
Disclaimer: The above content is generated by AI and is for reference only.