Practical IP-Level Unsupervised Classification Using HDBSCAN and K-Means
Solves the problem of distinguishing two consumer brands sharing the same ASN by leveraging DNS resolver IP as a hardware fingerprint, since home routers often serve as the DNS resolver rather than public upstream resolvers Uses HDBSCAN clustering combined with reverse DNS (PTR) lookups as an annotation engine to discover brand-distinguishing patterns in the absence of any labelled training data The final production classifier is a pure rule-based lookup system combining four signals in priority
Analysis
TL;DR
- Solves the problem of distinguishing two consumer brands sharing the same ASN by leveraging DNS resolver IP as a hardware fingerprint, since home routers often serve as the DNS resolver rather than public upstream resolvers
- Uses HDBSCAN clustering combined with reverse DNS (PTR) lookups as an annotation engine to discover brand-distinguishing patterns in the absence of any labelled training data
- The final production classifier is a pure rule-based lookup system combining four signals in priority order: IPv6 DNS sub-prefix, IPv4 DNS + public IP prefix, geographic coverage maps, and Wi-Fi router MAC OUI
- Key technical insight: the second hextet of IPv6 DNS addresses maps directly to brand-specific sub-allocations within a shared /20 block, providing a clean discriminating feature
- Demonstrates a general ML lifecycle pattern where clustering and discovery methods serve development-time annotation, then get replaced by lightweight deterministic rules at inference
Why It Matters
This approach provides a practical blueprint for solving unsupervised disambiguation problems in network analytics where ground truth labels are unavailable—a common scenario in real-world data science. The methodology of using clustering as an annotation engine rather than a final product, then hardening discovered patterns into interpretable rules, is broadly applicable beyond this specific use case. It also highlights the underutilized value of PTR records and MAC OUI data in network-level classification tasks.
Technical Details
- Data structure: Each row represents aggregated device sessions grouped by DNS resolver IP, with features including IP range bounds (IPv4/IPv6), ASN organization string, geographic region, connection public IP, and session count. No labelled training data exists.
- Feature engineering: Wildcard IP cleaning (replacing
xplaceholders with0), prefix extraction (first two octets for IPv4, first two hextets for IPv6), log transformation of skewed numerics (usingmath.log2()for IPv6 ranges due to Python arbitrary-precision integers that NumPy cannot handle), frequency encoding of high-cardinality categoricals, and StandardScaler normalization. - Clustering pipeline: K-Means (k=3) served as a baseline but failed due to spherical cluster assumptions. DBSCAN was limited by its single global epsilon parameter. HDBSCAN was selected as the final approach with
min_cluster_size=120,min_samples=5, Euclidean metric, parallel core distance computation, andprediction_data=Truefor soft cluster assignment—successfully handling varying cluster densities and isolating outliers as noise. - PTR-based label assignment: Reverse DNS lookups via
dnspythonwithasyncio, concurrency control via semaphore, and exponential backoff across multiple upstream resolvers. Router hostnames in PTR records act as hardware fingerprints for brand identification. - Multi-signal fusion classifier: Four signals combined in priority order—(1) IPv6 DNS second hextet mapping to brand sub-allocations, (2) IPv4 DNS /16 + public IP /16 prefix combination, (3) geographic coverage table as tiebreaker, (4) Wi-Fi router MAC OUI revealing CPE hardware brand. Each signal is independently validatable and maintainable.
Industry Insight
- The clustering-then-rules pattern should be considered a standard approach for unsupervised annotation problems: use ML to discover structure and generate labels, then replace the model with interpretable, maintainable rules for production. This reduces computational cost, eliminates ML dependencies at inference, and improves auditability.
- Network operators and analytics teams should systematically leverage PTR records and MAC OUI data as underutilized signals for device and service identification, especially in scenarios involving shared infrastructure or co-branded services.
- The explicit multi-signal priority chain design is more robust and maintainable than monolithic models—each signal can be validated, updated, or disabled independently, making the system more resilient to data quality issues and easier to debug in production.
Disclaimer: The above content is generated by AI and is for reference only.