Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora
Data Scout inverts the traditional domain-specific corpus construction approach by using targeted web crawling instead of filtering large web archives like CommonCrawl An LLM expands a root topic into a taxonomy and thousands of search queries, with URLs grouped by subdomain and screened by a user-supplied classifier (the "probe") Relevance exhibits a sharp boundary at the subdomain level: in mathematics, a page is 21x more likely to be relevant than one on a sibling subdomain Using FineMath as
Analysis
TL;DR
- Data Scout inverts the traditional domain-specific corpus construction approach by using targeted web crawling instead of filtering large web archives like CommonCrawl
- An LLM expands a root topic into a taxonomy and thousands of search queries, with URLs grouped by subdomain and screened by a user-supplied classifier (the "probe")
- Relevance exhibits a sharp boundary at the subdomain level: in mathematics, a page is 21x more likely to be relevant than one on a sibling subdomain
- Using FineMath as the probe, 21.9% of crawled pages are high-quality math content—70x the 0.31% rate from filtering a comparable web sample
- 63.2% of Data Scout pages are missing from CommonCrawl entirely, and continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens matches FineMath corpus performance on GSM8k
Why It Matters
Data Scout addresses a critical bottleneck in building domain-specific pretraining corpora: specialized domains often contain sparse, non-popular content that popularity-driven crawlers and static archives like CommonCrawl systematically miss. By inverting the filtering paradigm and using targeted crawling guided by a lightweight domain classifier, the approach dramatically improves data efficiency and unlocks previously inaccessible content—making it directly relevant for any practitioner building domain-specialized models in fields like law, medicine, or technical domains.
Technical Details
- Architecture: An LLM expands a root topic into a taxonomy and generates thousands of search queries; returned URLs (seeds) are grouped by subdomain and screened using a user-supplied classifier called the "probe"
- Subdomain-level screening: Admission of each subdomain is based on a small sample, leveraging the observation that relevance has a sharp boundary at the subdomain level (21x higher relevance within the same subdomain vs. sibling subdomains in mathematics)
- Empirical results: With FineMath classifier as probe, 21.9% of crawled pages are high-quality math content versus 0.31% from filtering a comparable web sample; 63.2% of these pages are absent from CommonCrawl
- Training evaluation: Continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens achieves performance on GSM8k matching that of the FineMath corpus
- Generality: The probe is the only domain-specific component, meaning Data Scout can theoretically apply to any domain with an available classifier
Industry Insight
- Organizations building domain-specific models should consider targeted crawling over archive filtering, especially for specialized or low-popularity domains where CommonCrawl coverage is inadequate
- The probe-based design enables rapid adaptation across domains with minimal engineering overhead—investing in a quality domain classifier yields compounding returns as the same pipeline can be reused
- The finding that 63.2% of high-quality crawled content is missing from CommonCrawl suggests that existing public corpora may systematically underrepresent specialized knowledge, creating an opportunity for organizations that invest in targeted data collection to gain a competitive edge in domain-specialized model performance
Disclaimer: The above content is generated by AI and is for reference only.