Why We Fine-Tuned SigLip (And Why That's Not Always the Right Call)
Alma Media built a multi-label image classifier for real estate listings with 23 classes (room types, floor plans, aerial shots, etc.) by fine-tuning the open-source SigLIP model rather than relying on third-party VLM APIs The article provides a practical business-context framework for choosing between API-based VLM prompting and custom-trained classifiers, covering promptability, training data needs, cost, and confidence calibration Cost analysis reveals a dramatic divergence at scale: API-base
Analysis
TL;DR
- Alma Media built a multi-label image classifier for real estate listings with 23 classes (room types, floor plans, aerial shots, etc.) by fine-tuning the open-source SigLIP model rather than relying on third-party VLM APIs
- The article provides a practical business-context framework for choosing between API-based VLM prompting and custom-trained classifiers, covering promptability, training data needs, cost, and confidence calibration
- Cost analysis reveals a dramatic divergence at scale: API-based classification costs ~$1,500 per million images vs. ~$0.37 for a self-hosted SigLIP classifier on AWS EC2 g4dn.xlarge (T4 GPU), a ~4,000x difference
- VLM-derived confidence scores are poorly calibrated, making custom classifiers preferable when reliable per-class confidence is needed for production decision-making
- A phased launch strategy is recommended: start with a prompted VLM to accumulate annotated data, then transition to a fine-tuned custom classifier once sufficient labeled data is available
Why It Matters
This article addresses a gap in the AI literature by focusing on business-context trade-offs rather than pure benchmark performance, providing actionable guidance for practitioners who must decide between quick-to-deploy API solutions and longer-term custom classifier investments. The cost analysis at scale is particularly relevant for any organization processing large volumes of image data, where API costs can become a significant operational expense.
Technical Details
- Model: Fine-tuned
google/siglip-base-patch16-224(open-source ViT foundation model from Google), deployed on AWS EC2 g4dn.xlarge instances with T4 GPUs, achieving 400+ images per second inference throughput - Task: Multi-label classification across 23 classes including LIVING ROOM, KITCHEN, BEDROOM, HALLWAY, LOFT, ALCOVE, floor plans, site plans, aerial shots, garden photos, and realtor marketing materials
- Data strategy: Leveraged a phased approach — initial predictions from a prompted VLM were used to bootstrap an annotated dataset, followed by a cleanup pass to correct inherited VLM errors before fine-tuning
- Cost comparison: API approach (Google Agent Platform, gemini-3.5-flash) at ~$1.50 per 1,000 images; self-hosted approach at ~$0.53/hour on EC2, yielding ~$0.37 per million images for inference compute only
- Confidence calibration: Custom classifiers provide well-calibrated per-class probability scores, whereas VLM token log-likelihoods and verbalized confidence estimates are known to be poorly calibrated for downstream decision-making
Industry Insight
- Organizations processing image data at scale should model total cost of ownership carefully before committing to API-based solutions; the 4,000x cost difference demonstrated here can be decisive for high-volume production systems
- The phased "VLM-first, then fine-tune" strategy is a practical blueprint for teams launching new classification products without existing labeled data, turning an initial accuracy trade-off into a long-term cost and calibration advantage
- Reliable confidence estimation should be treated as a first-class requirement in classifier design; when downstream systems depend on confidence thresholds (e.g., filtering search results), custom-trained models with calibrated scores are likely to outperform API-based VLMs despite potentially lower raw accuracy
Disclaimer: The above content is generated by AI and is for reference only.