ZeroR@CHiPSAL 2026: Two-Stage Vision-Language Adaptation with Contrastive Learning for Nepali Meme Classification
Two-stage vision-language adaptation pipeline using Qwen3-VL-8B-Instruct for Nepali meme classification, combining generative and contrastive learning approaches First stage uses LoRA fine-tuning with an MLP projection head for generative classification; second stage applies contrastive backbone fine-tuning with supervised InfoNCE loss Class imbalance addressed through minority oversampling, image augmentation, and focal loss, with ensemble inference combining token probabilities and classifier
Analysis
TL;DR
- Two-stage vision-language adaptation pipeline using Qwen3-VL-8B-Instruct for Nepali meme classification, combining generative and contrastive learning approaches
- First stage uses LoRA fine-tuning with an MLP projection head for generative classification; second stage applies contrastive backbone fine-tuning with supervised InfoNCE loss
- Class imbalance addressed through minority oversampling, image augmentation, and focal loss, with ensemble inference combining token probabilities and classifier scores
- Achieved 2nd place on hate speech detection (F1: 0.797) and 4th place on sentiment analysis (F1: 0.518) in the CHiPSAL 2026 shared task
- End-to-end approach eliminates error propagation from separate OCR and translation pipelines by leveraging native Devanagari script support
Why It Matters
This work demonstrates a practical framework for adapting large vision-language models to low-resource South Asian languages, addressing a critical gap in multimodal NLP for underrepresented languages. The two-stage approach combining generative and contrastive learning offers a replicable blueprint for multimodal classification tasks beyond hate speech detection. For practitioners working with Devanagari-script languages, the native script support eliminates the compounding errors typical of OCR-translation-classification pipelines.
Technical Details
- Base Model: Qwen3-VL-8B-Instruct, a vision-language model with native Devanagari support, eliminating the need for separate OCR and translation components
- Two-Stage Training Pipeline: Stage 1 applies LoRA fine-tuning with an MLP projection head for generative classification; Stage 2 performs contrastive backbone fine-tuning using supervised InfoNCE loss to improve representation learning
- Imbalance Mitigation: Combines minority class oversampling, image augmentation techniques, and focal loss to address class imbalance in both hate speech and sentiment analysis tasks
- Ensemble Inference: Integrates Stage 1 token probabilities with Stage 2 classifier scores using validation-tuned weights to optimize final predictions
- Benchmark Results: 2nd place on hate speech detection (F1: 0.797) and 4th place on three-class sentiment analysis (F1: 0.518) in the CHiPSAL 2026 shared task
Industry Insight
- The native Devanagari support in modern VLMs represents a strategic shift away from fragile multi-component pipelines (OCR → translation → classification), reducing error accumulation and simplifying deployment for South Asian language applications
- The two-stage generative-contrastive paradigm can be generalized to other low-resource multimodal tasks, particularly where labeled data is scarce and class imbalance is prevalent
- Sentiment analysis performance (F1: 0.518) lagging behind hate speech detection suggests that nuanced sentiment classification in low-resource languages remains an open challenge, warranting further investigation into fine-grained annotation strategies and data augmentation techniques
Disclaimer: The above content is generated by AI and is for reference only.