Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment
Self-supervised respiratory audio encoders lack clinical semantic grounding, limiting their zero-shot inference capability without task-specific labeled data The proposed framework aligns SSL respiratory encoders with medical terminology in a shared latent space using a medical LLM to synthesize structured reports from metadata as dense semantic anchors Training combines sigmoid-based contrastive loss with the encoder's native SSL objective and similarity-aware negative sampling to sharpen patho
Analysis
TL;DR
- Self-supervised respiratory audio encoders lack clinical semantic grounding, limiting their zero-shot inference capability without task-specific labeled data
- The proposed framework aligns SSL respiratory encoders with medical terminology in a shared latent space using a medical LLM to synthesize structured reports from metadata as dense semantic anchors
- Training combines sigmoid-based contrastive loss with the encoder's native SSL objective and similarity-aware negative sampling to sharpen pathological boundaries
- Across 9 tasks on 6 datasets, the method achieves 61.3% mean zero-shot AUC, surpassing CLAP (51.4%) and Qwen2-Audio (54.9%)
- The approach reaches 71.6% linear probing AUC using only 43% of the data compared to full-scale baselines, demonstrating structured semantic alignment outperforms large general-purpose models in clinical diagnostics
Why It Matters
This work addresses a critical gap in clinical audio AI: self-supervised audio encoders trained on general data lack the medical semantic understanding needed for real-world diagnostic zero-shot applications. By leveraging medical LLMs to bridge the audio-text gap, the framework enables clinically meaningful inference without extensive labeled data, which is scarce and expensive in healthcare domains.
Technical Details
- LLM-Augmented Alignment: Uses a medical LLM to synthesize structured clinical reports from sparse metadata, creating dense semantic anchors for contrastive learning between audio and text representations
- Sigmoid-Based Contrastive Loss: Combines a custom contrastive loss with the encoder's native self-supervised learning objective, enabling joint optimization of semantic alignment and audio representation quality
- Similarity-Aware Negative Sampling: Introduces a sampling strategy that selects hard negatives based on semantic similarity, sharpening pathological boundaries in the latent space
- Evaluation: Tested across 9 classification tasks on 6 respiratory sound datasets, comparing against CLAP and Qwen2-Audio as general-purpose audio-text baselines
- Data Efficiency: Achieves state-of-the-art linear probing performance (71.6% AUC) using less than half the training data of full-scale baselines
Industry Insight
- Medical LLMs can serve as powerful data augmentation tools for bridging modality gaps in clinical AI, especially where paired audio-text data is scarce
- Structured semantic alignment approaches may generalize beyond respiratory sounds to other clinical audio domains (cardiac, neurological), offering a scalable template for medical audio foundation models
- The 43% data efficiency advantage suggests that domain-specific semantic grounding can compensate for reduced training scale, which is valuable in resource-constrained healthcare settings
Disclaimer: The above content is generated by AI and is for reference only.