LC-SEPLM: long-range contact-supervised adaptation for sequence-only protein representation learning
LC-SEPLM adapts ESM2 with LoRA and long-range residue-pair contact supervision to incorporate structural information into sequence-only protein representations. The model uses pair-specific queries with cross-attention over the full sequence to capture global context associated with spatial contacts. Trained on 500,000 AlphaFold Swiss-Prot proteins, LC-SEPLM outperforms ESM2 across all eight evaluated protein-level tasks. Largest improvement observed in remote-homology recognition (macro-F1: +0.
Analysis
TL;DR
- LC-SEPLM adapts ESM2 with LoRA and long-range residue-pair contact supervision to incorporate structural information into sequence-only protein representations.
- The model uses pair-specific queries with cross-attention over the full sequence to capture global context associated with spatial contacts.
- Trained on 500,000 AlphaFold Swiss-Prot proteins, LC-SEPLM outperforms ESM2 across all eight evaluated protein-level tasks.
- Largest improvement observed in remote-homology recognition (macro-F1: +0.0647) and EC classification (max absolute gain: +0.1771).
- Demonstrates that contact supervision can effectively inject structural knowledge without sacrificing sequence-only inference efficiency.
Why It Matters
This work addresses a key limitation of current protein language models by integrating 3D structural constraints during training while maintaining compatibility with downstream sequence-based applications. For AI practitioners working in bioinformatics, it offers a practical framework to enhance representation learning without altering deployment pipelines — bridging the gap between sequence modeling and structural biology.
Technical Details
- Architecture: Based on ESM2, enhanced with Low-Rank Adaptation (LoRA) for efficient fine-tuning and added long-range contact supervision via pair-specific cross-attention modules.
- Training Data: 500,000 high-confidence protein structures from AlphaFold DB mapped to Swiss-Prot sequences.
- Supervision Signal: Residue-residue contact maps derived from predicted 3D structures used as auxiliary targets during pretraining.
- Inference Mode: At test time, only sequence embeddings are used — no structural inputs required, preserving computational efficiency.
- Evaluation Benchmarks: Eight protein-level tasks including remote homology detection, function prediction, and enzyme commission (EC) classification; compared against ESM2 and ESM-S baselines.
Industry Insight
Protein AI teams should consider incorporating geometric or topological priors (like contact maps) into sequence model training to boost performance on structure-sensitive tasks without compromising scalability. This approach enables hybrid learning that leverages both evolutionary patterns and physical folding principles — potentially accelerating drug discovery, enzyme design, and variant effect prediction workflows where structural awareness is critical yet traditionally costly to integrate.
Disclaimer: The above content is generated by AI and is for reference only.