ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science ClinLens：面向长时序多模态临床数据科学的长周期编码代理研究

TL;DR

• ClinLens introduces a new benchmark for long-horizon coding agents in clinical data science, featuring 200 executable tasks over five linked MIMIC resources.
• The benchmark includes a 4 x 5 taxonomy crossing four patient-time scopes with five analysis capabilities, emphasizing auditable and correct clinical analyses.
• Despite high execution success rates, the strongest model-scaffold configuration achieves only 56.3% scope-macro STRICTPASS, highlighting a significant gap between runnable submissions and correct clinical insights.

Why It Matters

ClinLens addresses a critical need in AI by providing a comprehensive benchmark that evaluates the ability of coding agents to handle longitudinal multimodal clinical data effectively. This is crucial for advancing clinical AI systems that can transform complex medical records into actionable insights, thereby improving healthcare outcomes through more reliable and interpretable AI models.

Technical Details

• Benchmark Composition: ClinLens consists of 200 executable tasks across five linked MIMIC resources, including structured electronic health records, notes, electrocardiograms, chest radiographs, and echocardiograms.
• Taxonomy Framework: A 4 x 5 taxonomy is used to categorize tasks based on four patient-time scopes and five analysis capabilities, ensuring a diverse evaluation of agent performance.
• Program-First Reverse Synthesis: Each task is paired with an evaluator-private reference workflow, which checks required artifacts, cohort and temporal semantics, and the final answer to ensure correctness.
• Performance Metrics: The benchmark uses metrics such as EXECSUCCESS and scope-macro STRICTPASS to evaluate the performance of different model-scaffold configurations.

Industry Insight

The substantial gap between runnable submissions and correct clinical analyses identified by ClinLens underscores the need for more robust and accurate AI systems in healthcare. This insight suggests that future research should focus on enhancing the interpretability and reliability of clinical AI agents, potentially through improved training methodologies or more sophisticated evaluation frameworks. Additionally, it highlights the importance of developing benchmarks that closely mimic real-world clinical scenarios to better assess the practical utility of AI technologies in healthcare settings.

TL;DR

• 提出ClinLens基准，包含200个可执行任务，覆盖结构化电子健康记录、文本、心电图、胸部X光和超声心动图五种多模态纵向临床数据。
• 采用程序优先反向合成方法，为每个任务绑定评估器私有的参考工作流，并严格校验工件、队列语义、时间语义及最终答案。
• 在126项固定任务中，最强模型配置达到56.3%的宏平均严格通过率（STRICTPASS），但代码执行成功率为100%，暴露出“可运行≠正确分析”的重大差距。
• 一个独立配置的编码Agent解决83/126任务，而五个适配GPTO-mini的生物医学系统最高仅达2.9% STRICTPASS，凸显当前AI系统在复杂临床推理上的严重不足。
• 构建4×5任务分类矩阵（4种患者时间范围 × 5种分析能力），推动从单一问答向长周期、多模态、可审计的临床科学代理演进。

为什么值得看

该研究首次系统性地构建了面向长周期、多模态临床数据的可执行任务基准，填补了现有医疗AI评测在真实世界复杂性与可审计性之间的空白，对开发具备临床落地能力的智能体具有关键指导意义。其揭示的“高执行率但低正确率”现象，警示行业需超越表面功能实现，深入关注逻辑一致性与领域合规性。

技术解析

• ClinLens基于MIMIC数据库的五类资源（EHR结构表、临床笔记、ECG、CXR、Echo）构建跨模态关联数据集，支持纵向追踪与多源融合分析。
• 任务设计采用4×5 taxonomy：四个患者时间 scope（如单次就诊、跨年度随访等）与五个分析 capability（如描述统计、趋势建模、风险预测、因果推断、异常检测）。
• 使用program-first reverse synthesis机制：先定义理想输出与工作流，再自动生成输入包与评估标准，确保任务具备明确验证路径和防作弊隔离。
• 核心指标包括EXECSUCCESS（代码能否运行）、SCOPE-MACRO STRICTPASS（在所有时间范围与分析类型上完全正确的比例），后者更强调临床语义准确性而非语法正确性。
• 实验对比显示，即使是最优的24种模型-骨架组合也无法突破56.3% STRICTPASS，而通用大模型微调版本表现极差（≤2.9%），说明当前架构缺乏对临床逻辑的深度理解能力。

行业启示

• 医疗AI研发应从“能跑通”转向“可信且可解释”，未来系统必须内嵌形式化验证模块以保障分析过程符合医学规范与伦理要求。
• 临床数据科学代理需整合多模态时序建模能力，单一模态或静态快照式处理无法满足真实诊疗场景需求，建议优先布局跨模态对齐与长期依赖捕捉技术。
• 建立标准化、可复现、带隐私保护屏障的 benchmark体系是推动产业落地的前提，ClinLens模式可作为其他垂直领域（如金融、法律）Agent评估的参考范式。