Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning 认证安全策展：安全离线强化学习的无分布保证

TL;DR

• Introduces Certified Safety Curation, a filter-then-clone pipeline for safe offline reinforcement learning under weak safety supervision (only segment comparisons and occasional episode-level budget labels)
• Uses a state-only value function trained from pairwise segment comparisons to score entire trajectories, enabling trajectory-level selection rather than transition-level reweighting
• Applies Learn-then-Test calibration to certify a distribution-free $(\alpha, \delta)$ bound on the unsafe fraction of selected data, with a closed-form refusal probability dependent on pool purity
• Achieves safety on 11 of 15 DSRL benchmark tasks, approaching the theoretical limit of cloning the ground-truth safe subset (which requires per-trajectory labels)
• Demonstrates that retraining strong full-label methods on the certified selection yields safe policies where no cost-target tuning could previously achieve safety

Why It Matters

This work addresses a critical practical gap: real-world safe RL systems often cannot afford transition-level cost labels, yet must still provide verifiable safety guarantees. By delivering distribution-free certification without assuming a parametric cost model, it enables deployment in safety-critical domains where both data efficiency and rigorous safety bounds are non-negotiable.

Technical Details

• Filter-then-clone pipeline: A state-only value function is trained from pairwise comparisons of short trajectory segments; this scorer then evaluates whole trajectories, which are filtered before behavior cloning
• Learn-then-Test calibration: A distribution-free statistical framework certifies a selection threshold with an $(\alpha, \delta)$ bound on the fraction of unsafe trajectories in the curated dataset, without requiring assumptions about the underlying data distribution
• Oracle justification: The authors prove that transition-level reweighting fails even with an exact value function, motivating trajectory-level selection as the correct abstraction
• Closed-form refusal probability: The certificate's failure probability has an explicit formula in terms of pool purity, estimated from a calibration sample and entered by the scorer only through this scalar
• Empirical evaluation: Tested on 15 DSRL (Deep Safety RL) tasks; the certified method satisfies cost budgets on 11 tasks, while the uncertified variant reaches 12, with the gap attributable to the cost of certification

Industry Insight

• The trajectory-level selection insight (vs. transition-level) should reshape how practitioners approach offline RL safety when cost labels are sparse, suggesting that coarser supervision can still yield strong guarantees if the aggregation unit is chosen correctly
• Distribution-free certification is a significant step toward deployable safe RL in regulated industries (autonomous vehicles, healthcare, robotics), where unverifiable "best-effort" safety is insufficient for compliance
• The closed-form refusal probability offers a practical calibration tool: practitioners can predictably tune the safety-certainty tradeoff without expensive cross-validation, making certified curation viable for production pipelines

摘要

介绍了认证安全筛选（Certified Safety Curation），这是一种在弱安全监督（仅有片段比较和偶发的剧集级预算标签）下实现安全离线强化学习的“筛选-克隆”流水线。
使用仅基于状态的奖励函数，通过成对片段比较进行训练，从而对完整轨迹进行评分，实现了轨迹级别的筛选而非转移级别的重新加权。
应用“学习-测试”校准方法，为所选数据中不安全部分的比例提供无分布的 $(\alpha, \delta)$ 界限，拒绝概率具有依赖于池纯净度的闭式解。
在 15 个 DSRL 基准任务的 11 个上实现了安全性，接近克隆真实安全子集的理论极限（这需要每条轨迹的标签）。
证明了对认证筛选结果重新训练强全标签方法，可以产生安全策略，而此前没有任何成本目标调节能实现安全性。

深度分析

一句话总结

• 介绍了认证安全筛选，这是一种在弱安全监督（仅有片段比较和偶发的剧集级预算标签）下实现安全离线强化学习的“筛选-克隆”流水线。
• 使用仅基于状态的奖励函数，通过成对片段比较进行训练，从而对完整轨迹进行评分，实现了轨迹级别的筛选而非转移级别的重新加权。
• 应用“学习-测试”校准方法，为所选数据中不安全部分的比例提供无分布的 $(\alpha, \delta)$ 界限，拒绝概率具有依赖于池纯净度的闭式解。
• 在 15 个 DSRL 基准任务的 11 个上实现了安全性，接近克隆真实安全子集的理论极限（这需要每条轨迹的标签）。
• 证明了对认证筛选结果重新训练强全标签方法，可以产生安全策略，而此前没有任何成本目标调节能实现安全性。

为什么重要

这项工作解决了一个关键的实际差距：现实世界的安全强化学习系统往往无法承担转移级别的成本标签，但仍必须提供可验证的安全性保证。通过提供无需假设参数化成本模型的无分布认证，它使得在数据效率和严格安全界限都不可妥协的安全关键领域中的部署成为可能。

技术细节

• 筛选-克隆流水线：仅基于状态的奖励函数通过短轨迹片段的成对比较进行训练；该评分器随后评估完整轨迹，筛选后的轨迹被克隆用于后续训练。