HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising HOBA：分层在线策略竞价代理用于自适应在线广告

TL;DR

• HOBA introduces a hierarchical reinforcement learning framework for online advertising bidding that decouples strategic reasoning, model selection, and bid execution across three time scales.
• The high-level component uses a large language model with a Think-Act-Observe-Reflect loop to infer hyperparameters from contextual signals.
• The mid-level SARSA agent dynamically selects among expert models while incorporating causal adjustment to eliminate selection bias.
• The low-level dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints, confining online learning to discrete expert selection rather than continuous bid optimization.
• Experiments on AuctionNet benchmark and large-scale A/B tests demonstrate consistent improvements over state-of-the-art baselines, with +3.6% increase in target cost in online deployment.

Why It Matters

This work addresses critical limitations in current online advertising bidding systems: lack of adaptability to non-stationary markets and reliance on costly manual tuning. By introducing a hierarchical approach that separates strategic decisions from execution, HOBA reduces exploration risk while maintaining the flexibility needed for real-world auction environments. The successful deployment showing measurable business value (+3.6% target cost increase) demonstrates the practical viability of this multi-agent paradigm for industry applications.

Technical Details

• Three-tiered architecture: High-level LLM for hyperparameter inference via Think-Act-Observe-Reflect loop with historical experience retrieval; Mid-level SARSA agent for expert model selection with causal adjustment; Low-level dynamic expert pool containing PID, MPC, IQL, and Decision Transformer models.
• Strategic decoupling across time scales enables confined online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while preserving adaptability.
• Causal adjustment mechanism at mid-level eliminates selection bias when choosing among expert models based on contextual signals.
• Evaluation conducted on AuctionNet benchmark and validated through large-scale A/B testing in production environment.
• Implementation leverages multiple established algorithms (PID controllers, model predictive control, offline RL policies) within a unified hierarchical framework.

Industry Insight

The hierarchical design offers a blueprint for deploying adaptable AI systems in volatile market environments where traditional monolithic models struggle. By separating strategic reasoning from execution, organizations can maintain stability while allowing higher-level components to respond to changing conditions without risking operational disruption through continuous policy updates. This approach suggests future bidding systems should prioritize modular architectures that enable controlled experimentation and rapid adaptation to market shifts, potentially reducing the need for extensive manual intervention and hyperparameter tuning cycles.

TL;DR

• 提出HOBA分层强化学习框架，解决在线广告竞价系统中缺乏在线适应性和超参数手动调优成本高的问题。
• 通过三层时间尺度解耦战略推理、模型选择和竞价执行：高层使用大语言模型结合历史经验检索推断超参数；中层SARSA代理动态选择专家模型并引入因果调整消除选择偏差；底层动态专家池（PID、MPC、IQL、Decision Transformer）在高约束下执行竞价。
• 将在线学习限制在离散专家选择而非连续竞价优化中，显著降低探索风险并保持适应性。
• 在AuctionNet基准和大规模A/B测试中持续优于最先进基线，线上部署实现目标成本+3.6%提升。

为什么值得看

该工作针对在线广告竞价系统的核心痛点——市场非平稳性与人工调优瓶颈，提出了一种兼具工程落地性与理论创新的分层RL架构，为工业界提供了可复用的自适应竞价范式。其“离线专家+在线选择”的设计思路有效平衡了探索效率与风险控制，对构建鲁棒性强的AI驱动营销系统具有重要参考价值。

技术解析

1. 分层架构设计：HOBA采用三层时间尺度分离策略，高层（宏观）负责超参数生成，中层（中观）做模型选择决策，底层（微观）执行具体竞价动作，形成清晰的职责边界。
2. 高层LLM推理机制：利用大语言模型通过Think-Act-Observe-Reflect循环，结合历史经验检索从上下文信号中动态推断bid bounds等超参数，减少对预设规则的依赖。
3. 中层因果SARSA选择器：基于SARSA算法构建专家选择器，并引入因果调整模块校正因历史数据分布导致的样本选择偏差，确保所选模型在当前环境下更可靠。
4. 底层多元专家池集成：整合PID控制器、模型预测控制(MPC)、离线强化学习(IQL)及Transformer类模型(Decision Transformer)作为基础执行单元，根据场景灵活切换最优策略。
5. 实验验证体系：在标准AuctionNet仿真平台进行对比测试，并在真实流量环境中开展大规模A/B评估，最终证明其在商业指标上的显著提升效果。

行业启示

1. 推动竞价系统向“自进化”演进：传统静态规则或单一预训练模型难以应对瞬息万变的市场环境，HOBA证明引入分层在线学习能力是实现系统自我优化的关键路径。
2. 降低AI落地门槛：通过将复杂的连续优化任务转化为离散的专家选择问题，大幅减少了在线训练带来的不稳定性和资源消耗，更适合企业级快速部署。
3. 重视因果思维在推荐/广告中的应用：文中明确指出需消除选择偏差才能保障模型泛化能力，这提示未来在设计类似决策系统时应更早融入因果推断组件以提升稳健性。