Personalizing Large Language Model Agents with Small Policy Models
FABLE (Factorized Adaptive Bandit Layer for Execution) is introduced as a lightweight policy layer that personalizes frozen LLM agents without fine-tuning the underlying model The approach factorizes execution decisions into memory, information-acquisition, and response components, enabling feedback-driven adaptation of user-specific preferences Bayesian contextual Thompson sampling learns residual preferences relative to a fixed default-and-cost score, with theoretical regret bounds under a lin
Analysis
TL;DR
- FABLE (Factorized Adaptive Bandit Layer for Execution) is introduced as a lightweight policy layer that personalizes frozen LLM agents without fine-tuning the underlying model
- The approach factorizes execution decisions into memory, information-acquisition, and response components, enabling feedback-driven adaptation of user-specific preferences
- Bayesian contextual Thompson sampling learns residual preferences relative to a fixed default-and-cost score, with theoretical regret bounds under a linear residual-reward model
- The method includes feasibility filtering, characterization of unidentifiable preferences under persistent constraints, and anytime-valid false-promotion control
- Empirical evaluations across personalized reasoning, controlled feedback, and executable tool-use tasks demonstrate improved preference-sensitive behaviors while maintaining competitive end-to-end task performance
Why It Matters
This work addresses a critical bottleneck in deploying LLM agents: the difficulty of personalizing execution decisions without expensive fine-tuning or access to proprietary model internals. By treating personalization as an online learning problem with scalar feedback, FABLE offers a practical, model-agnostic solution that can be layered onto existing black-box agents. This is particularly relevant as the industry moves toward deploying personalized AI assistants at scale.
Technical Details
- FABLE operates as an external policy layer that wraps a potentially black-box host agent, learning per-user execution policies from scalar feedback observed only for executed actions
- The framework factorizes agent decisions into three components—memory retrieval, information-acquisition (e.g., asking clarifying questions), and response generation—allowing feedback to selectively update related choices
- Action exploration is constrained through an externally specified feasible set, and user-specific preferences are learned as residuals against a fixed default-and-cost baseline using Bayesian contextual Thompson sampling
- Theoretical guarantees include an expected-regret bound against the best feasible action under a linear residual-reward model, along with characterization of preferences that remain unidentifiable under persistent feasibility constraints
- The method provides anytime-valid false-promotion control to prevent spurious preference learning, and evaluations span personalized-reasoning benchmarks, controlled-feedback scenarios, and executable tool-use tasks
Industry Insight
- FABLE's model-agnostic design enables personalization of proprietary or closed-source LLM agents without requiring fine-tuning access, lowering the barrier for enterprise deployment of customized AI assistants
- The factorization of execution decisions into independent components suggests a modular architecture for personalization that could be extended to other agent capabilities beyond memory and tool use
- The theoretical framework combining regret bounds with false-promotion control offers a principled approach to safe online personalization, which could inform industry standards for user-specific AI behavior adaptation
Disclaimer: The above content is generated by AI and is for reference only.