From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls
The paper compares two function-surface representation approaches for SLM-based vehicle agents: Functional Tokens (FT) and Schema-in-Prompt (SIP) A new benchmark of 9,822 single-turn examples across 79 Android Automotive functions was introduced, including held-out functions and refusal cases On seen functions, model scaling provides limited benefit (270M matches 1.7B; peak performance at 0.6B), but SIP generalizes to unseen functions while FT achieves zero accuracy by construction FT risks invo
Analysis
TL;DR
- The paper compares two function-surface representation approaches for SLM-based vehicle agents: Functional Tokens (FT) and Schema-in-Prompt (SIP)
- A new benchmark of 9,822 single-turn examples across 79 Android Automotive functions was introduced, including held-out functions and refusal cases
- On seen functions, model scaling provides limited benefit (270M matches 1.7B; peak performance at 0.6B), but SIP generalizes to unseen functions while FT achieves zero accuracy by construction
- FT risks invoking unavailable functions on out-of-scope requests, whereas SIP more reliably refuses based on offered functions, at the cost of higher memory and latency
- The key finding is that function-surface representation, not model scale alone, determines the capabilities and failure modes of SLM-based vehicle function calling
Why It Matters
This research directly addresses a critical deployment challenge for in-vehicle AI assistants: how to effectively represent function catalogs for resource-constrained small language models running on-device. For AI practitioners building agent systems, the findings provide actionable guidance on the trade-offs between compact token-based representations and flexible schema-based prompting, with implications for reliability, generalization, and system design in safety-critical domains like automotive.
Technical Details
- Functional Tokens (FT): Each vehicle function is represented by a dedicated token, enabling compact inference but restricting the model to functions seen during training only
- Schema-in-Prompt (SIP): Function schemas are provided directly in the prompt, enabling generalization to unseen functions at the cost of longer prompts and higher inference overhead
- Benchmark: 9,822 single-turn examples spanning 79 vehicle functions derived from Android Automotive, with held-out functions and out-of-scope refusal requests
- Models evaluated: Four SLMs ranging from 270M to 1.7B parameters, all under matched fine-tuning conditions
- Theoretical analysis: Explains how SIP enables generalization to unseen functions and why longer schema contexts increase inference cost
Industry Insight
- For on-device automotive AI, a ~0.6B parameter model with SIP may offer the best accuracy-latency trade-off for known functions, while SIP is essential when the function catalog needs to evolve without retraining
- System designers should prioritize function-surface representation strategy over simply scaling model size when deploying SLMs for tool-use agents in constrained environments
- The zero generalization of FT to unseen functions highlights a hard limitation of token-based function encoding, suggesting that any production vehicle assistant requiring dynamic function updates should adopt SIP or a hybrid approach despite the latency cost
Disclaimer: The above content is generated by AI and is for reference only.