AFDBench: A Reasoning-First AI Scientist for National Weather Service Forecast Discussions
AFDBench is the first benchmark for evaluating generative meteorological reasoning, containing 7,732 expert-written Area Forecast Discussions (AFDs) from 13 NWS offices paired with real AI weather forecast inputs from Google's WeatherNext 2. Three complementary evaluation metrics are introduced: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot open-source LLMs perform poorly on Style-Align (~0.33) and o
Analysis
TL;DR
- AFDBench is the first benchmark for evaluating generative meteorological reasoning, containing 7,732 expert-written Area Forecast Discussions (AFDs) from 13 NWS offices paired with real AI weather forecast inputs from Google's WeatherNext 2.
- Three complementary evaluation metrics are introduced: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data).
- Zero-shot open-source LLMs perform poorly on Style-Align (~0.33) and only moderately on Input-Grounding (~0.88), failing to adopt the professional NWS register or faithfully use input data.
- Applying Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance nearly doubles Style-Align (0.318 → 0.619) and improves Input-Grounding (0.881 → 0.940) on held-out samples from unseen NWS offices.
- A 7B-parameter model fine-tuned with RL can be taught to write like a professional meteorologist and faithfully interpret AI-generated weather data.
Why It Matters
This work addresses the critical problem of LLM hallucination in high-stakes meteorological communication, where numerical inaccuracies can directly impact public safety. It demonstrates that reinforcement learning with domain-specific rewards can effectively align relatively small open-source models with professional domain expertise, offering a scalable blueprint for AI systems in other specialized scientific fields.
Technical Details
- Dataset: 7,732 expert-written AFDs from 13 National Weather Service offices, paired with structured AI weather forecast data from Google's WeatherNext 2.
- Metrics: Met-Align measures numerical accuracy of forecast values; Style-Align evaluates adherence to the professional NWS writing register; Input-Grounding quantifies fidelity to the source weather data.
- Method: Group Relative Policy Optimization (GRPO) applied to a 7B-parameter open-source LLM, with domain-specific reward functions targeting temperature accuracy, synoptic correctness, and format compliance.
- Evaluation: Zero-shot baseline performance was poor on Style-Align (~0.33) and moderate on Input-Grounding (~0.88); post-GRPO results on 1,033 held-out samples from two unseen NWS offices showed Style-Align improving to 0.619 and Input-Grounding to 0.940.
Industry Insight
- Domain-specific reinforcement learning with carefully designed reward functions can bridge the gap between general-purpose LLMs and professional domain expertise, even with modestly-sized models.
- The three-metric evaluation framework (numerical accuracy, style adherence, input grounding) provides a transferable template for benchmarking AI systems in other high-stakes scientific domains such as seismology, climate science, and medical reporting.
- Organizations deploying AI for public-facing scientific communication should prioritize input-grounding and style alignment alongside raw accuracy, as hallucinated numbers in meteorological contexts carry direct safety implications.
Disclaimer: The above content is generated by AI and is for reference only.