Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)
A pipeline was built to automate financial news summarization by pulling data from News API, Wikipedia, and Yahoo Finance for ten major companies (AAPL, MSFT, GOOGL, AMZN, META, TSLA, JPM, NVDA, WMT, DIS) Stock price numerical tables were converted into natural language narratives via a custom template, since LLMs cannot directly process tabular data Falcon-7B-Instruct with Summarize Chains achieved the best summarization results, covering news events accurately and coherently RAG with FAISS cau
Analysis
TL;DR
- A pipeline was built to automate financial news summarization by pulling data from News API, Wikipedia, and Yahoo Finance for ten major companies (AAPL, MSFT, GOOGL, AMZN, META, TSLA, JPM, NVDA, WMT, DIS)
- Stock price numerical tables were converted into natural language narratives via a custom template, since LLMs cannot directly process tabular data
- Falcon-7B-Instruct with Summarize Chains achieved the best summarization results, covering news events accurately and coherently
- RAG with FAISS caused severe repetition in Falcon and hallucinated facts in BART-Large when the retrieval count (k) was large
- Both LLM-based approaches outperformed a simple Lead-3 baseline on ROUGE-1, and a Streamlit dashboard was built for interactive stock visualization
Why It Matters
This study provides early empirical evidence on the practical limitations of RAG for financial summarization, particularly hallucination risks in smaller open-source models — a concern that remains highly relevant as RAG-based financial tools have since become widespread. It offers practitioners a concrete comparison of summarization strategies and highlights the importance of handling structured numerical data in LLM pipelines.
Technical Details
- Data pipeline: Integrated three sources — News API for articles, Wikipedia for company background, and Yahoo Finance for stock price data across ten major companies
- Numerical-to-text conversion: A template-based approach was developed to transform stock price tables into natural language narratives, addressing the limitation that LLMs cannot directly process numerical tables
- Models evaluated: Three open-source models for news summarization (Falcon-7B-Instruct, DistilBART-CNN-12-6, BART-Large-XSum) and GPT (text-davinci-003) for stock summaries
- Summarization approaches: Compared Summarize Chains versus Retrieval-Augmented Generation with FAISS, evaluated using ROUGE-1 against a Lead-3 baseline
- Failure modes documented: RAG induced severe repetition in Falcon-7B and factual hallucinations in BART-Large when k (number of retrieved documents) was large
Industry Insight
- RAG is not a universal solution — practitioners should be cautious about blindly applying RAG to smaller models, as retrieval can amplify hallucination and repetition rather than improve output quality
- The template-based approach for converting structured financial data into natural language offers a practical, low-cost pattern for any domain requiring LLM processing of numerical tables
- The persistent relevance of these 2023 findings suggests that even as RAG tools have matured, careful evaluation of retrieval parameters and model selection remains critical for financial NLP applications
Disclaimer: The above content is generated by AI and is for reference only.