Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge
This paper examines temporal and cross-sectional biases in large language models' financial knowledge using extensive revenue data from U.S. companies.
What it examines
This paper studies how large language models (LLMs) show biased financial knowledge over time. It analyzes over 197k revenue questions using U.S. financial data and logistic regression to understand gaps in responses. The study aims to identify temporal and cross-sectional biases affecting LLM accuracy.
What it concludes
The study finds that LLMs perform better with recent data and larger firms, but they can also hallucinate facts. Future work should improve model accuracy and expand analysis to other areas. Applications include improved investment advice and better understanding of financial data biases in automated systems.
Evidence objects
The study reveals substantial temporal biases in LLM performance, with models excelling on post-1995 data following the SECs EDGAR filing system yet struggling and hallucinating details for older financial information.
key_findings bullet 1 · key_findings · validation V0
Surprisingly, even companies with seemingly correct revenue predictions experience simultaneous false data generation, highlighting LLMs paradox: they reliably capture recent figures yet produce consistently misleading numbers for major, renowned enterprises.
key_findings bullet 2 · key_findings · validation V0
Researchers introduce a novel Revenue Prompt Dataset of 200,000 questions, using logistic regression to link market capitalization, investor attention, and filing readability for quantifying retrograde knowledge bias among U.S. companies.
key_findings bullet 3 · key_findings · validation V0
Investigating LLM comprehension of historical financial data, this paper introduces a systematic, data-driven approach highlighting both temporal and cross-sectional biases, notably retrograde knowledge bias. Its originality stems from linking LLM performance with company characteristics, offering novel insights that impact financial modeling and investment decisions, rendering it a compelling, innovative read.
key_findings bullet 4 · key_findings · validation V0
Raw abstract and provenance
- … breadth of LLMs’ knowledge using financial data of US publicly traded companies by … Our results reveal that LLMs are less informed about past financial performance, but …
Source row: 312 · abstract type: snippet