Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks
Benchmarking zero-shot performance of ChatGPT and other LLMs against fine-tuned models for financial NLP tasks.
What it examines
This paper investigates the zero-shot performance of large language models (LLMs) like ChatGPT in financial NLP tasks, comparing them with fine-tuned models like RoBERTa. It aims to address data annotation, performance gaps, and the feasibility of using generative models in finance.
What it concludes
The research underscores ChatGPT's potential in financial NLP tasks despite its limitations. Future work should address contamination issues and labeling time challenges. Potential applications include financial sentiment analysis, numerical claim detection, and named entity recognition in financial texts.
Evidence objects
The research underscores ChatGPT's potential in financial NLP tasks despite its limitations. Future work should address contamination issues and labeling time challenges. Potential applications include financial sentiment analysis, numerical claim detection, and named entity recognition in financial texts.
key_findings bullet 1 · key_findings · validation V0
Raw abstract and provenance
Abstract: Recently large language models (LLMs) like ChatGPT have shown impressive performance on many natural language processing tasks with zero-shot. In this paper, we investigate the effectiveness of zero-shot LLMs in the financial domain. We compare the performance of ChatGPT along with some open-source generative LLMs in zero-shot mode with RoBERTa fine-tuned on annotated data. We address three inter-… ▽ More Recently large language models (LLMs) like ChatGPT have shown impressive performance on many natural language processing tasks with zero-shot. In this paper, we investigate the effectiveness of zero-shot LLMs in the financial domain. We compare the performance of ChatGPT along with some open-source generative LLMs in zero-shot mode with RoBERTa fine-tuned on annotated data. We address three inter-related research questions on data annotation, performance gaps, and the feasibility of employing generative models in the finance domain. Our findings demonstrate that ChatGPT performs well even without labeled data but fine-tuned models generally outperform it. Our research also highlights how annotating with generative models can be time-intensive. Our codebase is publicly available on GitHub under CC BY-NC 4.0 license. △ Less
Source row: 2177 · abstract type: unknown