Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance
A new study shows that merging large language models (LLMs) continually pre-trained (CPT) on different skills, such as finance, math, and Japanese, can create powerful multi-skilled models for finance tasks. The research uses an evaluation framework on 18 tasks from 8 datasets. Notably, adding a Japanese CPT model improves English performance, suggesting cross-lingual benefits. Task Arithmetic merging gives the best results but is hard to tune. Merging three CPTs often causes interference and lower performance.
What it examines
This study explores merging large language models (LLMs) that have been continually pre-trained in finance, math, and Japanese. It introduces a three-stage framework to analyze knowledge recovery, complementarity, and emergent skills, using three merging methods and a financial benchmark of 18 tasks from 8 datasets.
What it concludes
Merging specialized LLMs can restore lost knowledge and create new cross-domain abilities, especially in finance. Task Arithmetic offers high gains but needs careful tuning, while TIES is more stable. Applications include building multi-skilled financial AI tools. Future work should test broader domains and larger models for wider use.
Evidence objects
Researchers show that merging large language models (LLMs) pre-trained on different skillslike finance, math, and Japanesecreates powerful, multi-skilled models, with surprising cross-lingual benefits and improved reasoning on complex financial tasks.
key_findings bullet 1 · key_findings · validation V0
A three-stage evaluation using 18 tasks from 8 financial datasets reveals that combining two CPT experts can restore lost knowledge and spark new abilities, but merging three often causes interference and lower performance.
key_findings bullet 2 · key_findings · validation V0
Task Arithmetic merging method delivers the highest gains but is tuning-sensitive, while TIES is more stable; however, findings are limited to one model family and domain, raising questions about broader applicability.
key_findings bullet 3 · key_findings · validation V0
This paper uniquely analyzes continual pretraining (CPT) model merging for financial LLMs, introducing a three-stage evaluation and a benchmark spanning 18 tasks from 8 datasets. While merging techniques are adapted, their novel application reveals knowledge recovery, complementarity, and emergent cross-domain skills, offering significant practical impact for financial AI and NLP.
key_findings bullet 4 · key_findings · validation V0
Raw abstract and provenance
- … like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and … We address this gap by creating financial LLMs from experts in finance, math, and …
Source row: 1326 · abstract type: snippet