The Open Source Economic Index of AI Adoption and Capability
Researchers have developed an open-source economic index to track how artificial intelligence, especially large language models, is used across jobs. They found AI adoption is highest in finance, computer science, and the arts, but unexpectedly low in legal services. By analyzing nearly 800,000 real AI chat logs and mapping them to job tasks, the study shows AI often struggles with detailed steps. Full job automation remains unlikely, as human oversight is still crucial for accuracy.
What it examines
This paper creates an open-source economic index to measure how and where AI, especially large language models, are used in different jobs. It also builds a benchmark to test how well AI can perform real work tasks, focusing on both adoption and capability across occupations.
What it concludes
The study finds that AI's ability to perform tasks is higher than its current adoption in workplaces. Human-AI collaboration works best, but full automation is rare. These tools can help policymakers, businesses, and researchers track AI’s workforce impact and guide responsible adoption. More data and tool coverage are needed.
Evidence objects
Researchers unveil an open-source economic index tracking AI adoption and capability across jobs, revealing highest uptake in finance, computer science, and the arts, but surprisingly low use in legal services.
key_findings bullet 1 · key_findings · validation V0
By mapping nearly 800,000 public AI chat conversations to O*NET job tasks and benchmarking AI agents on real workplace scenarios, the study exposes AIs strengths in high-level workflows but frequent errors in detailed steps.
key_findings bullet 2 · key_findings · validation V0
A key finding: AIs theoretical abilities outpace its real-world adoption, with full job automation unlikely soon as human-AI collaboration remains crucial for accuracy; limitations include tool coverage and outdated chat data.
key_findings bullet 3 · key_findings · validation V0
This paper presents an open-source economic index measuring AI adoption and capability across occupations using public LLM chat data and O*NET tasks. Its transparent, reproducible, and performance-based methodology addresses prior limitations, offering a novel, scalable framework. Highly relevant for analyzing AIs economic impact, it is compelling and original within its context.
key_findings bullet 4 · key_findings · validation V0
Raw abstract and provenance
- … -LLM chat data and O*NET tasks to replicate studies produced by frontier AI labs, finding that occupations in the finance, … ; we plot the AI agent capability and real-world AI …
Source row: 1981 · abstract type: snippet