Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics
Researchers have unveiled Structural Reinforcement Learning (SRL), a new method for solving complex macroeconomic models with diverse agents and overall risk. By using low-dimensional equilibrium prices instead of high-dimensional agent distributions, SRL bypasses the difficult Master equation. The approach, powered by a structural policy gradient algorithm, quickly finds global solutions in models like Huggett, Krusell-Smith, and HANK. Surprisingly, policies based on current prices are nearly as effective as those using longer price histories, streamlining analysis.
What it examines
This paper introduces a new method called structural reinforcement learning (SRL) to solve complex macroeconomic models with many different agents and aggregate risk. Instead of tracking every agent, it uses low-dimensional prices as state variables, allowing agents to learn equilibrium price dynamics efficiently from simulated data.
What it concludes
SRL enables fast and accurate solutions for models that were previously too difficult or slow to solve, such as those with tricky market-clearing conditions. This approach can be used for economic policy analysis, financial crisis modeling, and studying how expectations form, with potential for further research in online learning and behavioral economics.
Evidence objects
Researchers unveil Structural Reinforcement Learning (SRL), a novel method that replaces complex agent distributions with low-dimensional equilibrium prices, enabling agents to learn price dynamics directly from simulated data and bypassing the Master equation.
key_findings bullet 1 · key_findings · validation V0
SRL, powered by a structural policy gradient algorithm, delivers global solutions to challenging macroeconomic models like Huggett, Krusell-Smith, and HANK in minutesdramatically faster than traditional approaches and highly scalable.
key_findings bullet 2 · key_findings · validation V0
Surprisingly, conditioning policies on current prices is nearly as effective as using longer price histories, suggesting most relevant information is captured; however, SRL relies on restricted perceptions equilibrium, potentially limiting rational expectations.
key_findings bullet 3 · key_findings · validation V0
This paper presents a novel 'structural reinforcement learning' (SRL) method, uniquely combining reinforcement learning with agents' structural knowledge to efficiently solve heterogeneous agent macroeconomic models with aggregate risk. By replacing high-dimensional distributions with low-dimensional prices, SRL bypasses the Master equation, offering a compelling, impactful advance for economics and computational finance.
key_findings bullet 4 · key_findings · validation V0
Raw abstract and provenance
Abstract: We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agent… ▽ More We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes. △ Less
Source row: 1871 · abstract type: unknown