LOB-Bench: Benchmarking Generative AI for Finance-an Application to Limit Order Book Data
LOB-Bench provides a comprehensive evaluation framework and Python package for assessing generative AI models on realistic limit order book simulation.
What it examines
This paper introduces LOB-Bench, a Python-based benchmark for evaluating generative AI models that simulate high-frequency limit order book data. The approach uses aggregated statistical scoring functions to compare generated and real market data, addressing noise, heavy tails, and error accumulation in autoregressive models.
What it concludes
LOB-Bench effectively benchmarks generative models and shows that autoregressive methods outperform traditional ones. Applications include realistic market simulation, improved trading strategy design, and counterfactual market analysis. Future work may enhance model robustness and extend the framework to other high-dimensional time-series tasks.
Evidence objects
LOB-Bench, an innovative benchmark tool, evaluates generative AI models for realistic limit order book data; autoregressive state-space models, particularly LOBS5 with 35 million parameters, outperform conventional methods under dynamic conditions.
key_findings bullet 1 · key_findings · validation V0
Researchers uncover an autoregressive trap causing minor prediction errors to cascade, while a comprehensive framework employing novel aggregator functions, $L_1$ norm, $Wasserstein-1$ distance, and adversarial scores enables detailed data comparisons.
key_findings bullet 2 · key_findings · validation V0
Despite promising results, models like RWKV variants show significant error divergences and struggle with diverse stock characteristics, underscoring challenges while offering insights, evaluation tools, and a basis for future research.
key_findings bullet 3 · key_findings · validation V0
Novel and impactful, the paper introduces a comprehensive benchmarking framework for generative limit order book models, employing innovative distributional and conditional comparisons with $L_1$ norm metrics and discriminator scores. Its systematic evaluation approach, highlighting pressing challenges like autoregressive sampling derailment, advances rigor, making it essential reading for financial market researchers.
key_findings bullet 4 · key_findings · validation V0
Raw abstract and provenance
Abstract: While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. To address this, we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative mess… ▽ More While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. To address this, we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format. Our framework measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting flexible multivariate statistical evaluation. The benchmark also includes features commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with scores from a trained discriminator network. Lastly, LOB-Bench contains "market impact metrics", i.e. the cross-correlations and price response functions for specific events in the data. We benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes. △ Less
Source row: 1235 · abstract type: unknown