Market data poses a unique modeling challenge because it blends continuous and discrete elements. While individual attributes such as price can vary continuously, the order book evolves through discrete events—order placements, cancellations, and executions—that occur at irregular intervals. This dual nature means that traditional generative models, which assume either purely continuous or purely categorical data, struggle to capture the microstructure of financial time series. In quantitative finance, practitioners typically forecast future prices from historical streams of market data, but generating realistic synthetic data requires preserving both the continuous dynamics of variables like price and timing and the discrete nature of event types. A Jane Street intern project led by Kavish explored whether autoregressive diffusion could bridge this gap, adapting techniques that have succeeded in image, video, robotics, and audio generation to the specific characteristics of order book events.
The team built an encoder‑diffuser architecture: a causally masked transformer encoder creates a latent embedding for each position in the sequence, which feeds a small categorical head predicting whether the next event is a trade or a best‑bid/offer (BBO) update, while a diffusion head generates continuous targets such as price, elapsed time, and size. Initial experiments used a Denoising Diffusion Probabilistic Model (DDPM) with a 1,000‑step cosine schedule and DDIM sampling, but this configuration proved unstable—generation trajectories quickly diverged, with 88‑95% of values exceeding eight standard deviations from the mean across all continuous targets. As one Hacker News comment noted, 'DDPM breaks because order book event intervals are discrete point masses, not smooth Gaussians.' Switching to flow matching, which interpolates linearly between noise and data and trains the network to predict velocity along that line, provided stable performance out‑of‑the‑box. To handle the inherent discontinuities in market data (e.g., timing clustered around zero and whole‑number seconds, prices concentrating at bid/ask midpoints), Kavish refined the categorical head into 20 classes representing common event subtypes and developed an 'atom smoothing' procedure that smooths spiky distributions then re‑sculpts the spikes to preserve probability mass without creating artificial discontinuities.
This image illustrates the encoder‑diffuser architecture described in the article, where a transformer encoder predicts event types and a diffusion head generates continuous variables such as price and time.
Evaluation showed that the flow‑matching model with atom squeezing produced marginal distributions much closer to the true data and generated more realistic samples when discriminated by a classifier trained to distinguish model output from real events. However, autoregressive rollouts revealed that the synthetic trajectories degrade over longer horizons—the spread in synthetic CME US prices widens relative to real data, indicating accumulating errors. Consequently, the model is not yet accurate enough to serve as a practical generator of market data. The internship clarified two open questions for future work: how much discreteness should be explicitly modeled versus left to the diffusion process, and how to best represent and smooth feature distributions so that diffusion can effectively learn the complex, spiky statistics of order book data. As another comment observed, 'The real story here is this wonderful exposition in applying diffusion models to a time series data that is neither discrete nor continuous.'