One Spectrum, Two Resources: Data–Memory Scaling in Autoregressive Prediction

Abstract

How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing μ for the resulting energy spectrum, we prove the minimax law R*value(n, B) ≍ Φ_μ(n^(−1)) + Φ_μ(τ_B), where Φ_μ(t) = ∫min{x, t}μ(dx), for n prediction blocks and a learned state with at most 2^B values. Data set the resolution 1/n; memory sets the level τ_B reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy–dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query–key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute–precision laws under two-sided arithmetic assumptions. Experiments recover the data–memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.