What Pretraining and Midtraining Make Learnable from Rewards?

Abstract

A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how task-independent prediction supplies the missing information and turns it into computation that rewards can use. In sequential state computation and contextual memory, we characterize exactly which mechanism-rule worlds agree on every training reward yet demand opposite held-out answers. Source observations expose the mechanism before its effects cancel in training rewards. Finite sampled Adam realizes this division of labor from random initialization in the same parameters: the sequential learner acquires execution and selects the task binding in two reward updates, reaching held-out success at least 7/8; the contextual learner acquires retrieval and an input-dependent task rule within explicit finite budgets, reaching 19/20, while randomly initialized frozen attention stays near chance. Across eight worlds, source midtraining from pretrained Qwen2.5 checkpoints supplies executable state predictions and Memory responses sensitive to stored values. With task-independent first-operation supervision, Sequential models reach 82.61% success against 44.15% for a private-random source control, with a 67.57-point task-selectivity advantage. Memory replay retains 92.21% original/value-flip pair accuracy during reward adaptation, compared with 0% under pure reward training. An independent eight-world Memory confirmation reaches 75.32% task success versus 49.86% after matched alternative-retrieval training. GSM8K and HotpotQA separate accuracy at reward entry, subsequent gain and final performance. The results connect three steps in learning from rewards: identifying a mechanism, acquiring its computation and learning its task-specific use.