Reinforcement Learning via
Fixed-point Flow Matching

Denis Blessing · Onur Celik · Gerhard Neumann

Autonomous Learning Robots Lab (ALR), Karlsruhe Institute of Technology (KIT)

In maximum-entropy and behaviour-constrained RL the optimal policy is a Boltzmann distribution. Flow policies trained with flow matching can represent such complex, multimodal distributions, but flow matching needs samples from the distribution it is fitting, and in RL there are none. Fixed-point flow matching (FFM) is a data-free procedure for training and fine-tuning flow policies in RL: it keeps the simple flow-matching regression and replaces the data-derived target with one read off the critic.

Method

The FFM policy-improvement step: generate, label, interpolate, regress.
The FFM policy-improvement step. The optimal policy π+ ∝ eQ/λ is a fixed point of this loop.
1 · Generate Clean actions X1 are sampled from the current policy by integrating the flow ODE — forward only, no gradients.
2 · Label The critic's action gradient ∇xQ(s, X1) is the regression target; for a Boltzmann target it is the score the velocity field must follow.
3 · Interpolate Fresh noise ε and X1 give a noised action Xt = α(t)X1 + β(t)ε.
4 · Regress A flow-matching-style regression pulls the velocity field vθ(Xt, t | s) toward the FFM target.

Learned policies

One successful offline-to-online rollout per OGBench domain, from the final 1.5M-step checkpoint. Videos loop; hover for controls.

Offline-to-online results on OGBench

FFM fine-tunes a BC-pretrained flow policy for 1M offline steps and then 500K online environment steps, against seven baselines under an identical protocol.

Per-domain success-rate curves of FFM and seven baselines over 1M offline and 500K online steps on ten OGBench domains.
Per-domain aggregated offline-to-online success rate (mean over the five tasks and all seeds) against training steps in millions. Offline (≤ 1M) and online (> 1M) phases are shaded; FFM (ours) is drawn in bold with a ±1 standard-deviation band.

FFM reaches the best average success after online fine-tuning (0.95 vs. 0.93 for the next-best method) and is the only method at or near the top on every locomotion domain. The same update also trains flow policies entirely from scratch — see the on-policy results below.

On-Policy Maximum Entropy RL

FFM is not limited to fine-tuning a pretrained policy: in the on-policy maximum-entropy setting the same critic-derived regression learns flow policies from scratch, as we demonstrate on the 20 MJX MuJoCo Playground tasks and on MJX's four humanoid joystick tasks, against eight baselines (10 seeds per task, 50M environment steps). FFM performs on par with the strongest baselines on the MJX Playground tasks, and performs favorably on the humanoid tasks in terms of both end performance and learning speed. On playground tasks where exploration is the bottleneck, FFM additionally benefits from temporally correlated Ornstein–Uhlenbeck exploration noise: FFM + OU uses the best OU time constant on those tasks and plain FFM elsewhere.

Aggregated IQM learning curves and final IQMs of FFM, FFM + OU and eight baselines on the 20 MJX Playground tasks and the four MJX humanoid tasks.
Aggregate results. IQM learning curves and final IQMs on the 20 MJX Playground tasks (left pair, return normalised by the suite maximum) and the four MJX humanoid joystick tasks (right pair). Bands and error bars are 95% stratified bootstrap confidence intervals (Agarwal et al., 2021).
Per-task IQM learning curves of FFM, FFM + OU and eight baselines on each of the 20 MJX Playground tasks.
Per-task curves, MJX Playground. IQM over the 10 seeds per task with 95% bootstrap confidence intervals; panels where FFM + OU uses an OU arm are annotated with the chosen time constant τou.
Per-task IQM learning curves of FFM and eight baselines on each of the four MJX humanoid joystick tasks.
Per-task curves, MJX humanoids. IQM over the 10 seeds per task with 95% bootstrap confidence intervals.

Paper

Reinforcement Learning via Fixed-point Flow Matching. Denis Blessing, Onur Celik, Gerhard Neumann. arXiv:2610.12848, 2026. [PDF]

If you find this work useful, please cite:

@misc{blessing2026reinforcementlearningfixedpointflow,
  title={Reinforcement Learning via Fixed-point Flow Matching},
  author={Denis Blessing and Onur Celik and Gerhard Neumann},
  year={2026},
  eprint={2610.12848},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.12848},
}

Code

The code is available at github.com/DenisBless/FFM_RL.