VAN-Flow: Variance-Averse $n$-Step Offline Reinforcement Learning
for Sparse Long-Horizon Environments

Reliable-Action Selection for Generative Actors under Heterogeneous Offline Data

Guhyeon Kang Minhae Kwon*

Department of Electrical and Computer Engineering Sungkyunkwan University, Republic of Korea

{guhyeon.kang, minhae.kwon}@skku.edu

*Corresponding author

NeurIPS 2026

📄 Paper 💻 Code
A variance-averse expectation over categorical return distributions enables reliable-action selection for generative actors, stabilizing $n$-step offline RL in sparse, long-horizon environments with heterogeneous behavior data.
antmaze-teleport
VAN-Flow
Other method
puzzle 3x3
VAN-Flow
Other method

Abstract

Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$-value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$-step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean–variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than $40$ tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.

Variance-Averse Expectation

Why variance-averse is needed
Why Variance-Averse is Needed?

Under heterogeneous offline data, generative actors faithfully fit the dataset distribution and may reproduce unreliable action modes—action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. The expected $Q$-value alone cannot distinguish reliable from unreliable modes, since two state–action pairs with identical $\mathbb{E}[Q]$ can differ drastically in dispersion. This issue is amplified by $n$-step returns in sparse, long-horizon environments.

Variance-averse expectation
Variance-Averse Expectation

VAN-Flow introduces a variance-averse expectation $\mathcal{E}(Z)$ that smoothly reweights atom probabilities of a categorical return distribution via a CDF-based aversion intensity $\delta$, jointly favoring high-return and low-variance actions. Unlike CVaR or mean–variance objectives, $\mathcal{E}(Z)$ requires no hard truncation or auxiliary penalty term and is natively compatible with categorical distributional critics.

VAN-Flow: Categorical Critic & Flow-based Actor

Categorical Critic
Categorical critic image
Categorical Distributional Critic

The critic $Z_\psi(\boldsymbol{s}_t,\boldsymbol{a}_t)$ models the full $n$-step return distribution over fixed categorical atoms, exposing the variance structure that a regression-based critic cannot. The critic is trained against the projected $n$-step distributional Bellman target with cross-entropy loss, using a variance-averse target action obtained via rejection sampling.

Flow-based Actor
Flow actor image
Flow-based Actor with $Q$-Guidance

The actor is parameterized by a state-conditioned velocity field and trained via flow matching with a single-step Euler $Q$-guidance term using the variance-averse expectation $\mathcal{E}(Z_\psi)$. At inference, $M$ candidate actions are generated and re-ranked via rejection sampling, selecting actions with high expected return and low return dispersion.

Simulation Results

[Q1.] Does the variance-averse expectation enable stable $n$-step learning in sparse, long-horizon offline RL?
AntMaze results table
Offline performance on D4RL AntMaze tasks. Scores are normalized following the D4RL protocol. Entries marked "--" indicate results not reported in prior work. VAN-Flow consistently achieves strong results from umaze to large-diverse, indicating that explicit return-variance control is crucial for stably leveraging $n$-step returns in offline RL.
[Q2.] Does VAN-Flow outperform existing flow-based policies under heterogeneous, high-variance datasets?
OGBench results table
Performance comparison on OGBench. Task-wise scores across Gaussian-based and Flow-based baselines. On high-variance datasets (e.g., antmaze-large-explore), naive flow-based $n$-step methods (FQL-$n$, BFN-$n$) collapse far more severely than their Gaussian counterpart, while VAN-Flow remains robust.
[Q3.] Does VAN-Flow serve as a strong initialization for off-to-online fine-tuning?
Offline-to-online results table
Offline-to-online fine-tuning performance. Gray scores indicate offline performance before fine-tuning; bold scores indicate performance after online fine-tuning. VAN-Flow adapts stably online and consistently improves with additional interaction, matching or surpassing the state-of-the-art horizon-reduction baseline.