Reliable-Action Selection for Generative Actors under Heterogeneous Offline Data
Department of Electrical and Computer Engineering Sungkyunkwan University, Republic of Korea
{guhyeon.kang, minhae.kwon}@skku.edu
NeurIPS 2026
Under heterogeneous offline data, generative actors faithfully fit the dataset distribution and may reproduce unreliable action modes—action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. The expected $Q$-value alone cannot distinguish reliable from unreliable modes, since two state–action pairs with identical $\mathbb{E}[Q]$ can differ drastically in dispersion. This issue is amplified by $n$-step returns in sparse, long-horizon environments.
VAN-Flow introduces a variance-averse expectation $\mathcal{E}(Z)$ that smoothly reweights atom probabilities of a categorical return distribution via a CDF-based aversion intensity $\delta$, jointly favoring high-return and low-variance actions. Unlike CVaR or mean–variance objectives, $\mathcal{E}(Z)$ requires no hard truncation or auxiliary penalty term and is natively compatible with categorical distributional critics.
The critic $Z_\psi(\boldsymbol{s}_t,\boldsymbol{a}_t)$ models the full $n$-step return distribution over fixed categorical atoms, exposing the variance structure that a regression-based critic cannot. The critic is trained against the projected $n$-step distributional Bellman target with cross-entropy loss, using a variance-averse target action obtained via rejection sampling.
The actor is parameterized by a state-conditioned velocity field and trained via flow matching with a single-step Euler $Q$-guidance term using the variance-averse expectation $\mathcal{E}(Z_\psi)$. At inference, $M$ candidate actions are generated and re-ranked via rejection sampling, selecting actions with high expected return and low return dispersion.