Simple Actors and Deep Critics for Scalable Reinforcement Learning (LAC)

Abstract

Expressive actors are expensive at every step. The critic isn’t.

Diffusion and flow-matching actors capture multimodal behavior in offline datasets, but they need multiple denoising or integration steps for every action at deployment. We revisit where capacity should go in an offline actor–critic method.

The critic exists only to shape the policy gradient during training and is discarded once training ends, while the actor runs at every decision step. So capacity in the critic is paid once; capacity in the actor is paid forever. LAC pairs a lightweight deterministic actor with a deep, well-trained critic—reaching an average success rate of 70 on OGBench against 36 for the strongest baseline, while cutting per-action latency by up to 4× relative to multi-step generative actors and matching one-step distilled policies without any distillation stage.

Is the architectural budget in offline actor–critic methods allocated to the right network?

Performance vs. inference latency on humanoidmaze-medium — **Performance vs. inference latency on `humanoidmaze-medium`.** Iterative generative actors (IDQL, IFQL, FBRAC, FAWAC) require multiple denoising or flow-integration steps per action. LAC (star) acts in a single forward pass, reaching a higher success rate at lower latency.

The Idea

Move the burden to the network that’s free at inference.

Rather than making expressive actors cheaper, LAC asks whether they’re necessary at all once the critic is strong enough. A single forward pass through a small MLP produces each action.

Actor

Light

deterministic MLP · 1 forward pass

Critic

Deep

32-layer ResMLP · discarded after training

Why deep critics fail offline

Three failure modes, three ingredients.

Naively deepening an MLP critic destabilizes value learning. The failure isn’t one problem—it’s three, each on a different axis, each needing its own remedy. No single ingredient is sufficient.

F1 → I1

Optimization

Gradients fail to propagate through many layers—vanishing gradients, dead activations, ill-conditioned Hessians.

Residual MLP backbone with LayerNorm restores gradient flow.

F2 → I2

Bootstrap noise

A deep critic fits its own noisy predictions; the 1-step bootstrap folds that noise back into the target every iteration, compounding errors.

n-step targets (n=4) swap self-prediction for real reward, shrinking the loop by γⁿ.

F3 → I3

Value-range drift

Unbounded MSE lets Q-values drift off any reasonable scale, causing abrupt mid-training collapse after hundreds of thousands of healthy steps.

Categorical cross-entropy over a bounded support turns regression into classification.

Benchmark

Evaluated on OGBench.

We evaluate on 7 challenging environments spanning long-horizon locomotion navigation (antmaze, humanoidmaze) and multi-step / combinatorial manipulation (scene, puzzle), for 35 tasks in total (5 goals per environment).

The 7 OGBench environments — **The OGBench environments used in our experiments.** Locomotion mazes pair an 8-DoF quadruped (`antmaze`) and a 21-DoF whole-body humanoid (`humanoidmaze`) on shared layouts; manipulation spans pick-place / tool-use (`scene`) and sliding-tile puzzles (`puzzle`).

Results

Critic depth pays off. Actor size doesn’t.

Critic depth scaling on hum-large — **Critic scales stably with depth.** On `hum-large`, the full LAC recipe climbs from near zero at 4–8 layers to ~80% at depth 32, while plain-MLP and residual-only baselines plateau or fail.

Actor scaling — **Actor capacity saturates fast.** With the critic fixed, performance plateaus at ~10⁵ actor parameters—LAC-S and LAC-L bracket the plateau. Extra actor capacity buys nothing but latency.

MSE vs categorical training curves — **n-step alone isn’t enough.** With a 32-layer critic and n=4, the MSE-regression critic collapses mid-training while the categorical critic of LAC stays stable. The categorical loss bounds the targets the bootstrap loop can generate.

n-step horizon vs critic depth heatmap — **n-step is an enabling condition for depth.** Sweeping depth × n on `hum-large`: all horizons fail while the critic is shallow, but the rows separate from depth 16 onward—n ≥ 2 stays stable while n = 1 degrades with further depth, evidencing the bootstrap-noise mechanism.

Spatial Q-value landscape, depth 4 vs 32 — **Depth sharpens the value landscape.** Averaged predicted Q-value over spatial locations on `hum-large`. The shallow 4-layer critic (left) is blurred; the 32-layer LAC critic (right) yields a sharp gradient aligned with distance to the goal (star).

avg success rate on OGBench (LAC-L), vs 36 for the best baseline

4×

lower per-action inference latency than iterative generative actors

0.24ms

LAC-S per-action latency on an RTX 3090 (batch size 1)—fastest in the comparison

Drop-in compatibility

The critic recipe improves any actor.

Swapping in the LAC critic—while keeping each baseline’s original actor and policy-extraction objective—consistently boosts performance on hum-medium (task1): TD3+BC +36, FAWAC +24, FQL +38, IDQL +30, CAC +59. The inference savings aren’t paid for by an algorithm restriction; the same recipe accelerates and improves a broad family of methods.

Cite

BibTeX

@inproceedings{kang2026simple,
  title     = {Simple Actors and Deep Critics for Scalable
               Reinforcement Learning},
  author    = {Kang, Guhyeon and Lee, Jaehwi and Kwon, Minhae},
  booktitle = {Proceedings of the 35th ACM International Conference
               on Information and Knowledge Management (CIKM '26)},
  year      = {2026},
  doi       = {10.1145/3799682.3840726}
}