We spent the last few days testing JEV internally. We tried to reproduce it, break it, and improve it.
5 things we learned:
1. JEV’s core insight is simple - and probably right. If the answer space is bounded, don’t generate the answer token by token. Remove autoregressive decoding and you can remove 1–2 orders of magnitude of inference work.
2. RLCD isn’t the moat. Data might be. Laya already open-sourced the implementation + weights. The missing piece is the synthetic data recipe. And our experiments strongly point in the same direction.
3. “Open source already beat JEV” is a benchmark illusion.
Same checkpoint:
In-distribution: 0.769
OOD: 0.541
Change the distribution and the apparent breakthrough largely disappears.
4. Compute-optimal ≠ learnability-optimal.
We moved state outside the problem sequence to save compute. Result: -27 points. Likely because we saved FLOPs while throwing away useful MLM pretraining priors.
Then we tested 6 more ideas in one night: anchor selection, continuity smoothing, asymmetric windows, bucketed temperature, two-stage retrieval, full RLCD.
All failed or failed to generalize. Only one thing consistently worked: More data.
1,200 → 123,475 examples
OOD: 0.4069 → 0.5498
5. The weirdest result: A decoder with zero additional training still beat every trained small encoder:
0.5671 vs 0.5628 / 0.5411 / 0.4069
Our takeaway: JEV’s breakthrough may not be a particular architecture or training algorithm.
It may be a much simpler idea: When the answer space is bounded, search it. Don’t generate it. And right now, the biggest bottleneck to making that work broadly doesn’t look like architecture.
It looks like data.