What Are RL Environments? Rubrics, Verifiable Rewards, and How Agents Learn at Scale
Quick Answer / TL;DR
A reinforcement learning environment is the system an agent acts inside of: it takes the agent's action, computes a reward, and hands back whatever the agent observes next, on repeat, for as many rollouts as you can afford to run. The interesting part is where the reward comes from. You can pay a human expert to grade every single attempt, which is accurate but slow and expensive, or you can build an environment that defines the grade (reward) itself. The environment does this from the rules of the task, a test suite, or a rubric, which define what good looks like. Once a reward is computable, the cost to run it millions of times goes down dramatically, which is the entire reason RL environments are how modern agents get their training signal at scale.
RL Environments
An RL environment can be understood as a dynamic sandbox which reacts to actions being taken inside of it. This enables an Agent to observe the consequences of its actions and iteratively learn from these observations to refine its ability to pick the right actions. Once defined and implemented, nothing about this process requires human intervention.
An environment can compute a reward however its designer wants: by checking the literal rules of a game, by running a test suite against generated code, by scoring a response against a checklist, or by asking something else to judge the result. The mechanics of the loop are the same either way. What changes is how expensive, how accurate, and how gameable that reward turns out to be.
A Chess Example
Consider training an agent to play chess by paying a grandmaster to look at every move it makes and grade it. That would work, in the sense that the grades would be accurate. It would also be extremely slow and expensive, since a strong agent needs to see many millions of positions before it develops good judgment, and no human grading operation scales to that.
In this example, there are clear rules to the game and you can write a script which tells you, with total precision, whether a game was won, lost, or drawn. Instead of asking an expert to grade each move, you let the agent play the game out and compute the reward from what actually happened. This is the idea behind reinforcement learning with verifiable rewards, shortened to RLVR: wherever success has a clean, checkable definition, whether that is a won chess game, a passing test suite, or a math answer that matches the known solution, you do not need a human in the loop at all. You need an environment that can play the situation out and check the result.
The scalability aspect is the reason RLVR is widely used for training agents on math and coding tasks specifically, since both have an answer you can check mechanically.
What are Rubrics
Most real tasks are not games with clear winners and loosers. Writing a good email, answering an open-ended question well, or completing a multi-step task safely does not reduce to a single checkable fact the way a chess outcome does. There is often no one right answer to compare against, but there is usually still a decomposable notion of what good looks like. That is what a rubric is: a structured, named list of criteria that can each be checked automatically.
A rubric for a customer support response might include whether it addressed the actual question asked, whether it stayed within policy, and whether it was appropriately concise, each scored separately and then combined into one reward. That is a meaningfully different thing from a single human or model saying "this response gets a 7 out of 10," because a rubric tells you exactly which part of the response was weak, and because each criterion, once written, can be checked automatically at scale. Reward signals may designed in several ways and are domain dependent, but 3 common categories come frequently.
Verifiable
A won chess game, a passing test suite, a math answer matching the solution. Cheapest to compute and hardest to game, but only available when success has one clear and checkable definition.
Rubric-based
A named checklist of criteria, each checked automatically, then combined. Built for tasks with no single right answer but a decomposable notion of quality.
Freeform judgment
One (opaque) score from a human or a model acting as judge. The most flexible option and the least interpretable, hardest to audit or scale cheaply.
Rubrics as a formal reward-design method were recently introduced in Gunjal et. al's 2025 paper, titled "Rubrics as Rewards". It proposed rubrics to address the gap between open-ended tasks and verifiable rewards. It has since become an active area with its own literature on how to construct rubrics well, and current RL training toolkits, Verifiers among them, now treat rubrics as a first-class building block alongside the environment and the training loop itself.
Writing Rubrics
For high-stakes or domain-specific work (e.g., medical or legal), the rubric is written by a subject matter expert, once, and then reused across every rollout that touches that task. For everyday scale, where you need thousands or millions of rubrics and cannot have an expert write each one, they are generated synthetically, often by another model prompted to infer good criteria from examples of strong and weak responses. A related approach elicits rubrics automatically from pairwise comparisons, having a model infer what distinguished a preferred response from a rejected one, as opposed to asking an expert to write the criteria down. A recent variation lets the rubric change during training itself, evolving alongside the model so that it keeps getting harder to satisfy as the model improves.
Gaming Rubrics
A verifiable reward is checking a fact: the game was won, the test passed. A rubric is checking a proxy for quality which can, in principle, be satisfied without the underlying thing it was meant to measure being true. Reward hacking occurs when a model exploits this gap. This has become a real problem with active research specifically on detecting and reducing reward hacking in rubric-based training, models finding a way to technically tick a criterion's box, citing a source that does not actually support the claim, for instance, without doing the harder work the criterion was meant to require.
This is exactly why verifiable rewards are preferred wherever they are available, and rubrics exist to cover the much larger space of tasks where they are not. The best practice is: use a verifiable reward if the task allows one, reach for a rubric when it does not, and treat freeform judgment as the fallback for whatever neither of the first two can capture.
Why it Scales
Once a reward is something an environment can compute on its own, the cost of running it a second time, or a millionth time, drops to whatever the compute costs, not whatever a person's time costs. That is the entire reason environments, not human graders, are what current agent training actually scales through. The exploration and the evaluation both become automated: the agent generates its own attempts, and the environment scores every one of them without anyone reviewing the individual rollout.
Reports on Qwen3-Coder's training setup describe scaling to roughly 20,000 parallel environments running at once, and MiniMax has described its Forge training infrastructure running across hundreds of thousands of environments. Numbers at that scale are only possible because nobody is sitting in the loop grading individual attempts.
Common questions
What exactly counts as a rubric?
A rubric is a named, structured list of criteria for judging a response or an action, with each criterion checkable on its own. The defining feature is not how the criteria get checked, that can be a rule, a script, or a model prompted to look for one specific thing, but that the checklist is explicit and each item can be inspected separately, which is what makes it interpretable and reusable across many rollouts.
Who writes the rubrics used in real RL training?
It depends on scale. Domain experts write them by hand for high-stakes, narrow tasks where precision is critical. For everyday large-scale training, rubrics are usually generated synthetically, often by prompting a model to infer good criteria from labeled examples, or elicited automatically from pairwise human comparisons.
Can a rubric be gamed?
Yes. This is an actively studied failure mode called reward hacking. Because each criterion is a proxy for the quality it is meant to represent, a model can sometimes find a way to satisfy the letter of a criterion without doing the substantive work behind it. It is the main reason verifiable rewards are preferred whenever a task actually allows for one, and why the newest rubric-based methods let the rubric evolve during training.
References
- Gunjal, Wang, Lau, Nath, Liu & Hendryx, “Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains” (2025): the paper that introduced rubric-based rewards as a named method.
- Rezaei, Vacareanu, Wang, Wang, Liu, He & Akyürek, “Online Rubrics Elicitation from Pairwise Comparisons” (2025): rubrics inferred automatically from comparisons rather than written by hand.
- “Reward Hacking in Rubric-Based Reinforcement Learning” and related work on reproducing and detecting the failure mode described above.
- “The Ultimate Guide to RL Environments: Building and Scaling Them in the LLM Era” (2026): the source of the Qwen3-Coder and MiniMax Forge scaling figures cited above.