Sayed Wayzi, CEO of Korra Data · · 6 min read
Why AI practice worlds are graded in code, and which business records fit
AI labs train agents in RL environments, which are copies of business tools filled with past work. Records with a clear result, such as a closed ticket or a won deal, can be checked by code and fit best.
Zapier's public test for AI agents uses only code to grade the work. No AI model is the judge.
Many RL environments now try to grade work this way. An RL environment is a practice world. In it, an AI agent learns to do office work. The method of grading tells you which business records fit these practice worlds.
For an owner, the short answer is this. Records that end in a clear result are easy to change into practice tasks. Records with no clear result are harder.
What a practice world for an AI agent is
An AI agent is a program that does actions in software. For example, it can find a customer in a CRM. It can reply to an email. It can update a ticket. To improve, the agent needs a safe place to practise.
That place is the practice world. It is a copy of the tools a company uses. Examples are an inbox, a CRM, a help desk and a ledger. The copy is full of work history. The agent gets a task and does its actions. Then a check finds out if the work is correct.
Scale AI wrote that nearly half of its new data training projects now use these environments. Scale said they are simulated worlds. It said they are populated with data mimicking real business workflows, and domain experts curate them. Scale said it builds in the parts of real work that cause problems for agents. These include incomplete data, records that do not agree, and tasks that use more than one tool.
How RL environments are built from business records
These are the basic steps. Each builder does the details differently.
- Copy the tools. The environment has a working copy of an inbox, a CRM, a calendar or a ticket queue.
- Put in the history. Add contacts, email threads, deals and tickets. Keep the messy parts.
- Change a request into a task. For example, a customer email that asks for a refund becomes the first instruction to the agent.
- Keep the real result as the answer key. Maybe the company gave the refund and closed the ticket. That final state is the correct answer. The agent's work is compared with it.
- Let the agent work. Then check the result. Reset the environment. You can then do the task again.
Scale said each task starts from a known setup. It said the check looks at real changes in the system. Scale said clearly that it wants more than text that looks correct.
- Why was this data created?
- What does one record hold?
- How was it collected, and over which dates?
- What was removed or changed?
- What is missing?
- Which uses does the seller allow?
- How was the sample chosen from the full set?
Why the grading is moving to code
The check at the end is called a verifier. The verifier decides if the agent passed.
One method uses a second AI model. That model reads the work and grades it against a list of criteria. Builders have reasons to be careful with this method. Wing Venture Capital wrote that it becomes harder to check the work as models improve. Wing gave a list of problems. Tests stop showing differences between models. Graders change over time. Reward hacking also occurs. Reward hacking is when the agent learns to get a good score but does not do the job.
The other method is a check in code. The code asks simple questions. Did the deal move to Won? Did the ticket close with the correct fix? Does the ledger balance? The code gives the same answer each time.
Zapier used this method for its AutomationBench test. Zapier said the scoring is deterministic. That means the same work always gets the same score. The test compares the final state of the tools with a set of success criteria. Zapier said no LLM is the judge. The test has six business areas: sales, marketing, operations, support, finance and HR. Zapier said it selected these areas from patterns across the 3.7M companies and 2B monthly tasks it sees.
The tasks are difficult. When the paper on arXiv about AutomationBench came out in April 2026, even the best frontier models scored below 10%. The paper also says agents must work around records that are not relevant. Some records give wrong directions. Real company history is like this.
Wing also said these environments grew large first in two areas: coding and computer use. Wing gave this reason. Code has tests. Tools such as CRMs and spreadsheets record each change. You can replay these changes and grade them.
Which business records fit
A code grader must have a result that it can read. So the best records have a field that shows the result. These records are the easiest to change into tasks with an answer key:
- CRM deals with their stages, which end in Won or Lost
- Support tickets with the fix and the closing status
- Invoices matched to payments
- Ledgers that reconcile
- Code changes with the tests that ran on them
Records with no clear result are harder to grade. A strategy memo has no final state. The tone of an email is an opinion. These records can still have value. But code cannot easily mark them correct or incorrect.
The most useful history follows one job across tools. A request comes in by email. Someone opens a ticket. The team works on it in chat. At the end, the customer pays an invoice. Zapier's paper describes single tasks that use a CRM, an inbox, a calendar and a messaging tool. So this type of linked history matches how the tests are built. For more, read what makes your records useful for AI.
Why a bigger company often has more to offer
This section is our own reasoning. No buyer has published it as a rule.
More staff usually means more tasks on record. It also means more handoffs. For example, sales gives a deal to operations, and operations gives it to finance. Larger companies usually fill in the fields in their tools. Teams must do this to work together. More people also means more variety and more mistakes. Agents learn from these.
Size is not the only factor. A 30-person company can have clean records that link from tool to tool. That company can be more useful than a 400-person company where half the fields are empty.
Before all of this, do one more check. Make sure the records are yours to license. Client files and records you keep for another person are a different question. Read do you own the records you want to license. At Korra, client records stay out unless the owner has written permission.
What you can check in your own records today
You do not need to export anything. Open your main systems and look.
Find the systems that have a result field. Examples are a deal stage, a ticket status or a paid date. Count how many years of history you still have. Find out if a plan limit or a change of tool deleted any history. Then select one job. Try to follow it from the first email to the final invoice.
Kept unless someone deletes it
- Slack on a paid plan
- Google Workspace mail and files
- Microsoft 365 under a retain rule or a hold
Gone on a clock
- Slack on the free plan
- Anything under a delete rule
- A leaver’s mailbox with no hold
Write down your answers. This is the start of a source inventory. This guide shows how to make a list of your data.
For an approximate value of records like yours, use the value calculator. It gives a starting range in a few minutes.
Questions
- Do labs need my live systems to build a practice world?
- No. Scale AI described its environments as simulated, controlled copies that can be reset between runs, so the agent never touches production software. With Korra, buyers never get live access to a business's systems.
- Will removing names make my records anonymous?
- No. Taking out names, companies and emails reduces risk, but it is not the same as anonymous. Details left in the text can still point to a person or a company.
- Why does a test hold some tasks back?
- Zapier said AutomationBench uses a public and private split. That way model makers can get checked results without training on the full set of tasks.
- Are support tickets better than email?
- Tickets usually carry a status and a fix, so code can check them. Email threads can still help when they link to a result in another tool, such as a paid invoice or a closed deal.
Sources
- AutomationBench — zapier.com
- AutomationBench — arXiv.org
- Insights & Research | Wing Venture Capital — wing.vc
- The Next Frontier of Data Training: RL Environments — Scale AI
Figures are as reported by the publishers above. Discussion threads show what people said; they are not facts or Korra results.
What could your records be worth?
Three questions give a rough range. No files, no logins.
Try the calculator
