Dataset quality
Make the source and the checks clear.
A useful dataset needs more than a large record count. Define how records were sourced, which tasks they show, and how suitability will be reviewed.
Source history
Record where the material came from and how it was prepared. This source history is sometimes called provenance.
- Source system, owner, date range, and extraction method
- Permission basis, restrictions, and approved scope
- Version, transformations, exclusions, and links to original tasks
Request, work, and result
Keep the parts of a task connected. State what each field means and distinguish an observed outcome from a proposed answer.
- The input request and relevant context
- Actions, decisions, handoffs, and applicable rules
- The outcome, status, and evidence of completion
- Role labels and sequence information where permitted
Duplicates and missing outcomes
Count usable tasks separately from messages or rows. A copied case should not silently become another independent example.
- Exact and near-duplicate records, including linked versions
- Incomplete tasks and missing final decisions
- Automatic replies and repeated templates
- Sampling choices, unusual cases, and known gaps
Documented review rules
Write down the checks a reviewer applies. A label should mean the same thing across records in the agreed scope.
- Field completeness, consistency, and content checks
- The criteria for accepted, rejected, or uncertain records
- How disputed labels and acceptable alternatives are handled
- Who reviewed which version and what changed
Separate learning from checking
If records support both training and evaluation, define separate groups. Keep a task and its near-duplicate versions together so they do not cross the boundary.
- The purpose and selection rule for each group
- Checks for overlap between training and evaluation cases
- Evaluation inputs separated from expected answers
- A versioned list of records included in each group
Agree the delivery format
Choose a structure that supports the buyer’s task. Formats and field definitions are agreement details, not proof of quality by themselves.
- JSONL: one structured JSON record per line, useful for nested task context
- CSV: a table format for flatter fields, with linked files defined separately
- A field dictionary, version notes, and handling instructions
- An agreed transfer method and acceptance process
A sample should answer
a defined question.
Agree a small review scope before preparing a sample. Define who can access it, what it may be used for, and when it must be returned or removed.
The sample review should compare the source against the stated task and acceptance rules. It does not authorize unrestricted training or later use.
Read the evaluation guideSample acceptance questions
- Does the record show the task and the required context?
- Are fields understandable and the outcome supported?
- Were exclusions and handling requirements applied?
- Are missing information and quality limits documented?
- Who accepts the sample, and what happens if it needs revision?
See a record structure.
This small pack contains three invented examples: a support case, an operational workflow, and a recorded AI correction. Synthetic means made up; no real customer data is included.
JSONL records, JSON Schema, and a README. Illustrative structure only; no available dataset or performance result is implied.
{
"synthetic": true,
"record_type": "operational-workflows",
"request": "Reschedule a shipment after a route closes.",
"work": [
"Check approved alternatives.",
"Record the required approval."
],
"result": {
"status": "completed-for-example",
"outcome": "A feasible route is approved."
},
"evaluation": {
"criteria": [
"Uses a permitted route.",
"Records the approval."
],
"acceptable_alternatives": [
"A plan requiring approval before release."
]
}
}Describe the source you hold.
Start with an inventory. We can discuss the task, rights, and quality questions before any records move.