How to judge a business data sample before you make an offer

What should come with a sample?

A datasheet. The idea comes from a research paper, Datasheets for Datasets. Its authors noted that in the electronics industry every component comes with a datasheet that describes its operating characteristics and recommended uses. They proposed that every dataset be accompanied by a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.

Google’s researchers built a similar tool, the Data Cards Playbook. Its template starts with a short summary of the dataset: the content and topic of the data, its sources, and the problems or use cases it is suitable for. It then asks who owns the dataset and who to contact.

A seller of business records does not need a research-grade document. One page that answers the questions below is enough.

What does one record hold?

This is the first thing to look at in a sample, because the unit of value in business data is a complete piece of work.

Scale AI’s page for data partners shows what one large buyer looks for. For chat tools it asks for full conversation histories, with threads, participants and reply structure preserved intact. For email it asks for threads with senders, recipients and the responses that connect them. For tickets and project records it asks for their complete history and metadata.

So check each sample record for three parts:

• The request. What was asked, and by what kind of person.
• The work. The steps, the back and forth, the decisions.
• The result. What happened, and whether it was confirmed.

A thousand messages with no thread structure are worth less to you than a hundred complete cases.

How was the sample chosen?

Ask this directly. A sample picked by hand shows the best of the data. A sample drawn at random shows the data.

Ask for the date range of the sample and of the full set. The Data & Trusted AI Alliance lists the range of dates for data generation and the method among the fields in its data provenance standards, with the data format and the intended data use. Those four answers alone tell you whether the sample speaks for the whole.

Then ask what was removed or changed before you received it. De-identification changes data. Names become stand-ins, some fields go blank, some records drop out. You need to know which, so you can tell a gap in the data from a gap made by the cleaning.

How do you test a sample against your task?

Decide the test before you open the file. The NIST AI Risk Management Framework is a voluntary framework meant to improve the ability to incorporate trustworthiness considerations into the design, development, use and evaluation of AI systems. In plain terms: know what you are measuring, and write it down.

A workable test has four parts:

1. Fit. Count how many sample records are a complete example of your task.
2. Noise. Count duplicates, automatic replies, empty threads and signatures.
3. Spread. Check whether the sample covers many cases or the same case many times.
4. Limits. List what the seller’s allowed uses rule out.

Write the result as a sentence you could show a colleague: “Of fifty records, thirty were complete cases of the task we train for.” That sentence is your basis for a price. Invent your own numbers; the point is the method.

How does the sample step work on a marketplace?

On Korra you read a de-identified card first, then the full listing once your account is approved. You ask the seller questions through us, and we read each answer before it is released to you. You can ask for a sample the same way. The seller de-identifies it and sends it to you directly. We mark that it was sent, and it does not pass through us.

We do not open the data, so the judgement of the sample is yours. Make your offer on what you tested. How buying works shows the full path, and what makes records useful explains the seller’s side of the same questions.