How to evaluate a dataset sample.

The sample is not the dataset. It is evidence.

A supplier sends a sample. Your job is to decide whether that evidence supports moving to a contract. You do not evaluate for curiosity. You evaluate against a spec that defines acceptance criteria, rejection reasons, and the tests you will run. If the sample passes, you write an acceptance memo. If it fails, you document the rejection and move on.

This procedure assumes you already have a written spec: modality, language and dialect, rights for training versus retrieval, scale, quality targets, and refresh cadence. If you do not have a spec, stop. Write it first. Then return here.

Confirm the sample matches the spec scope.

Open the manifest. Compare every column to your spec.

• Modality: audio, text, video, or multimodal. Does the sample contain the exact codecs, sample rates, and container formats you require?
• Language and dialect: Are the locale tags present? For Hebrew and Arabic speech, do the tags distinguish Modern Standard Arabic from Levantine, Gulf, or Maghrebi? Does Hebrew include diacritics where your spec requires them?
• Rights: Does each row carry a rights tag that maps to training use, retrieval use, or both? A sample tagged only for retrieval cannot be accepted for a training license.
• Scale: Is the sample size proportional to the claimed corpus? A 1,000-utterance sample from a claimed 500,000-utterance corpus is a 0.2% draw. That may be sufficient for a label audit but thin for duplicate detection.
• Quality targets: Does the sample include the quality metrics your spec demands (WER, CER, MOS, annotation agreement scores)?
• Refresh: Does the sample include a timestamp or version field that lets you verify refresh cadence later?

If any field is missing or misaligned, log a spec deviation. Do not proceed to deeper tests until the supplier provides a corrected manifest or a written explanation you can evaluate.

Draw your own subsample. Do not trust the supplier’s selection.

The supplier chose the sample. You must choose the subsample you will audit. Use a deterministic seed and a documented method.

1. Stratify by the spec dimensions that matter most. For speech: language, dialect, channel, noise condition, speaker gender, age band. For text: domain, language, source type, document length bucket.
2. Set the subsample size per stratum. Use your error-rate threshold and desired confidence. A common starting point is 95% confidence with a 2.5-point margin on the label error rate, which takes about 1,500 items per stratum (the standard worst-case calculation). A smaller margin needs many more, so many teams audit fewer items and accept a wider margin. Increase for rare strata.
3. Generate the draw with a fixed random seed. Record the seed, the script, and the output list of IDs. Store the script in your evaluation repo.
4. Pull the full records for those IDs. Request the raw files, annotations, and metadata from the supplier. If they cannot deliver the exact records, that is a rejection reason: inability to verify.

Document the draw in your evaluation log. The log becomes part of the acceptance memo.

Run a label audit against a pass/fail threshold you define.

Define the threshold before you look at a single label. Example: “Label error rate must not exceed 2% per stratum at 95% confidence.”

• Define error types. Transcription error, segmentation error, speaker ID error, dialect mislabel, entity span error, POS tag error. Your spec should list the types and their severity weights.
• Use two independent annotators. They must be native speakers for the target dialects. For Hebrew and Arabic speech, this is non-negotiable. Calculate inter-annotator agreement (Cohen’s kappa or Krippendorff’s alpha). If agreement falls below your spec floor, pause. The audit is invalid until agreement is resolved.
• Adjudicate disagreements. A third senior annotator decides. Record every decision.
• Compute the error rate per stratum. Compare to your threshold. If any stratum fails, the sample fails. Do not average across strata.

Record the raw counts, the rates, the confidence intervals, and the adjudication log. Attach to the acceptance memo.

Check duplicates within the sample and against your existing corpus.

Duplicates inflate scale claims and bias training.

• Exact duplicates: Hash the content (audio waveform, normalized text). Flag collisions.
• Near duplicates: Use perceptual hashing for audio, MinHash or SimHash for text. Set a similarity threshold in your spec (e.g., 0.95 Jaccard for text, 0.98 perceptual hash for audio).
• Cross-corpus duplicates: Run the sample hashes against your internal corpus index. Any match is a red flag unless the rights explicitly permit overlap.

Report: duplicate cluster count, cluster sizes, and whether clusters cross strata. If duplicates exceed your spec tolerance, reject the sample. Document the rejection reason: “Duplicate rate exceeds spec threshold.”

Verify metadata completeness on every subsample record.

A record without complete metadata is unusable for training or compliance.

For each record in your subsample, confirm the presence of:

• Source identifier and collection date
• Speaker or author metadata (age, gender, region, consent status)
• Channel and environment metadata (device, codec, SNR, background noise class)
• Annotation provenance (annotator ID, tool version, guideline version, timestamp)
• Rights metadata (license type, territory, expiration, training vs retrieval flag, third-party encumbrances)
• Quality scores (WER, MOS, agreement score) if your spec requires them

Create a completeness matrix: rows = records, columns = required fields. Any blank cell is a metadata defect. Tally defects per field. If any field has a defect rate above your spec tolerance, reject. Document: “Metadata completeness for [field] below threshold.”

Cross-check rights documents against the sample.

Rights evidence must be specific to the records in the sample. A generic license PDF is not sufficient.

• Map each subsample record to a rights artifact. The artifact can be a contributor agreement, a license grant, a public domain dedication, or a data processing addendum. The artifact must name the record ID or a verifiable batch ID that includes the record.
• Verify the grant scope. Training rights must be explicit. Retrieval-only grants do not satisfy a training spec. Commercial use rights must match your deployment model.
• Check territory and duration. Your spec defines the required territory and term. The artifact must cover both.
• Look for encumbrances. Third-party content, background music, incidental capture of non-consenting speakers. Any encumbrance without a clear sub-license is a rejection reason.

If the supplier cannot produce a rights artifact for a subsample record, that record fails. If the failure rate exceeds your spec tolerance, the sample fails. Log every missing or mismatched artifact.

Red flags that trigger immediate rejection.

Do not waste time on deep audits when these appear. Document and move to the next supplier.

1. Manifest drift. The delivered sample does not match the manifest (missing files, wrong formats, swapped IDs).
2. Rights opacity. The supplier refuses to share rights artifacts before contract, or provides only a master agreement without record-level mapping.
3. Synthetic contamination. Undisclosed TTS, MT, or generative content in a human-data spec. Detect via metadata gaps, acoustic artifacts, or linguistic patterns.
4. PII leakage. Identifiable personal data in a spec that requires de-identification. Names, phone numbers, exact addresses, medical record numbers.
5. Version mismatch. The sample reflects an older annotation guideline version than the one referenced in the spec.
6. Supplier cannot reproduce the draw. You request the raw files for your subsample IDs; they deliver different files or claim the IDs are internal only.
7. Refresh ghost. The spec requires quarterly refresh; the sample contains no records newer than 18 months.

Each red flag gets a line in your rejection log with the evidence.

Write the acceptance memo or the rejection memo.

One document. Two possible outcomes.

Acceptance memo structure:

• Spec reference (version, date)
• Sample reference (supplier, manifest ID, delivery date)
• Subsample draw method (seed, script hash, stratum counts)
• Label audit results (error rates per stratum, confidence intervals, adjudication log reference)
• Duplicate analysis (within-sample, cross-corpus, threshold comparison)
• Metadata completeness matrix (defect rates per field)
• Rights cross-check (artifact mapping table, scope verification, encumbrance clearance)
• Red flag scan (none found, or list with disposition)
• Decision: ACCEPT. The sample meets all acceptance criteria. Proceed to contract amendment.
• Signatures: Procurement engineer, ML lead, Legal counsel.

Rejection memo structure:

• Same header fields.
• First failing criterion (stop on first fail, but document all checked).
• Evidence reference (log lines, artifact IDs, screenshots).
• Decision: REJECT. Sample does not meet acceptance criteria.
• Next steps: Return to supplier with specific remediation requests, or disqualify.
• Signatures: Procurement engineer, ML lead.

File the memo in your procurement record. It is the audit trail for the contract.

Checklist: Sample evaluation gate.

• Spec exists and is versioned.
• Sample manifest matches spec scope on every field.
• Subsample drawn with documented seed, stratification, and size rationale.
• Raw files for subsample IDs retrieved and verified.
• Label audit completed by two native annotators per stratum.
• Inter-annotator agreement meets spec floor.
• Error rates per stratum below defined thresholds.
• Duplicate scan (exact and near) within tolerance.
• Cross-corpus duplicate scan clean.
• Metadata completeness matrix meets per-field thresholds.
• Rights artifacts mapped to every subsample record.
• Rights scope covers training, territory, term, and commercial use.
• No undisclosed encumbrances.
• Red flag scan clear.
• Acceptance or rejection memo drafted and signed.

Next step: send a data request.

You have a procedure. You have a spec. You need a supplier who can deliver a sample that survives this gate. We find and qualify licensed training data against model requirements - modality, language and dialect, rights for training versus retrieval, scale, quality, refresh. Our strength is Hebrew and Arabic speech and text in specialist domains. We only offer data with documented rights; we are not a marketplace with a public catalog.

When you are ready to move from procedure to procurement, send us your spec. We will source suppliers, run the evaluation, and deliver the acceptance memo.

Send a data request

Where to go next.

A sample is only as useful as the spec behind it: see how to write a training data purchase brief and buying Hebrew speech data. Rights questions are covered in training rights vs retrieval rights.

Editorial notes: numerical examples are illustrative. Review our editorial and corrections policy.