Skip to main content

Dataset Remediation — Audit, Cleaning & Repair

In addition to platform generation service and storage tools, Knowledgator provides managed agent with skills (dataset-tools and dataset-sync) for diagnosis, anomaly detection, Capped Hybrid rebalancing, and deterministic remediation.

What Problems Does Remediation Solve?​

Dataset remediation automatically audits and repairs common data flaws that degrade model training:

  • Mislabeled Samples: Identifies contradictory ground-truth labels using zero-shot semantic cross-checking.
  • Severe Class Imbalance: Rebalances starved (under-represented) and bloated (over-represented) classes via Capped Hybrid quotas.
  • Data Duplication & Near-Duplicates: Detects repeated text, redundant templates, and semantic duplicates.
  • Low Linguistic Diversity & Complexity: Highlights overly simplistic phrasing or limited vocabulary spread.
  • Schema & Format Inconsistencies: Enforces strict structural adherence and missing field validation.

This architecture enables agents to operate on datasets with thousands or tens of thousands of records (10k+) in zero-dirty-context mode: datasets are processed as streaming files on disk without polluting the LLM prompt context with raw data samples.


Architecture: MCP Tools vs. Managed Agent Skills​

DimensionMCP Tools (knowledgator-mcp)Managed Agent Skills (dataset-tools, dataset-sync)
Execution EnvironmentRemote Knowledgator API GatewayIsolated Managed Agent Environment
Primary PurposeSynthesize brand new datasets from a natural language promptAudit, rebalance, and repair existing datasets on disk
Context LoadStandard JSON prompt responseZero Dirty Context
Tool Calling MethodMCP tool invocation (tools/call)Bash CLI execution (dataset-probe, dataset-plan, etc.)

5D Dataset Diagnostic Framework​

Datasets are evaluated along five mathematical and semantic quality dimensions via dataset-probe:

1. Balance Score (S_balance in [0, 1])​

Measures class uniformity across categories using normalized Shannon Evenness:

H = - sum(p_i * ln(p_i)),   S_balance = H / ln(K)
  • Healthy: S_balance >= 0.85
  • Warning / Deficit: S_balance < 0.85 (indicates skewed class distributions).

2. Quality Score (S_quality in [0, 1])​

Verifies semantic label agreement. Zero-shot GLiClass inference evaluates text records against ground truth annotations to detect mislabels or confusing descriptions:

S_quality = N_valid / N_total
  • Healthy: S_quality >= 0.95

3. Diversity Score (S_diversity in [0, 1])​

Quantifies semantic non-redundancy and vocabulary spread across samples:

S_diversity = 1.0 - MeanPairwiseCosineSimilarity(X)
  • Identifies near-duplicate samples, template repetition, or narrow synthetic phrasing.

4. Complexity Score (S_complexity in [0, 1])​

Measures syntactic difficulty and token richness:

S_complexity = 0.5 * (min(1.0, L_avg / 250) + TTR)

where L_avg is average word length and TTR = |Unique Tokens| / |Total Tokens|.

5. Consistency Score (S_consistency in [0, 1])​

Checks schema compliance, required fields presence, and determinism in format structure.


Capped Hybrid Rebalancing Strategy​

When rebalancing imbalanced datasets, simple oversampling causes severe data duplication, while simple undersampling discards valuable data. The dataset-plan tool calculates a Capped Hybrid distribution:

  1. Calculate mean class count: N_avg = N_total / K
  2. Compute dynamic floor and ceiling quotas:
    • Floor = max(1, floor(0.8 * N_avg))
    • Cap = max(Floor, ceil(1.5 * N_avg))
  3. Class Actions:
    • Starved Classes (N_i < Floor): Flagged for delta augmentation by +(Floor - N_i).
    • Bloated Classes (N_i > Cap): Flagged for trimming down to Cap.
    • Balanced Classes (Floor <= N_i <= Cap): Left intact.

Tool Suite​

The dataset manipulation suite follows strict separation of single responsibilities:

ToolResponsibilityInput/Output
dataset-validateCanonical schema validation for all tasksFile on disk $\to$ Exit code (0=ok, 1=invalid)
dataset-viewSafe sample inspection without context blowup--index, --slice, --sample $\to$ Clean Markdown
dataset-probe5D mathematical and quality diagnosticsSummary card or detailed dimension metrics
dataset-profileClass distribution and Shannon Evenness breakdownHeadroom, deficit counts, quota analysis
nl-grepSemantic natural language search via GLiClassFilter records by query or extract mislabeled rows
dataset-planGenerate Capped Hybrid remediation manifestmanifest.json with relabel, drop, and augment quotas
dataset-patchDeterministic local file transformationProduces patched dataset without LLM hallucination
dataset-syncFastMCP gateway streaming upload/downloadValidates schema pre-flight and manages revisions

6-Step Remediation Pipeline​

flowchart TD
A["1. dataset-probe / dataset-profile"] --> B["2. nl-grep --mismatches (GLiClass)"]
B --> C["3. dataset-plan (Capped Hybrid)"]
C --> D["4. Agent Synthesizes Missing Delta Only"]
D --> E["5. dataset-patch (Deterministic Apply)"]
E --> F["6. dataset-validate & dataset-sync Upload"]

CLI Command Reference​

# 0. Validate schema and inspect sample
dataset-validate --file data/dataset.json
dataset-view --file data/dataset.json --index 0 --format md

# 1. Diagnose initial deficits
dataset-probe --file data/dataset.json --mode full --format md
dataset-profile --file data/dataset.json --format md

# 2. Extract semantic anomalies
nl-grep --file data/dataset.json --mismatches --format json > data/mismatches.json

# 3. Formulate remediation manifest
dataset-plan --file data/dataset.json --mismatches-file data/mismatches.json --out data/manifest.json --format md

# 4. Agent synthesizes missing delta records into data/synthetic.json

# 5. Apply patch
dataset-patch --file data/dataset.json --manifest data/manifest.json --augment-file data/synthetic.json --out data/remediated.json

# 6. Verify score recovery and publish
dataset-validate --file data/remediated.json
dataset-probe --file data/remediated.json --mode full --format md
dataset-sync upload --file data/remediated.json --name "my-remediated-dataset"

Remediation Manifest Schema (manifest.json)​

{
"strategy": "capped_hybrid",
"floor_quota": 4,
"cap_quota": 7,
"n_avg": 4.67,
"relabel_rules": [
{
"index": 11,
"original_label": "home_goods",
"new_label": "cardiology",
"reason": "Mismatched prediction (confidence: 0.9961)"
}
],
"drop_indices": [],
"augment_quotas": {
"cardiology": 2
},
"downsample_targets": {
"home_goods": 7
},
"metadata": {
"source_name": "ecommerce_dataset",
"total_samples": 14
}
}