Dataset Remediation — Audit, Cleaning & Repair
In addition to platform generation service and storage tools, Knowledgator provides managed agent with skills (dataset-tools and dataset-sync) for diagnosis, anomaly detection, Capped Hybrid rebalancing, and deterministic remediation.
What Problems Does Remediation Solve?
Dataset remediation automatically audits and repairs common data flaws that degrade model training:
- Mislabeled Samples: Identifies contradictory ground-truth labels using zero-shot semantic cross-checking.
- Severe Class Imbalance: Rebalances starved (under-represented) and bloated (over-represented) classes via Capped Hybrid quotas.
- Data Duplication & Near-Duplicates: Detects repeated text, redundant templates, and semantic duplicates.
- Low Linguistic Diversity & Complexity: Highlights overly simplistic phrasing or limited vocabulary spread.
- Schema & Format Inconsistencies: Enforces strict structural adherence and missing field validation.
This architecture enables agents to operate on datasets with thousands or tens of thousands of records (10k+) in zero-dirty-context mode: datasets are processed as streaming files on disk without polluting the LLM prompt context with raw data samples.
Architecture: MCP Tools vs. Managed Agent Skills
| Dimension | MCP Tools (knowledgator-mcp) | Managed Agent Skills (dataset-tools, dataset-sync) |
|---|---|---|
| Execution Environment | Remote Knowledgator API Gateway | Isolated Managed Agent Environment |
| Primary Purpose | Synthesize brand new datasets from a natural language prompt | Audit, rebalance, and repair existing datasets on disk |
| Context Load | Standard JSON prompt response | Zero Dirty Context |
| Tool Calling Method | MCP tool invocation (tools/call) | Bash CLI execution (dataset-probe, dataset-plan, etc.) |
5D Dataset Diagnostic Framework
Datasets are evaluated along five mathematical and semantic quality dimensions via dataset-probe:
1. Balance Score (S_balance in [0, 1])
Measures class uniformity across categories using normalized Shannon Evenness:
H = - sum(p_i * ln(p_i)), S_balance = H / ln(K)
- Healthy:
S_balance >= 0.85 - Warning / Deficit:
S_balance < 0.85(indicates skewed class distributions).
2. Quality Score (S_quality in [0, 1])
Verifies semantic label agreement. Zero-shot GLiClass inference evaluates text records against ground truth annotations to detect mislabels or confusing descriptions:
S_quality = N_valid / N_total
- Healthy:
S_quality >= 0.95
3. Diversity Score (S_diversity in [0, 1])
Quantifies semantic non-redundancy and vocabulary spread across samples:
S_diversity = 1.0 - MeanPairwiseCosineSimilarity(X)
- Identifies near-duplicate samples, template repetition, or narrow synthetic phrasing.
4. Complexity Score (S_complexity in [0, 1])
Measures syntactic difficulty and token richness:
S_complexity = 0.5 * (min(1.0, L_avg / 250) + TTR)
where L_avg is average word length and TTR = |Unique Tokens| / |Total Tokens|.
5. Consistency Score (S_consistency in [0, 1])
Checks schema compliance, required fields presence, and determinism in format structure.
Capped Hybrid Rebalancing Strategy
When rebalancing imbalanced datasets, simple oversampling causes severe data duplication, while simple undersampling discards valuable data. The dataset-plan tool calculates a Capped Hybrid distribution:
- Calculate mean class count:
N_avg = N_total / K - Compute dynamic floor and ceiling quotas:
Floor = max(1, floor(0.8 * N_avg))Cap = max(Floor, ceil(1.5 * N_avg))
- Class Actions:
- Starved Classes (
N_i < Floor): Flagged for delta augmentation by+(Floor - N_i). - Bloated Classes (
N_i > Cap): Flagged for trimming down toCap. - Balanced Classes (
Floor <= N_i <= Cap): Left intact.
- Starved Classes (
Tool Suite
The dataset manipulation suite follows strict separation of single responsibilities:
| Tool | Responsibility | Input/Output |
|---|---|---|
dataset-validate | Canonical schema validation for all tasks | File on disk $\to$ Exit code (0=ok, 1=invalid) |
dataset-view | Safe sample inspection without context blowup | --index, --slice, --sample $\to$ Clean Markdown |
dataset-probe | 5D mathematical and quality diagnostics | Summary card or detailed dimension metrics |
dataset-profile | Class distribution and Shannon Evenness breakdown | Headroom, deficit counts, quota analysis |
nl-grep | Semantic natural language search via GLiClass | Filter records by query or extract mislabeled rows |
dataset-plan | Generate Capped Hybrid remediation manifest | manifest.json with relabel, drop, and augment quotas |
dataset-patch | Deterministic local file transformation | Produces patched dataset without LLM hallucination |
dataset-sync | FastMCP gateway streaming upload/download | Validates schema pre-flight and manages revisions |
6-Step Remediation Pipeline
flowchart TD
A["1. dataset-probe / dataset-profile"] --> B["2. nl-grep --mismatches (GLiClass)"]
B --> C["3. dataset-plan (Capped Hybrid)"]
C --> D["4. Agent Synthesizes Missing Delta Only"]
D --> E["5. dataset-patch (Deterministic Apply)"]
E --> F["6. dataset-validate & dataset-sync Upload"]
CLI Command Reference
# 0. Validate schema and inspect sample
dataset-validate --file data/dataset.json
dataset-view --file data/dataset.json --index 0 --format md
# 1. Diagnose initial deficits
dataset-probe --file data/dataset.json --mode full --format md
dataset-profile --file data/dataset.json --format md
# 2. Extract semantic anomalies
nl-grep --file data/dataset.json --mismatches --format json > data/mismatches.json
# 3. Formulate remediation manifest
dataset-plan --file data/dataset.json --mismatches-file data/mismatches.json --out data/manifest.json --format md
# 4. Agent synthesizes missing delta records into data/synthetic.json
# 5. Apply patch
dataset-patch --file data/dataset.json --manifest data/manifest.json --augment-file data/synthetic.json --out data/remediated.json
# 6. Verify score recovery and publish
dataset-validate --file data/remediated.json
dataset-probe --file data/remediated.json --mode full --format md
dataset-sync upload --file data/remediated.json --name "my-remediated-dataset"
Remediation Manifest Schema (manifest.json)
{
"strategy": "capped_hybrid",
"floor_quota": 4,
"cap_quota": 7,
"n_avg": 4.67,
"relabel_rules": [
{
"index": 11,
"original_label": "home_goods",
"new_label": "cardiology",
"reason": "Mismatched prediction (confidence: 0.9961)"
}
],
"drop_indices": [],
"augment_quotas": {
"cardiology": 2
},
"downsample_targets": {
"home_goods": 7
},
"metadata": {
"source_name": "ecommerce_dataset",
"total_samples": 14
}
}