Kavach /deck

Presentation mode

Back to war room

Your data platform breaks at 2am.

Kavach is the war room that detects, investigates, fixes, and learns — before the dashboard turns red.

Incidents are slow because context is scattered.

Lineage, ML deployments, ownership, and postmortems live in different tabs. Agents need one surface.

The self-healing loop

Inject chaos → Sentinel detects → Investigator roots cause → Fixer ships a PR → Scribe writes back → MTTR drops.

Chaos
Sentinel
Investigator
Fixer
Scribe
Flywheel

How Kavach works

Data sources first, then MCP connections, then every upstream surface we consume from datahub-project/datahub.

Thesis: Kavach is armor for the data platform — AI agents on DataHub’s context graph that detect, diagnose, fix, and learn from incidents. Health for systems, health for humans: the same agents heal a retail warehouse and a healthcare domain. Loop: chaos breaks it → agents detect → root-cause → gate the model → open a real PR → write knowledge back.

Two use cases

  • systems

    Health for systems

    Retail / ops / ML reliability

    Schema drift, null spikes, value corruption, freshness lag on DuckDB + dbt. Blast radius hits marts, dashboards, and the demand-forecast deployment. Live data probes use NYC TLC Yellow Taxi (public).

    NYC TLC Yellow Taxi (Open Data)

  • humans

    Health for humans

    Healthcare governance / clinical quality

    PHI exposure and medication null spikes. Blast radius is who can read the mart and who drops out of a cohort — not a revenue dashboard. Probes use Synthea-style synthetic patients (no real PHI).

    Synthea — synthetic patients (no real PHI)

Stack

DataHub

What it is
Open-source metadata platform — lineage, incidents, ML entities, glossary, Context Documents.
What we use it for
The context graph agents read and write: lineage for RCA, blast radius, incidents, assertions, postmortems, glossary tags.
Where in repo
backend/app/datahub/
How Kavach uses DataHub
CapabilityAccessPath
MCP Server (JSON-RPC / Streamable HTTP)READ|WRITEbackend/app/datahub/mcp.py
Agent Context Kit (datahub-agent-context)READ|WRITEbackend/app/datahub/context_kit.py
search / get_entitiesREADbackend/app/datahub/client.py
get_lineage (table + column)READbackend/app/datahub/service.py
list_schema_fieldsREADbackend/app/datahub/service.py
get_dataset_queries / find_sql_contextREADbackend/app/agents/nodes/investigator.py
draft_sql_for_tablesREADbackend/app/agents/nodes/fixer.py
ML entities (feature → model → deployment)READ|WRITEml/lineage.py
Incidents (create / resolve)READ|WRITEbackend/app/agents/nodes/sentinel.py
AssertionsWRITEexamples/assertions/
Context Documents (search / grep / save)READ|WRITEbackend/app/flywheel/
Tags / glossary terms (add_tags, add_terms)WRITEbackend/app/agents/nodes/scribe.py
Domains / ownership (set_domains, add_owners)WRITEbackend/app/agents/nodes/scribe.py
Descriptions (update_description)WRITEbackend/app/agents/nodes/scribe.py
Cursor MCP integrationREAD|WRITE.cursor/mcp.json
Skills (datahub-incident-response)READskills/datahub-incident-response/
Analytics Agent / Ask DataHubREADbackend/app/analytics/ (Cloud-only — labeled honestly)

Data sources

  • fiction-retail

    DataHub datapack

    Primary retail catalog + lineage backdrop for chaos scenarios.

    Official DataHub sample retail metadata — safe to ingest and cite in an Apache-2.0 repo.

    Apache-2.0 (hackathon resources)

  • healthcare

    DataHub datapack

    Second domain: health for humans (PII exposure / clinical null spikes).

    Synthetic patient metadata (~55k records) with planted DQ issues — no real PHI.

    Apache-2.0 (hackathon resources)

  • nyc-taxi

    DataHub datapack

    Taxi-domain freshness / SLA catalog context alongside live TLC probes.

    Official DataHub taxi metadata pack — catalog only, no proprietary rows.

    Apache-2.0 (hackathon resources)

  • NYC TLC Yellow Taxi

    Public trip data

    Systems domain live probe (negative fares, zero passengers, bad timestamps).

    City-published trip records; we cite anomalies that already exist in the feed.

    NYC Open Data / TLC terms (public)

    Open source

  • Synthea synthetic patients

    Synthetic clinical data generator

    Humans domain probe link + PHI / medication-null scenarios.

    Synthetic patients only — zero real PHI. War-room Humans probes are patterned after Synthea, not live PHI.

    Apache-2.0 (Synthea project)

    Open source

Connections

  • MCP endpoint

    http://34.60.67.85:8080/mcp (self-hosted GMS)

    Live tools target {DATAHUB_GMS_URL}/mcp when set; otherwise fixtures power offline demos.

    backend/app/datahub/mcp.py

  • Self-hosted vs cloud

    Self-hosted OSS (GCP VM) · Cloud trial optional

    Default demo uses OSS self-hosted DataHub on GCP. DataHub Cloud (Ask DataHub) is optional / trial-only.

    deploy/README.md

  • Cursor mcp.json

    .cursor/mcp.json (env-driven)

    Repo ships .cursor/mcp.json wired to env vars (DATAHUB_GMS_URL, DATAHUB_TOKEN) — never hardcoded secrets.

    .cursor/mcp.json

  • Mutations

    TOOLS_IS_MUTATION_ENABLED / live GMS token

    Write path enabled when live: incidents, tags, glossary, Context Documents, assertions. Fixture mode appends to writeback.jsonl.

    backend/app/datahub/service.py

From datahub-project/datahub

Surfaces we pull from the upstream Context Platform repo and docs — not a fork, a real consumer of their OSS stack.

  • mcp-server-datahub

    acryldata/mcp-server-datahub · docs/features/feature-guides/mcp.md

    Sidecar + JSON-RPC client: initialize, tools/list, tools/call with mutations enabled.

    deploy/docker-compose.yml · backend/app/datahub/mcp.py

  • Agent Context Kit

    pip: datahub-agent-context · docs/dev-guides/agent-context/

    build_langchain_tools(include_mutations=True) inside LangGraph — not a hand-rolled fake kit.

    backend/app/datahub/context_kit.py · backend/pyproject.toml

  • acryl-datahub Python SDK

    metadata-ingestion / Python SDK in datahub-project/datahub

    Ingestion recipes, ML lineage emit, datapack load on the OSS quickstart VM.

    data/ingestion/ · ml/lineage.py

  • Metadata model entities

    entity docs: Dataset, MLModel, Incident, Assertion, Domain, GlossaryTerm, Document

    Agents read/write the graph entities judges care about — not just search demos.

    backend/app/datahub/models.py · examples/

  • Official Cursor guide

    docs/dev-guides/agent-context/cursor

    Ship .cursor/mcp.json so judges cloning Kavach get DataHub MCP in the editor.

    .cursor/mcp.json

  • datahub-skills contribution

    datahub-project/datahub-skills (companion to core)

    Opened PR #61: datahub-incident-response skill distilled from our agent loop.

    skills/datahub-incident-response/

  • Docker quickstart

    docs/quickstart · ~/.datahub/quickstart compose

    Self-hosted GMS for build/demo; Cloud trial reserved for judging Ask DataHub.

    docs/handoffs/H20-vm-datapacks/RUN.md

  • Sample datapacks

    hackathon Resources + static-assets datapacks

    showcase-ecommerce + bootstrap loaded on VM; retail/healthcare/nyc framing in Atlas.

    frontend/lib/site-content.ts (ATLAS_DATA_SOURCES)

Real vs simulated

  • real

    Schema drift (supplier qty rename)

    Deterministic chaos on DuckDB/dbt retail warehouse; Fixer PR artifacts in examples/prs/.

  • real

    Null spike on orders.customer_id

    Injected nulls in the retail pipeline; agents remediate with tests + PR.

  • real

    Value corruption in order_items.unit_price

    War-room probe also shows genuine NYC TLC negative fares; Fixer PR merged in demo-pipeline.

  • real

    Freshness lag on upstream orders feed

    Stalled partition + freshness SLA on the retail feed.

  • simulated

    Healthcare PII exposure

    Declared simulation — synthetic patients / no real PHI; labeled in UI and fixtures.

  • simulated

    NYC taxi freshness SLA breach

    Declared simulation for the scenario path; TLC probe rows themselves are real public data.

DataHub is the memory

Lineage traversal, incidents, assertions, ML entities, and Context Documents power every agent decision.

OSS contribution

We contribute a datahub-incident-response skill so every team can adopt the same playbook.

Results: measurable MTTR flywheel

Repeated schema_drift incidents resolve faster after postmortem writebacks — reproducible in replay mode.