Architecture

Ingest-Time Contract Gateway (DataContractGate) vs. Post-Load Warehouse Testing (Great Expectations, Soda, dbt tests)

Warehouse tests tell you a table is wrong after the rows have landed. DataContractGate checks each payload as it arrives, keeps the failing ones in quarantine, and forwards a payload whose declared PII is already masked, hashed, or dropped.

Two places a contract can run

Producer

API, SDK, Zapier, file drop

payload

→

DataContractGate
engine p99 < 50µs
pass or quarantine

→

Warehouse table

only accepted rows

stored

→

GE · Soda · dbt

scans and model tests

after load

What changes when the check moves in front of storage

	DataContractGate	Great Expectations	Soda	dbt tests
When it runs	On the payload, before storage	After a batch or table exists	Scheduled checks against the warehouse	During a dbt run, on a built model
Unit of work	One event, or a batch of up to 1,000	A table, query, or batch expectation suite	A dataset check	A SQL assertion on a model
Latency	Validation engine p99 under 50µs per event	A warehouse or Spark scan	A warehouse scan on a schedule	The model's query time
A bad payload	Quarantined with violations. Downstream sees the passing payload.	Already written. The suite reports the failure.	Already written. The check opens an incident.	Already in the model inputs. The test fails the run.
PII	mask, hash, or drop on the contract, applied before storage and again on egress	The raw value is already in the table the suite reads	The raw value is already in the table the check reads	The raw value is already in the model inputs
Fixing a reject	One-click replay, or `POST /quarantine/replay`	Re-run the suite after a cleanup job	Re-run the checks after a cleanup job	Fix the model or source and run dbt again
Strongest fit	Types, required fields, enums, ranges, regex, freshness, PII policy on the event	Distribution checks and custom expectations on landed data	Freshness, volume, and metric monitors	Relationships, uniqueness, and accepted values across models

Stop the payload, then keep the table tests

A negative amount, an unknown currency, or a free-text email in an id column is a property of the payload. DataContractGate can decide that in the validation engine, at a p99 under 50µs, and hold the event in quarantine with the field, the kind of violation, and a suggestion. The warehouse load that follows receives the transformed event. Declared string fields with transform.kind of mask, hash, or drop are rewritten before anything is stored. The same transform runs on POST /egress, which is the path AISchemaGate uses when an LLM payload leaves the system.

Great Expectations, Soda, and dbt tests answer a different question: given what did land, do the tables still agree? Row counts across a join, a freshness delay on a partition, a distribution shift, a relationships test between models. Those checks need the warehouse. They also need the bad row to be absent, or they spend the run explaining a producer bug. Running DataContractGate at ingest makes the warehouse suite quieter. Teams that already trust their dbt tests can keep them and add the gate in front. Teams whose incidents are "the producer shipped the wrong shape" can let the gate carry that contract and leave the warehouse tests for relationships and aggregates.

Contract on the event

Author DataContractGate YAML, or infer a draft in the playground. Promote a stable version. The contract is the allow-list for what may be stored.

Gate, then load

Producers call POST /v1/ingest/{contract_id}. Passing events are safe to load. Failing events return a quarantine id and stay out of the table.

Replay, then scan

Fix the producer or the contract and replay with one action in the dashboard or POST /quarantine/replay. dbt, Soda, and Great Expectations keep watching the tables that remain.

A reject never needs a cleanup job

POST /v1/ingest/{contract_id}
X-Api-Key: $CONTRACTGATE_API_KEY

# pass  → transformed_event is what you load
# fail  → quarantine_id, violations[].field, violations[].kind

POST /quarantine/replay
{ "ids": ["<quarantine_id>"] }

Replay re-runs the stored payload against the live contract. A pass is forwarded. A second failure stays in quarantine with a new violation report. There is no "delete the bad partition and backfill" step for a payload the gate already held.

The same quarantine list is what the Zapier New Quarantined Record trigger polls, so a Slack or Jira Zap can open when either an ingest record or an AISchemaGate egress check fails.