Data vault
Outbound data egress: export your Pact tenant to your own S3, GCS, Azure Blob, warehouse, or stream — with export runs, signed manifests, CDC subscriptions, and reverse-ETL.
Data vault
The Data Vault is Pact's outbound data-egress engine: it pushes your
tenant's records to destinations you own — object stores, cloud
warehouses, and streaming targets — so your warehouse team works against a
copy of Pact data on their own infrastructure. Unlike the inbound
sources and pipelines surfaces,
the vault's exporters are real: run_export in core/data_vault/service.py
renders records with the JsonlExporter and uploads them through
destination-specific adapters.
This is a live, enforced surface
Export runs actually write data. core/data_vault/exporters and
core/data_vault/destinations contain working adapters that render gzipped
JSONL chunks and put_object them to the target. Runs are recorded in
vault_export_runs, every action is written to an access log, and a signed
manifest is produced per run. Backend from alembic 0067_data_vault; REST at
/v1/vault/*.
Destination kinds
Destinations are grouped by capability class in
core/data_vault/destinations:
| Class | Kinds |
|---|---|
| Object store | s3, gcs, azure_blob, local, memory |
| Warehouse | postgres, redshift, snowflake, bigquery, databricks |
| Stream | kafka, kinesis |
Full export runs are object-store only
run_export requires an object-store destination (OBJECT_STORE_KINDS);
it raises if you point a full export at a warehouse or stream. Warehouse and
stream targets are driven through CDC subscriptions (below) and
reverse-ETL rather than full-table export runs. Adapter maturity varies by
kind — verify a destination with the test endpoint before relying on it in
production.
Managing destinations
GET /v1/vault/destinations list
POST /v1/vault/destinations create
GET /v1/vault/destinations/{id} fetch one
PATCH /v1/vault/destinations/{id} edit
DELETE /v1/vault/destinations/{id} remove
POST /v1/vault/destinations/{id}/test verify credentials
POST /v1/vault/destinations/{id}/run trigger an export run
test initializes the adapter and calls test_connection(), flipping the row
to verified or error with a status message. run kicks off a full export.
Export runs and manifests
GET /v1/vault/export-runs recent runs
POST /v1/vault/export-runs create a run
GET /v1/vault/export-runs/{id} run detail
GET /v1/vault/export-runs/{id}/manifest manifest JSON
A run streams each exportable entity in chunk_size-row pages (default 10,000),
gzips them as JSONL, and uploads each chunk under a run-{id}/ prefix. Failures
are caught and recorded as status='failed' with an error message rather than
crashing the worker tick. The manifest (manifest_version: 1) enumerates the
files, row counts, and schema versions for the run.
Change Data Capture (CDC)
For incremental, near-real-time egress, subscribe to changes instead of running full exports:
GET /v1/vault/cdc-subscriptions list
POST /v1/vault/cdc-subscriptions create
PATCH /v1/vault/cdc-subscriptions/{id} pause / resume
DELETE /v1/vault/cdc-subscriptions/{id} remove
GET /v1/vault/cdc-subscriptions/{id}/lag current lag
POST /v1/vault/cdc-subscriptions/{id}/drain flush buffer to destination
The CDC streamer (core/data_vault/cdc_streamer.py) installs SQLAlchemy event
listeners that buffer row-level changes; drain flushes the buffer to the
subscription's destination.
CDC requires ORM writes
Change capture hangs off SQLAlchemy ORM events. Bulk writes that bypass the ORM
(raw INSERT/UPDATE via sa.text) are not captured. Producers whose
changes must reach the vault should write through the ORM.
Reverse-ETL
The vault also runs reverse-ETL: a customer-authored SQL query against your
own warehouse whose result columns map onto Pact objects — enriching contacts
and companies with warehouse-computed fields, or driving segment membership.
Definitions live in reverse_etl_syncs with append-only reverse_etl_runs
history (alembic 0271_reverse_etl), exposed at /v1/warehouse/reverse-etl/*.
Access control
Reads require Permission.TENANT_READ; every write — create, patch, delete,
test, run, and drain — requires Permission.ADMIN_FLAGS, the same
tenant-administrator gate the team UI uses, since data egress is an
administrative concern. Cross-tenant ids return 404, never 403.