Databricks delivery

Runway's job at the workspace edge is to make what you gated exactly what you deploy, and to make the deploy itself boring: a wrapped databricks bundle deploy, run by a trusted worker, recorded as an audited event. This page walks the full path from a built bundle to a running Databricks App.

The immutable artifact contract

The core idea is content addressing. A build is uploaded to a Unity Catalog Volume at a path derived from its own SHA-256 tree digest, and the digest is re-verified at the workspace before anything is deployed.

1. Stage (CLI side). fr deploy computes the Harness tree digest of the build directory and uploads it to a content-addressed path: artifactRef = <artifactVolume>/<app-slug>/<sha256>. Symlinks and non-file entries are rejected. The deploy request carries only the /Volumes/... reference and the 64-character SHA-256 digest — never the bytes.
2. Register. runway.artifact_register records the source revision, Harness version, and App/worker component manifest only after workspace verification succeeds. It returns a stable content-derived artifactId. Every required manifest entrypoint must exist inside the verified bytes; traversal and absolute paths are rejected. Production deploy actions accept that id, not caller-entered paths.
3. Materialize + verify (workspace edge). The Databricks adapter streams the Volume object into an isolated temp directory (with path-traversal protection), recomputes the digest, and throws on any mismatch before the bundle is validated. Tampering between stage and deploy is caught here.
4. Deploy. Only then does the adapter run databricks bundle validate and databricks bundle deploy --target <env> from the bundle root.

Schema enforces the shape: artifactRef must match ^/Volumes/<catalog>/<schema>/<volume>/… and artifactDigest must be a lowercase 64-hex SHA-256.

Asset Bundles

The scaffolded databricks.yml is a standard Databricks Asset Bundle with one target per environment. Runway passes the target and a per-environment secret scope variable (BUNDLE_VAR_secret_scope = runway-<appId>-<environment>) into the CLI; secrets are passed over stdin, never argv, and configured secret strings are scrubbed from all CLI stdout, stderr, and error messages.

For previews it first creates an isolated Lakebase copy-on-write branch and passes BUNDLE_VAR_lakebase_branch plus BUNDLE_VAR_lakebase_endpoint. It also passes BUNDLE_VAR_preview_suffix=-pr-N; the bundle root, App name, secret scope, Unity Catalog resources, and other non-database resources are therefore owned by that exact pull request and can be destroyed without touching another preview. See preview environments for lifecycle and failure behavior.

resources:
  apps:
    support-agent:
      name: support-agent-${bundle.target}
      source_code_path: ./
      resources:
        - name: ai-search-index
          secret:
            scope: ${var.secret_scope}
            key: ai-search-index

Databricks Apps

The template ships a long-running Databricks App. Its app.yaml declares the runtime command and the environment it reads, including secret-backed values resolved from the app's scope:

command: ["node", "dist/server.mjs"]
env:
  - name: DATABRICKS_APP_PORT
    value: "8080"
  - name: DATABRICKS_MODEL
    value: system.ai.gpt-oss-20b
  - name: DATABRICKS_AI_SEARCH_INDEX
    valueFrom: ai-search-index

After bundle deploy, Harness activates the App snapshot so the new revision serves traffic. Both app.yaml and databricks.yml are regenerated by the build step (fh build --target databricks-app) — you edit the agent, not the wiring.

Inside the installed Runway App, each governed adapter attempt obtains a fresh short-lived token from the App service principal and supplies it to Harness' CLI edge. The token is never a Platform action parameter or event field, and retries resolve a new credential rather than persisting an expiring bearer token. Harness command failures preserve stderr—or a stdout diagnostic when a CLI generation emits errors there—after Runway scrubs the short-lived token and client secret.

The same App exposes the shared Runway MCP registry at /api/v1/orgs/<org>/mcp. The route uses the App's verified user boundary and tenant membership, then delegates every tool to the existing org-scoped API. Register that URL as a schema-scoped HTTP connection and Unity Catalog MCP Service when AI Gateway or Agent Bricks should call Runway. See MCP tools for tool selection, permissions, health, expected behavior, and failure modes.

The customer build packages a pinned official Linux Databricks CLI archive after validating its published SHA-256. It splits the archive below the Databricks App per-file limit and re-verifies it before extraction at startup, so deployment does not inherit an outdated CLI from the managed App image or require runtime GitHub egress.

Durable external-effect evidence

Runway does not implement Databricks authentication or a second governance pipeline. Production composes Platform Host 0.7.0 with @fabric-harness/databricks/platform. Before policies run, the host persists the exact Databricks resource footprint and the App service principal delegated by the requesting actor. After each successful workspace adapter call, it records an execution attestation that identifies the operation and resource using only redacted adapter input and audit-safe output.

These records live in the same Lakebase transaction boundary and tenant/space namespace as action, policy, event, and adapter records. The bridge cannot serialize a bearer token or client secret. Runway's existing multi-approver release model remains the approval authority; the optional single-decision Host HITL evaluator stays unconfigured.

What a client workspace must provide

To run this path for real (not the local demo), a workspace needs:

• A Unity Catalog artifact Volume with WRITE_VOLUME, referenced by artifactVolume.
• The databricks CLI authenticated (or a service principal via --profile).
• A Lakebase / PostgreSQL database for the Platform Host durable store.
• CAN MANAGE on the configured Lakebase project so the trusted App principal can create and delete preview branches; branch TTL defaults to 24 hours.
• For Gateway-correlated evidence, a separate account-admin-authorized evidence reader and a SQL warehouse. The current Gateway Beta documents system.ai_gateway.usage as account-admin-only; do not elevate the ordinary Runway App principal solely to read it.
• A trusted worker service principal to execute prod (RUNWAY_TRUSTED_WORKER_IDS).
• For production, a Temporal Cloud namespace reachable from Databricks Apps over TLS, plus an API key or client certificate pair. A production install refuses to start without one; see Orchestration.
• Databricks Apps' default current-user and access-control read scopes for OBO workspace-group resolution; Runway does not request a broader user API scope.
• A Fabric Experiments API endpoint, immutable organization id, and release-to-dataset plus evaluator resolver for gates. Runway business-unit tenant ids remain independent from that organization, and governance gate labels remain independent from concrete evaluator names.
• If that endpoint is a Databricks App, CAN_USE for the Runway App service principal; Runway resolves a fresh upstream App OAuth token per Experiments request while the typed client also supplies the organization API key.

DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
RUNWAY_ARTIFACT_VOLUME=/Volumes/main/runway/artifacts
RUNWAY_EXPERIMENTS_URL=https://your-experiments-api
RUNWAY_EXPERIMENTS_ORGANIZATION_ID=org_enterprise
RUNWAY_EXPERIMENTS_ORGANIZATION_SLUG=enterprise-platform
RUNWAY_TRUSTED_WORKER_IDS=svc-runway-worker
RUNWAY_LAKEBASE_PROJECT=projects/my-runway-project
RUNWAY_LAKEBASE_SOURCE_BRANCH=production
RUNWAY_PREVIEW_BRANCH_TTL_SECONDS=86400

Credentials come from Databricks App resources, workload identity, or secret references — never from Runway action parameters or bundle variables. See production readiness for the full workspace checklist.

RUNWAY_SQL_WAREHOUSE_ID is intentionally absent from the default customer bundle. Set it only in a separately reviewed account-admin evidence-worker composition that injects createDatabricksGatewayEvidenceSource(), and set RUNWAY_GATEWAY_EVIDENCE_ACCOUNT_ADMIN_ACK=1 to acknowledge that privilege boundary. Production startup rejects the warehouse setting without that acknowledgement. Without the source, signed evidence reports the Gateway projection as not-configured. RUNWAY_REQUIRE_GATEWAY_EVIDENCE=1 is appropriate only in that worker; otherwise it would make every export fail while preserving the honest missing-source state.

Repeatable live certification

Runway includes a disposable live-workspace suite built on @fabricorg/databricks-testkit. It proves the real path rather than only checking that a bundle validates:

1. The Databricks test kit writes and removes a small Volume probe.
2. The suite probes system.ai_gateway.usage up front, so a certifier that cannot read the table is reported in seconds instead of surfacing later as an ambiguous correlation timeout.
3. Runway provisions a disposable internal tenant through request, worker bootstrap, and completion actions, grants each certification principal a role, then installs a deterministic two-party certification policy.
4. Runway asserts the authorization boundary refuses an ungranted principal, an operator attempting to grant itself admin, and a principal reaching outside its tenant. A suite of positive checks alone would pass identically with authorization disabled, so the denial is certified explicitly.
5. Harness stages the immutable certification App into the named Volume.
6. Runway submits app_register, artifact_register, and deploy_request through Platform Host.
7. Runway creates a TTL-bound Lakebase preview branch. Harness materializes and verifies the digest during registration, then validates/deploys the preview bundle against that branch and activates the App.
8. Runway sends one correlation-tagged AI Gateway model request for the governed deploy. This is synchronous and independent of any system table.
9. Runway reconciles that correlation against system.ai_gateway.usage. The join is eventually consistent, so it is checked separately from step 7 and a miss is classified rather than reported as a bare timeout.
10. Runway records a certification gate, collects distinct agent and human endorsements, pins the cycle policy, and finalizes through the trusted-worker action. The certification gate deliberately disables Experiments provenance; each client must separately certify its real dataset mapping.
11. Fabric Experiments requires the App to be running and its authenticated health route to pass.
12. Runway runs audit_export against a real Unity Catalog Volume and reads the JSONL and manifest back, confirming the manifest digest matches the retained bytes.
13. Runway runs evidence_export, reads the signed report back from the Volume, and verifies its SHA-256 and Ed25519 signature over the retained bytes. It then asserts the embedded Gateway projection joins the deploy correlation and that the signed observedAt is within ten minutes of the recorded request — a host-local timestamp parse would shift it by the host's UTC offset and fail the check.
14. Runway submits preview_destroy, removes the exact preview-owned App and Lakebase branch, and completes governed tenant retirement. The staged artifact directory is then removed.

The Gateway projection in a signed report is normally partial, not matched: a deployment spans several governed actions and only the deploy correlation carries a model call, so most queried correlations legitimately have no usage row. What the certification asserts is that the deploy correlation specifically is joined.

The command is gated because it creates and deletes workspace resources:

RUNWAY_CERTIFY_LIVE=1 pnpm certify:databricks:live -- \
  --profile <databricks-cli-profile> \
  --catalog <scratch-catalog> \
  --lakebase-project <project-id> \
  --sql-warehouse <warehouse-id> \
  --gateway-model <system.ai.model-service>

--catalog creates a run-scoped schema and managed Volume, then deletes both. To reuse a pre-provisioned test Volume instead, pass --volume /Volumes/<catalog>/<schema>/<artifact-volume>; the caller must already have USE CATALOG, USE SCHEMA, READ VOLUME, and WRITE VOLUME. Use --lakebase-source-branch <branch-id> only when the disposable preview must fork from something other than production. The caller also needs CAN MANAGE on the named Lakebase project. The current certifier identity must be an account admin with CAN USE on the SQL warehouse.

Gateway correlation is certified as two separate checks, because sending a tagged request and having its usage row retained fail for different reasons:

• runway-gateway-request sends one correlation-tagged AI Gateway request and passes as soon as the workspace accepts it. Synchronous and deterministic.
• runway-gateway-evidence polls system.ai_gateway.usage for that exact correlation for up to fifteen minutes (--gateway-evidence-timeout-seconds to override). A successful model response is not enough: the retained row must exist.

When the row never arrives, the suite classifies the miss instead of reporting a bare timeout — not-yet-delivered, no-access, no-recent-traffic, untagged-traffic, correlation-tag-absent, or lag-or-missing — and records the delivery frontier and observed request-tag keys. That distinguishes a denied reader from a tag that never landed from ordinary delivery lag, which a timeout alone cannot.

not-yet-delivered is evaluated first: when the table's newest row is older than the request, delivery simply has not reached it and no other cause is knowable. Because that frontier can run over an hour behind, the retained-row half can also be reconciled out of band instead of polling:

pnpm reconcile:gateway-evidence -- \
  --profile <profile> \
  --sql-warehouse <warehouse-id> \
  --correlation <corr_…> \
  --request-sent-at <iso-8601>

It exits 0 with the matched gatewayRequestIds once the row lands, and 1 with a classified diagnosis until then.

Because the usage table is a Beta surface with eventually-consistent delivery, you can certify the governed delivery path without waiting on it. --gateway-evidence-optional (or RUNWAY_CERTIFICATION_GATEWAY_EVIDENCE_OPTIONAL=1) demotes the reader-access and reconciliation checks to advisory: they still run and still record pass/fail in the evidence report, but they no longer gate suite success. The report's required array records exactly what was gated, so an advisory run cannot be mistaken for a full one. runway-gateway-request stays required either way. Do not enable RUNWAY_REQUIRE_GATEWAY_EVIDENCE=1 in production until runway-gateway-evidence has passed live on that account.

Evidence is written to reports/databricks/runway-live.json and JUnit output to reports/databricks/runway-live.xml. Missing profile/Volume configuration fails before mutation. Missing Lakebase, warehouse, or Gateway model configuration also fails before mutation. The suite uses the real Databricks adapter and governed domain pipeline but an ephemeral in-process projection; the customer-installation drill additionally certifies Lakebase restart recovery and the real Fabric Experiments resolver. If governed cleanup cannot run after a partial failure, the suite performs a narrowly scoped emergency bundle destroy for the unique preview suffix and reports the certification as failed; it never sweeps unrelated Apps or Volume paths.