From 7b83989ffc458d48d9b3da3500fc8d72edc7ee9d Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Tue, 13 Oct 2026 14:32:56 -0500 Subject: [PATCH] docs: describe downstream Spark consumers generically Five places pointed readers at material outside these docs that they cannot open: a repository, a decisions file, an issue tracker, and two internal names. Describe the downstream consumer as a Databricks silver layer instead. Every technical statement stays. --- src/about/whats-new-23.md | 2 +- src/assets/stylesheets/extra.css | 4 ++-- src/explanation/spark-adapters.md | 1 - src/reference/specs/spark-adapter.md | 6 +++--- src/reference/specs/trace.md | 2 +- 5 files changed, 7 insertions(+), 8 deletions(-) diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index 4581266b..ad01b9ca 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -109,7 +109,7 @@ DataJoint's `` codec stores arbitrary Python values as serialized binary - It's a small, opt-in `@runtime_checkable` `Protocol` — no inheritance, no abstract-method burden on existing codecs. - Generic codecs (``, ``) remain non-adapting by design; typed codecs (``, ``, and future shapes) opt in. -- The downstream driver is Databricks Linked Delta Tables (the "silver layer"), where every column must render to a Spark-native type to be queryable. +- The downstream driver is publishing to a Databricks silver layer, where every column must render to a Spark-native type to be queryable. See the [SparkAdapter Codec Protocol Specification](../reference/specs/spark-adapter.md) and the [explainer](../explanation/spark-adapters.md). diff --git a/src/assets/stylesheets/extra.css b/src/assets/stylesheets/extra.css index 3930b1aa..22f69e79 100644 --- a/src/assets/stylesheets/extra.css +++ b/src/assets/stylesheets/extra.css @@ -55,8 +55,8 @@ /* links */ /* Links: body-size text needs AA (4.5:1); DataJoint Blue on white is 2.96:1. #00537A is the brand-blue family's AA text value (7.35:1) — the same value the - diagram theme uses for Imported-tier text. Provisional under dj-brand#22 - (the working-palette ruling); swap to the ratified token when it lands. */ + diagram theme uses for Imported-tier text. Provisional until the brand's + working palette is ratified; swap to the ratified token then. */ --md-typeset-a-color: #00537A; /* single interactive emphasis: hover, focus, highlights */ --md-accent-fg-color: var(--dj-orange); diff --git a/src/explanation/spark-adapters.md b/src/explanation/spark-adapters.md index 2bc76c1c..4f9d0aae 100644 --- a/src/explanation/spark-adapters.md +++ b/src/explanation/spark-adapters.md @@ -86,4 +86,3 @@ For pipelines that are predominantly internal (no analytic publish), `` e - Spec: [Codec API](../reference/specs/codec-api.md) — the base `Codec` interface. - How-to: [Use Plugin Codecs](../how-to/use-plugin-codecs.md) — installing and registering codec plugins. - How-to: [Create Custom Codecs](../how-to/create-custom-codec.md) — writing your own codec. -- Strategic background: Databricks integration's Linked Delta Tables — covered in `datajoint-databricks/DECISIONS.md` (internal repo; summary on request). diff --git a/src/reference/specs/spark-adapter.md b/src/reference/specs/spark-adapter.md index 5a32addc..f7099e01 100644 --- a/src/reference/specs/spark-adapter.md +++ b/src/reference/specs/spark-adapter.md @@ -5,7 +5,7 @@ This document specifies the `SparkAdapter` Protocol — an opt-in contract that The Protocol is intentionally small: a single method, no inheritance, no abstract-method surface area on the existing `Codec` base class. Consumers detect support via `isinstance(codec, SparkAdapter)`. !!! version-added "New in DataJoint 2.3" - Introduced for the Databricks Linked Delta Tables ("silver layer") integration. Independent of any consumer: any codec author can implement `to_spark()` and any downstream tool can check for it. + Introduced for publishing DataJoint tables to a Databricks silver layer. Independent of any consumer: any codec author can implement `to_spark()` and any downstream tool can check for it. For the `Codec` base class itself, see [Codec API Specification](codec-api.md). For background on why the protocol is needed, see [Spark Adapters and Silver-Layer Publishing](../../explanation/spark-adapters.md). @@ -13,7 +13,7 @@ For the `Codec` base class itself, see [Codec API Specification](codec-api.md). DataJoint's `` codec stores arbitrary Python values — numpy arrays, lists, dicts, custom objects — as serialized binary. That generality is the right call for most use cases, but it makes the column **opaque** to consumers that need typed access at query time: SQL engines, dataframe libraries, BI tools, federated query systems. Such consumers can only treat blob columns as `BINARY` and rely on the application to decode. -The downstream concrete case is **Databricks Linked Delta Tables** (the "silver layer"): every column must render to Spark-native types so the data is queryable with Spark SQL, exposed through Delta Sharing, and usable by Genie / BI without round-tripping through DataJoint. A `BINARY` blob doesn't qualify. +The downstream concrete case is a **Databricks silver layer**: every column must render to Spark-native types so the data is queryable with Spark SQL, exposed through Delta Sharing, and usable by Genie / BI without round-tripping through DataJoint. A `BINARY` blob doesn't qualify. `SparkAdapter` is the minimum framework-side contract to make typed rendering possible **without** forcing every codec to support it. Generic codecs (``, ``) remain unsupported by design. Typed codecs (``, ``, future shapes) opt in by implementing the method. @@ -155,7 +155,7 @@ class PointWithLabelCodec(dj.Codec): ## Consumer pattern -A simplified silver-layer publish loop (the actual `datajoint-databricks` consumer is more elaborate): +A simplified silver-layer publish loop (a production consumer is more elaborate): ```python def publish_row_to_silver(table, key, target_table): diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 27b9fe2e..738a015d 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -289,7 +289,7 @@ Teams adopt the read surface incrementally: ## What is not in this specification - **Enforcement of the `make()` contract.** `trace` and `self.upstream` make the contract easy to follow and to inspect, but the DataJoint Library does not check it at runtime. Confirming that a `make()` reads only from its declared ancestors and writes only to its target, across every access path, is a code-inspection problem — performed by the DataJoint Platform's code-deployment CI/CD (static analysis where sound, agentic review for the dynamic cases), backed by commit-pinned governed execution. See the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract). -- **Row-level lineage metadata in storage**. The features here provide the *graph operation* and the *per-`make()` read surface*; persisting per-row upstream projections (the dj-delta-style silver-layer features) is a downstream consumer concern, tracked separately. +- **Row-level lineage metadata in storage**. The features here provide the *graph operation* and the *per-`make()` read surface*; persisting per-row upstream projections (as silver-layer publishing does) is a downstream consumer concern, tracked separately. ## References