Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/about/whats-new-23.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ DataJoint's `<blob@>` codec stores arbitrary Python values as serialized binary

- It's a small, opt-in `@runtime_checkable` `Protocol` — no inheritance, no abstract-method burden on existing codecs.
- Generic codecs (`<blob@>`, `<hash@>`) remain non-adapting by design; typed codecs (`<float_array@>`, `<image_2d@>`, and future shapes) opt in.
- The downstream driver is Databricks Linked Delta Tables (the "silver layer"), where every column must render to a Spark-native type to be queryable.
- The downstream driver is publishing to a Databricks silver layer, where every column must render to a Spark-native type to be queryable.

See the [SparkAdapter Codec Protocol Specification](../reference/specs/spark-adapter.md) and the [explainer](../explanation/spark-adapters.md).

Expand Down
4 changes: 2 additions & 2 deletions src/assets/stylesheets/extra.css
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@
/* links */
/* Links: body-size text needs AA (4.5:1); DataJoint Blue on white is 2.96:1.
#00537A is the brand-blue family's AA text value (7.35:1) — the same value the
diagram theme uses for Imported-tier text. Provisional under dj-brand#22
(the working-palette ruling); swap to the ratified token when it lands. */
diagram theme uses for Imported-tier text. Provisional until the brand's
working palette is ratified; swap to the ratified token then. */
--md-typeset-a-color: #00537A;
/* single interactive emphasis: hover, focus, highlights */
--md-accent-fg-color: var(--dj-orange);
Expand Down
1 change: 0 additions & 1 deletion src/explanation/spark-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,3 @@ For pipelines that are predominantly internal (no analytic publish), `<blob@>` e
- Spec: [Codec API](../reference/specs/codec-api.md) — the base `Codec` interface.
- How-to: [Use Plugin Codecs](../how-to/use-plugin-codecs.md) — installing and registering codec plugins.
- How-to: [Create Custom Codecs](../how-to/create-custom-codec.md) — writing your own codec.
- Strategic background: Databricks integration's Linked Delta Tables — covered in `datajoint-databricks/DECISIONS.md` (internal repo; summary on request).
6 changes: 3 additions & 3 deletions src/reference/specs/spark-adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ This document specifies the `SparkAdapter` Protocol — an opt-in contract that
The Protocol is intentionally small: a single method, no inheritance, no abstract-method surface area on the existing `Codec` base class. Consumers detect support via `isinstance(codec, SparkAdapter)`.

!!! version-added "New in DataJoint 2.3"
Introduced for the Databricks Linked Delta Tables ("silver layer") integration. Independent of any consumer: any codec author can implement `to_spark()` and any downstream tool can check for it.
Introduced for publishing DataJoint tables to a Databricks silver layer. Independent of any consumer: any codec author can implement `to_spark()` and any downstream tool can check for it.

For the `Codec` base class itself, see [Codec API Specification](codec-api.md). For background on why the protocol is needed, see [Spark Adapters and Silver-Layer Publishing](../../explanation/spark-adapters.md).

## Why this exists

DataJoint's `<blob@>` codec stores arbitrary Python values — numpy arrays, lists, dicts, custom objects — as serialized binary. That generality is the right call for most use cases, but it makes the column **opaque** to consumers that need typed access at query time: SQL engines, dataframe libraries, BI tools, federated query systems. Such consumers can only treat blob columns as `BINARY` and rely on the application to decode.

The downstream concrete case is **Databricks Linked Delta Tables** (the "silver layer"): every column must render to Spark-native types so the data is queryable with Spark SQL, exposed through Delta Sharing, and usable by Genie / BI without round-tripping through DataJoint. A `BINARY` blob doesn't qualify.
The downstream concrete case is a **Databricks silver layer**: every column must render to Spark-native types so the data is queryable with Spark SQL, exposed through Delta Sharing, and usable by Genie / BI without round-tripping through DataJoint. A `BINARY` blob doesn't qualify.

`SparkAdapter` is the minimum framework-side contract to make typed rendering possible **without** forcing every codec to support it. Generic codecs (`<blob@>`, `<hash@>`) remain unsupported by design. Typed codecs (`<float_array@>`, `<image_2d@>`, future shapes) opt in by implementing the method.

Expand Down Expand Up @@ -155,7 +155,7 @@ class PointWithLabelCodec(dj.Codec):

## Consumer pattern

A simplified silver-layer publish loop (the actual `datajoint-databricks` consumer is more elaborate):
A simplified silver-layer publish loop (a production consumer is more elaborate):

```python
def publish_row_to_silver(table, key, target_table):
Expand Down
2 changes: 1 addition & 1 deletion src/reference/specs/trace.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ Teams adopt the read surface incrementally:
## What is not in this specification

- **Enforcement of the `make()` contract.** `trace` and `self.upstream` make the contract easy to follow and to inspect, but the DataJoint Library does not check it at runtime. Confirming that a `make()` reads only from its declared ancestors and writes only to its target, across every access path, is a code-inspection problem — performed by the DataJoint Platform's code-deployment CI/CD (static analysis where sound, agentic review for the dynamic cases), backed by commit-pinned governed execution. See the [make() reproducibility contract](autopopulate.md#43-the-make-reproducibility-contract).
- **Row-level lineage metadata in storage**. The features here provide the *graph operation* and the *per-`make()` read surface*; persisting per-row upstream projections (the dj-delta-style silver-layer features) is a downstream consumer concern, tracked separately.
- **Row-level lineage metadata in storage**. The features here provide the *graph operation* and the *per-`make()` read surface*; persisting per-row upstream projections (as silver-layer publishing does) is a downstream consumer concern, tracked separately.

## References

Expand Down