Repository navigation
Detect notebooks with the workspace's first-line rule - #7290
Open
Sankalp-Mittal wants to merge 2 commits into
Open
Sankalp-Mittal wants to merge 2 commits into
Sankalp-Mittal wants to merge 2 commits into
Conversation
The CLI classifies a local source file as a notebook only to predict the remote name the workspace will give it (sync snapshot, translate_paths, deployment state, workspace import-dir). It required the first line to equal the language-specific header exactly, while the import-file endpoint only requires the first line of the leading 200 bytes to contain "Databricks notebook source" (NotebookUtils.checkSourceFileNotebook). A BOM-prefixed SQL notebook therefore landed as notebook "bom" while the sync snapshot tracked "bom.sql"; removing it locally deleted a path that did not exist and left the notebook behind. detectSource now applies the server's rule. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/bundle/sync/replace-node-type/DMS=: branch ✅ | main (df1a491) ❌ | latest (v1.23.0) ❌main (df1a491): latest (v1.23.0): same output as main (df1a491) TestAccept/bundle/sync/replace-node-type/DMS=true: branch ✅ | main (df1a491) ❌ | latest (v1.23.0) ❌main (df1a491): latest (v1.23.0): same output as main (df1a491) |
Collaborator
Integration test reportCommit: b483479
Top 6 slowest tests (at least 2 minutes):
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
libs/notebooknow classifies a source file the way the workspace'simport-fileendpoint does (NotebookUtils.checkSourceFileNotebookin universe): a.py,.sql,.scalaor.rfile is a notebook when the first line of its leading 200 bytes containsDatabricks notebook source. Previously the CLI required the first line to equal the language-specific header exactly, so a leading byte order mark, or a different comment prefix, made the CLI treat a file the workspace imports as a notebook as a plain file.The CLI only uses this classification to predict the remote name the workspace will give a file: in the sync snapshot,
translate_paths, deployment state andworkspace import-dir. When the prediction disagrees with the server, the snapshot tracksbom.sqlwhile the workspace stored notebookbom. Removing the file locally then deletes a path that doesn't exist, which sync tolerates, and the notebook is orphaned. This is the badness documented in #7247.Why
Since #7218 the server classifies uploads again, so the CLI's local rule has to match the server's for the snapshot to address the right remote object.
Tests
bundle/sync/replace-node-typenow also removes both local files and syncs again; bothbomandplain.sqlare deleted. Passes locally and onaws-cliin both DMS variants. On the real workspace the old code'sbom.sqldelete 404'd and leftbombehind. This supersedes the test extension in Add acceptance coverage for BOM notebook deletion badness #7247.TestDetectBytesgains cases for the marker mid-line, another language's comment prefix, the marker on the second line, and the marker past the 200-byte buffer.libs/notebook:libs/sync,libs/fileset,libs/filer,bundle/config/mutator,bundle/deploy,bundle/generate,cmd/workspace/workspace.templates/default-python/*,templates/pydabs/deploy-classic,artifacts/issue_3109,migrate/command/default-python), all becauseuvcan't resolvehatchlingthrough the internal proxy in my environment. CI covers them.This pull request and its description were written by Isaac.