[WIP] Add the databricks pipelines test command

.nextchanges/cli/pipelines-test.md

@@ -0,0 +1 @@
+* Add the `databricks pipelines test` command, which runs a pipeline's tests in a test-only update. ([#7297](https://github.com/databricks/cli/pull/7297))

acceptance/pipelines/databricks-cli-help/output.txt

@@ -30,6 +30,7 @@ Main Commands
   run                   Run a pipeline
   show                  Preview a table's columns and sample rows
   stop                  Stop a pipeline
+  test                  *Preview* Run the tests of a pipeline
 
 Management Commands
   apply-environment     *Public Preview* Apply the latest environment to the pipeline.

acceptance/pipelines/test/arguments/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/arguments/output.txt

@@ -0,0 +1,60 @@
+
+>>> [CLI] pipelines test --help
+Run the tests of a pipeline in a test-only update.
+
+This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
+
+PIPELINE_KEY is the unique name of the pipeline, as defined in its YAML file.
+It is required when test selections are given. If there is only one pipeline
+in the project and no arguments are given, it is selected automatically.
+Outside a project, identify the pipeline with --pipeline-id; all positional
+arguments are then test selections.
+
+TEST_SELECTION is a test directory, file, or pytest test ID, for example
+tests/test_orders.py::test_backfill. In a project, a relative path is a local
+path, relative to the working directory, and is converted to its deployed
+workspace path. With --pipeline-id, a relative path is relative to the
+pipeline's root_path. Paths that start with / are workspace paths. Without a
+selection, all of the pipeline's tests run.
+
+Each --pytest-arg passes one argument to pytest, in the order given. Only -k
+and -m are supported, each followed by its value in the next --pytest-arg, for
+example:
+
+  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
+
+Usage:
+  databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]
+
+Pipeline Test Flags:
+      --pipeline-id string       ID of the pipeline to test, instead of a PIPELINE_KEY.
+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
+
+Flags:
+  -h, --help   help for test
+
+Global Flags:
+      --debug            enable debug logging
+  -o, --output type      output type: text or json (default text)
+  -p, --profile string   ~/.databrickscfg profile
+  -t, --target string    bundle target to use (if applicable)
+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
+
+=== The command is experimental and off by default
+>>> musterr [CLI] pipelines test --pipeline-id 1234
+Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
+
+=== pytest arguments must be passed with --pytest-arg
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
+Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
+Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
+
+=== Invalid --pytest-arg values fail before any request
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
+Error: unsupported pytest argument "-x": only -k and -m are supported
+
+=== JSON output is not supported
+>>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
+Error: --output json is not supported by pipelines test

acceptance/pipelines/test/arguments/script

@@ -0,0 +1,14 @@
+trace $CLI pipelines test --help
+
+title "The command is experimental and off by default"
+(unset DATABRICKS_CLI_PIPELINES_TEST; trace musterr $CLI pipelines test --pipeline-id 1234)
+
+title "pytest arguments must be passed with --pytest-arg"
+trace musterr $CLI pipelines test --pipeline-id 1234 -k orders
+trace musterr $CLI pipelines test --pipeline-id 1234 -- -k orders
+
+title "Invalid --pytest-arg values fail before any request"
+trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=-x
+
+title "JSON output is not supported"
+trace musterr $CLI pipelines test --pipeline-id 1234 -o json

acceptance/pipelines/test/immutable/databricks.yml

@@ -0,0 +1,13 @@
+bundle:
+  name: test-pipeline-test-immutable
+
+workspace:
+  sync_mode: immutable
+
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-test-immutable
+      libraries:
+        - file:
+            path: pipeline_file.py

acceptance/pipelines/test/immutable/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/immutable/output.txt

@@ -0,0 +1,53 @@
+
+>>> [CLI] pipelines deploy
+Created internal_immutable_snapshots.immutable
+Created pipelines.my_pipeline
+Files: 0 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
+
+=== Local test paths are not supported with workspace.sync_mode immutable
+>>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
+
+=== Absolute workspace paths and running all tests still work
+>>> [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> [CLI] pipelines test my_pipeline
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true,
+    "test_options": {
+      "test_selection": [
+        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
+      ]
+    }
+  }
+}
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true
+  }
+}

acceptance/pipelines/test/immutable/pipeline_file.py

@@ -0,0 +1,6 @@
+from pyspark import pipelines as dp
+
+
+@dp.table
+def orders():
+    return spark.range(10)

acceptance/pipelines/test/immutable/script

@@ -0,0 +1,10 @@
+trace $CLI pipelines deploy
+rm "$OUT_REQUESTS"
+
+title "Local test paths are not supported with workspace.sync_mode immutable"
+trace musterr $CLI pipelines test my_pipeline tests/test_orders.py::test_backfill
+
+title "Absolute workspace paths and running all tests still work"
+trace $CLI pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill
+trace $CLI pipelines test my_pipeline
+trace print_requests.py //updates

acceptance/pipelines/test/immutable/tests/test_orders.py

@@ -0,0 +1,2 @@
+def test_backfill():
+    assert True

acceptance/pipelines/test/pipeline-id/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/pipeline-id/output.txt

@@ -0,0 +1,46 @@
+
+=== Relative test selections resolve against the pipeline root_path
+>>> [CLI] pipelines test --pipeline-id [UUID] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests
+Pipeline: [UUID]
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true,
+    "test_options": {
+      "test_selection": [
+        "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill",
+        "/Workspace/Users/someone@example.com/other/tests"
+      ]
+    }
+  }
+}
+
+=== Without a root_path, test selections must be workspace paths
+>>> musterr [CLI] pipelines test --pipeline-id [UUID] tests
+Error: cannot resolve test selection "tests": the pipeline has no root_path; use an absolute workspace path such as /Workspace/Users/someone@example.com/tests
+
+>>> [CLI] pipelines test --pipeline-id [UUID] /Workspace/Users/someone@example.com/tests/test_other.py::test_x
+Pipeline: [UUID]
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true,
+    "test_options": {
+      "test_selection": [
+        "/Workspace/Users/someone@example.com/tests/test_other.py::test_x"
+      ]
+    }
+  }
+}
+
+=== Outside a project, --pipeline-id is required
+>>> musterr [CLI] pipelines test tests
+Error: outside a project, identify the pipeline with --pipeline-id

acceptance/pipelines/test/pipeline-id/script

@@ -0,0 +1,15 @@
+pipeline_id=$($CLI pipelines create --json '{"name": "orders", "root_path": "/Workspace/Users/someone@example.com/orders"}' | jq -r .pipeline_id)
+no_root_pipeline_id=$($CLI pipelines create --json '{"name": "no-root"}' | jq -r .pipeline_id)
+rm "$OUT_REQUESTS"
+
+title "Relative test selections resolve against the pipeline root_path"
+trace $CLI pipelines test --pipeline-id $pipeline_id tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests
+trace print_requests.py //updates
+
+title "Without a root_path, test selections must be workspace paths"
+trace musterr $CLI pipelines test --pipeline-id $no_root_pipeline_id tests
+trace $CLI pipelines test --pipeline-id $no_root_pipeline_id /Workspace/Users/someone@example.com/tests/test_other.py::test_x
+trace print_requests.py //updates
+
+title "Outside a project, --pipeline-id is required"
+trace musterr $CLI pipelines test tests

acceptance/pipelines/test/selection/databricks.yml

@@ -0,0 +1,10 @@
+bundle:
+  name: test-pipeline-test-selection
+
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-test-selection
+      libraries:
+        - file:
+            path: pipeline_file.py

acceptance/pipelines/test/selection/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/selection/output.txt

@@ -0,0 +1,83 @@
+
+>>> [CLI] pipelines deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files...
+Created pipelines.my_pipeline
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
+
+=== Run all tests of the only pipeline
+>>> [CLI] pipelines test
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true
+  }
+}
+
+=== Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms
+>>> [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg -m --pytest-arg smoke,fast
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true,
+    "test_options": {
+      "test_selection": [
+        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests",
+        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_backfill",
+        "/Workspace/Users/someone@example.com/tests/test_other.py"
+      ],
+      "pytest_args": [
+        "-k",
+        "orders and not slow",
+        "-m",
+        "smoke,fast"
+      ]
+    }
+  }
+}
+
+=== Relative paths resolve from the working directory
+>>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order
+Recommendation: This command runs the last deployed version of the code
+
+If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
+
+Pipeline: my_pipeline ([UUID])
+Update: [UUID]
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/[UUID]/updates",
+  "body": {
+    "test_only": true,
+    "test_options": {
+      "test_selection": [
+        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_valid_order"
+      ]
+    }
+  }
+}
+
+=== Test selections require the pipeline KEY first
+>>> musterr [CLI] pipelines test tests/test_orders.py
+Error: resource with key "tests/test_orders.py" not found; the first argument must be a pipeline KEY when test selections are given

acceptance/pipelines/test/selection/pipeline_file.py

@@ -0,0 +1,6 @@
+from pyspark import pipelines as dp
+
+
+@dp.table
+def orders():
+    return spark.range(10)

acceptance/pipelines/test/selection/script

@@ -0,0 +1,19 @@
+trace $CLI pipelines deploy
+rm "$OUT_REQUESTS"
+
+title "Run all tests of the only pipeline"
+trace $CLI pipelines test
+trace print_requests.py //updates
+
+title "Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms"
+trace $CLI pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg='orders and not slow' --pytest-arg -m --pytest-arg smoke,fast
+trace print_requests.py //updates
+
+title "Relative paths resolve from the working directory"
+cd tests
+trace $CLI pipelines test my_pipeline test_orders.py::test_valid_order
+cd ..
+trace print_requests.py //updates
+
+title "Test selections require the pipeline KEY first"
+trace musterr $CLI pipelines test tests/test_orders.py

acceptance/pipelines/test/selection/tests/test_orders.py

@@ -0,0 +1,6 @@
+def test_valid_order():
+    assert True
+
+
+def test_backfill():
+    assert True

acceptance/pipelines/test/submit-error/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/submit-error/output.txt

@@ -0,0 +1,13 @@
+
+=== A submit error from the service is shown as is, and no update ID is printed
+>>> musterr [CLI] pipelines test --pipeline-id 1234
+Error: An active update '[UUID]' already exists for pipeline '1234'.
+
+>>> print_requests.py //updates
+{
+  "method": "POST",
+  "path": "/api/2.0/pipelines/1234/updates",
+  "body": {
+    "test_only": true
+  }
+}

acceptance/pipelines/test/submit-error/script

@@ -0,0 +1,3 @@
+title "A submit error from the service is shown as is, and no update ID is printed"
+trace musterr $CLI pipelines test --pipeline-id 1234
+trace print_requests.py //updates