[WIP] Wait for and print the results of databricks pipelines test

.nextchanges/cli/pipelines-test-results.md

@@ -0,0 +1 @@
+* `databricks pipelines test` waits for the test results, prints them, and fails when a test fails or the results are incomplete. ([#7298](https://github.com/databricks/cli/pull/7298))

acceptance/pipelines/test/arguments/output.txt

@@ -23,12 +23,20 @@ example:
 
   databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
 
+The command waits for the results and succeeds only if at least one test was
+collected and none failed or errored. --no-wait returns after the update starts.
+--timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
+to read the results later. To stop the update, run 'pipelines stop'.
+
 Usage:
   databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]
 
 Pipeline Test Flags:
+      --no-wait                  Return after the test update starts, without waiting for results.
       --pipeline-id string       ID of the pipeline to test, instead of a PIPELINE_KEY.
       --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
+      --resume string            Read the results of an existing test update instead of starting one.
+      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
 
 Flags:
   -h, --help   help for test
@@ -55,6 +63,25 @@ Error: the -- separator is not supported: pass pytest arguments with --pytest-ar
 >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
 Error: unsupported pytest argument "-x": only -k and -m are supported
 
+=== Incompatible run-control flags
+>>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
+Error: --resume cannot be used with test selections
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
+Error: --resume cannot be used with --pytest-arg
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
+Error: --resume cannot be used with --no-wait
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
+Error: --timeout cannot be used with --no-wait
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
+Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
+
+>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 
+Error: --resume "" is not an update ID; pass the update ID that pipelines test printed
+
 === JSON output is not supported
 >>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
 Error: --output json is not supported by pipelines test

acceptance/pipelines/test/arguments/script

@@ -10,5 +10,13 @@ trace musterr $CLI pipelines test --pipeline-id 1234 -- -k orders
 title "Invalid --pytest-arg values fail before any request"
 trace musterr $CLI pipelines test --pipeline-id 1234 --pytest-arg=-x
 
+title "Incompatible run-control flags"
+trace musterr $CLI pipelines test --pipeline-id 1234 tests --resume 5678
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume 5678 --no-wait
+trace musterr $CLI pipelines test --pipeline-id 1234 --no-wait --timeout 10m
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume "5678' OR '1'='1"
+trace musterr $CLI pipelines test --pipeline-id 1234 --resume ""
+
 title "JSON output is not supported"
 trace musterr $CLI pipelines test --pipeline-id 1234 -o json

acceptance/pipelines/test/immutable/output.txt

@@ -15,15 +15,15 @@ If you've made local changes, run 'databricks pipelines deploy' first to ensure
 Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
 
 === Absolute workspace paths and running all tests still work
->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill
+>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
 Recommendation: This command runs the last deployed version of the code
 
 If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
 
 Pipeline: my_pipeline ([UUID])
 Update: [UUID]
 
->>> [CLI] pipelines test my_pipeline
+>>> [CLI] pipelines test my_pipeline --no-wait
 Recommendation: This command runs the last deployed version of the code
 
 If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.

acceptance/pipelines/test/immutable/script

@@ -5,6 +5,8 @@ title "Local test paths are not supported with workspace.sync_mode immutable"
 trace musterr $CLI pipelines test my_pipeline tests/test_orders.py::test_backfill
 
 title "Absolute workspace paths and running all tests still work"
-trace MSYS_NO_PATHCONV=1 $CLI pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill
-trace $CLI pipelines test my_pipeline
+trace MSYS_NO_PATHCONV=1 $CLI pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
+# Like the service, the test server rejects a new update while one is active.
+$CLI pipelines stop my_pipeline >> LOG.stop 2>&1
+trace $CLI pipelines test my_pipeline --no-wait
 trace print_requests.py //updates

acceptance/pipelines/test/many-cases/databricks.yml

@@ -0,0 +1,11 @@
+bundle:
+  name: test-pipeline-test-many-cases
+
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-test-many-cases
+      root_path: .
+      libraries:
+        - file:
+            path: pipeline_file.py

acceptance/pipelines/test/many-cases/out.test.toml

@@ -0,0 +1 @@
+Cloud = false

acceptance/pipelines/test/many-cases/output.txt

@@ -0,0 +1,13 @@
+
+>>> [CLI] pipelines deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-many-cases/default/files...
+Created pipelines.my_pipeline
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
+
+=== Every case is counted once across pages, polls, and the final read
+>>> tail -n 1 LOG.many
+200 tests: 200 passed, 0 failed, 0 skipped, 0 errors
+Event pages read: 8
+Pages read with page_token: 4

acceptance/pipelines/test/many-cases/pipeline_file.py

@@ -0,0 +1,6 @@
+from pyspark import pipelines as dp
+
+
+@dp.table
+def orders():
+    return spark.range(10)

acceptance/pipelines/test/many-cases/script

@@ -0,0 +1,9 @@
+trace $CLI pipelines deploy
+rm "$OUT_REQUESTS"
+
+title "Every case is counted once across pages, polls, and the final read"
+$CLI pipelines test my_pipeline &> LOG.many
+trace tail -n 1 LOG.many
+print_requests.py --get //events --oneline > LOG.events
+echo "Event pages read: $(wc -l < LOG.events | tr -d ' ')"
+echo "Pages read with page_token: $(grep -c page_token LOG.events)"