[tharunbhuk-db]

🥞 Stacked PR

• stack/pipelines-test-pr1-command [Files changed]
  • stack/pipelines-test-pr2-output [Files changed] ← this PR
    • stack/pipelines-test-pr3-junit [Files changed]
      • stack/pipelines-test-pr4-bundles [Files changed]

---

Design doc: https://docs.google.com/document/d/1atW-JG_hQu-G-YX-m1M7OodNZdH7LMc9H7CBlQAv5gw/edit

Changes

• Wait for the test-only update, and print each test case as it finishes, the failure details and a summary line. Test IDs are shown in the shortest form that selects the test again.
• --no-wait, --resume <update-id> and --timeout. A timeout or Ctrl+C ends only the local wait and prints the --resume command. --resume rejects a value that is not an update ID, and an update that the service reports is not a test update.
• Exit code 0 only for a complete, passing run. Otherwise exit code 1, with RUN INCOMPLETE: <reason> when results are missing.

Why

CI needs the results in the log, and an exit code that fails the build when a test fails. The events are read with a raw call, because the Go SDK does not have their test fields yet, and its iterator repeats order_by with page_token, which the service rejects.

Tests

• Unit tests for the event reader, the completeness check, the output, the exit-code rules and the fake service.
• Acceptance tests in acceptance/pipelines/test/{results,run-control,many-cases,pipeline-id,selection,immutable,arguments}.

This pull request and its description were written by Isaac.

[github-actions]

Commit: 854236574 | Workflow run

Regression test report

Selected 8 changed tests (limit=20, 0 not selected)

8 tests: regression=8 unreleased=0 coverage=0 failing=0

test	branch	main (aed6702)	latest (v1.23.0)
TestAccept/pipelines/databricks-cli-help	✅	❌	❌
TestAccept/pipelines/test/arguments	✅	❌	❌
TestAccept/pipelines/test/immutable	✅	❌	❌
TestAccept/pipelines/test/many-cases	✅	❌	❌
TestAccept/pipelines/test/pipeline-id	✅	❌	❌
TestAccept/pipelines/test/results	✅	❌	❌
TestAccept/pipelines/test/run-control	✅	❌	❌
TestAccept/pipelines/test/selection	✅	❌	❌

TestAccept/pipelines/databricks-cli-help: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help1454244489/002/output.txt
        @@ -30,7 +30,6 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/arguments: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments2924068585/002/output.txt
        @@ -1,97 +1,850 @@
         
         >>> [CLI] pipelines test --help
        -Run the tests of a pipeline in a test-only update.
        +The Lakeflow Spark Declarative Pipelines API allows you to create, edit,
        +  delete, start, and view details about pipelines.
         
        -PIPELINE_KEY is the unique name of the pipeline, as defined in its YAML file.
        -It is required when test selections are given. If there is only one pipeline
        -in the project and no arguments are given, it is selected automatically.
        -Outside a project, identify the pipeline with --pipeline-id; all positional
        -arguments are then test selections.
        +  Spark Declarative Pipelines is a framework for building reliable,
        +  maintainable, and testable data processing pipelines. You define the
        +  transformations to perform on your data, and Spark Declarative Pipelines
        +  manages task orchestration, cluster management, monitoring, data quality, and
        +  error handling.
         
        -TEST_SELECTION is a test directory, file, or pytest test ID, for example
        -tests/test_orders.py::test_backfill. In a project, a relative path is a local
        -path, relative to the working directory, and is converted to its deployed
        -workspace path. With --pipeline-id, a relative path is relative to the
        -pipeline's root_path. Paths that start with / are workspace paths. Without a
        -selection, all of the pipeline's tests run.
        +  Instead of defining your data pipelines using a series of separate Apache
        +  Spark tasks, Spark Declarative Pipelines manages how your data is transformed
        +  based on a target schema you define for each processing step. You can also
        +  enforce data quality with Spark Declarative Pipelines expectations.
        +  Expectations allow you to define expected data quality and specify how to
        +  handle records that fail those expectations.
         
        -Each --pytest-arg passes one argument to pytest, in the order given. Only -k
        -and -m are supported, each followed by its value in the next --pytest-arg, for
        -example:
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -  databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -The command waits for the results and succeeds only if at least one test was
        -collected and none failed or errored. --no-wait returns after the update starts.
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -Usage:
        -  databricks pipelines test [PIPELINE_KEY] [TEST_SELECTION...] [flags] [--pytest-arg=ARG ...]
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -Pipeline Test Flags:
        -      --no-wait                  Return after the test update starts, without waiting for results.
        -      --pipeline-id string       ID of the pipeline to test, instead of a PIPELINE_KEY.
        -      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        -
         Flags:
        -  -h, --help   help for test
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        -      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
         === pytest arguments must be passed with --pytest-arg
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        -Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -msmoke
        -Error: -m is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
... (702 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/immutable: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable1929069959/002/output.txt
        @@ -8,46 +8,108 @@
         
         === Local test paths are not supported with workspace.sync_mode immutable
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown command "test" for "databricks pipelines"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": local test paths are not supported with workspace.sync_mode immutable; pass an absolute workspace path
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         === Absolute workspace paths and running all tests still work
         >>> [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/many-cases: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1341
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1082
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:641
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] pipelines deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-many-cases/default/files...\nCreated pipelines.my_pipeline\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\nView your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]\n\n=== Every case is counted once across pages, polls, and the final read\n>>> tail -n 1 LOG.many\n200 tests: 200 passed, 0 failed, 0 skipped, 0 errors\nEvent pages read: 8\nPages read with page_token: 4\n"
        	            	actual  : "\n>>> [CLI] pipelines deploy\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-many-cases/default/files...\nCreated pipelines.my_pipeline\nFiles: 3 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\nView your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]\n\n=== Every case is counted once across pages, polls, and the final read\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -9,6 +9,3 @@
        	            	 === Every case is counted once across pages, polls, and the final read
        	            	->>> tail -n 1 LOG.many
        	            	-200 tests: 200 passed, 0 failed, 0 skipped, 0 errors
        	            	-Event pages read: 8
        	            	-Pages read with page_token: 4
        	            	+Exit code: 1
        	            	 
        	Test:       	TestAccept/pipelines/test/many-cases
        	Messages:   	pipelines/test/many-cases/output.txt vs /tmp/TestAcceptpipelinestestmany-cases2613729136/002/output.txt
    acceptance_test.go:1119: 
        LOG.many: Error: unknown command "test" for "databricks pipelines"
        LOG.many: 
        LOG.many: Usage:
        LOG.many:   databricks pipelines [flags]
        LOG.many:   databricks pipelines [command]
        LOG.many: 
        LOG.many: Main Commands
        LOG.many:   deploy                Deploy pipelines
        LOG.many:   destroy               Destroy a pipelines project
        LOG.many:   dry-run               Validate correctness of the pipeline's graph
        LOG.many:   generate              Generate pipeline configuration
        LOG.many:   history               Retrieve past runs for a pipeline
        LOG.many:   init                  Initialize a new pipelines project
        LOG.many:   logs                  Retrieve events for a pipeline
        LOG.many:   open                  Open a pipeline in the browser
        LOG.many:   run                   Run a pipeline
        LOG.many:   show                  Preview a table's columns and sample rows
        LOG.many:   stop                  Stop a pipeline
        LOG.many: 
        LOG.many: Management Commands
        LOG.many:   apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        LOG.many:   clone                 Clone a pipeline.
        LOG.many:   create                Create a pipeline.
        LOG.many:   delete                Delete a pipeline.
        LOG.many:   get                   Get a pipeline.
        LOG.many:   get-update            Get a pipeline update.
        LOG.many:   list-pipeline-events  List pipeline events.
        LOG.many:   list-pipelines        List pipelines.
        LOG.many:   list-updates          List pipeline updates.
        LOG.many:   start-update          Start a pipeline.
        LOG.many:   update                Edit a pipeline.
        LOG.many: 
        LOG.many: Permission Commands
        LOG.many:   get-permission-levels Get pipeline permission levels.
        LOG.many:   get-permissions       Get pipeline permissions.
        LOG.many:   set-permissions       Set pipeline permissions.
        LOG.many:   update-permissions    Update pipeline permissions.
        LOG.many: 
        LOG.many: Flags:
        LOG.many:   -h, --help          help for pipelines
        LOG.many:       --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        LOG.many: 
        LOG.many: Global Flags:
        LOG.many:       --debug            enable debug logging
        LOG.many:   -o, --output type      output type: text or json (default text)
        LOG.many:   -p, --profile string   ~/.databrickscfg profile
        LOG.many:   -t, --target string    bundle target to use (if applicable)
        LOG.many: 
        LOG.many: Use "databricks pipelines [command] --help" for more information about a command.

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/pipeline-id: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/pipeline-id/output.txt
        +++ /tmp/TestAcceptpipelinestestpipeline-id2839528330/002/output.txt
        @@ -1,66 +1,55 @@
         
         === Relative test selections resolve against the pipeline root_path
         >>> [CLI] pipelines test --pipeline-id [UUID] tests/test_orders.py::test_backfill /Workspace/Shared/tests --no-wait
        -Pipeline: [UUID]
        -Update: [UUID]
        +Error: unknown flag: --pipeline-id
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill",
        -        "/Workspace/Shared/tests"
        -      ]
        -    }
        -  }
        -}
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -=== Test case IDs are shown relative to the pipeline root_path
        ->>> [CLI] pipelines test --pipeline-id [UUID] tests
        -Pipeline: [UUID]
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -=== Without a root_path, test selections must be workspace paths
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID] tests
        -Error: cannot resolve test selection "tests": the pipeline has no root_path; use an absolute workspace path such as /Workspace/Users/someone@example.com/tests
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        ->>> [CLI] pipelines test --pipeline-id [UUID] /Workspace/Shared/tests/test_shared.py::test_x --no-wait
        -Pipeline: [UUID]
        -Update: [UUID]
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Shared/tests/test_shared.py::test_x"
        -      ]
        -    }
        -  }
        -}
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -=== Without a root_path, test case IDs are absolute workspace paths
        ->>> [CLI] pipelines test --pipeline-id [UUID] /Workspace/Users/someone@example.com/orders/tests/test_orders.py
        -Pipeline: [UUID]
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
         
        -=== Outside a project, --pipeline-id is required
        ->>> musterr [CLI] pipelines test tests
        -Error: outside a project, identify the pipeline with --pipeline-id
        +Exit code: 1
        
    acceptance_test.go:1119: 
        LOG.import: Importing files from orders
        LOG.import: tests/test_orders.py -> /Workspace/Users/someone@example.com/orders/tests/test_orders.py
        LOG.import: Import complete

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/results: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/results/output.txt
        +++ /tmp/TestAcceptpipelinestestresults4111308121/002/output.txt
        @@ -8,139 +8,55 @@
         
         === All selected tests pass
         >>> [CLI] pipelines test my_pipeline tests/test_orders.py::test_valid_order
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown command "test" for "databricks pipelines"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        ->>> print_requests.py --get //events
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]' AND timestamp >= '[TIMESTAMP]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID]/events",
        -  "q": {
        -    "filter": "update_id='[UUID]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== Failed, errored, and skipped tests
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py
        -Recommendation: This command runs the last deployed version of the code
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        -tests/test_orders.py::test_backfill_fails  FAILED (1.00s)
        -tests/test_orders.py::test_legacy_format_skipped  SKIPPED (1.00s)
        -tests/test_orders.py::test_fixture_error  ERROR (1.00s)
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        -FAILURES
        -tests/test_orders.py::test_backfill_fails
        -  AssertionError: assert 100 == 150
        -tests/test_orders.py::test_fixture_error
        -  RuntimeError: fixture 'orders' setup failed
         
        -4 tests: 1 passed, 1 failed, 1 skipped, 1 error
        -
        -=== No tests collected
        ->>> musterr [CLI] pipelines test my_pipeline tests/helpers.py
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -
        -No tests were collected.
        -
        -=== The update fails after all tests passed
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_delivery.py
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_delivery.py::test_breaks_update_after_summary  PASSED (1.00s)
        -
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        -Error: update [UUID] is FAILED: ResultDeliveryException: Test results could not be verified.
        -
        -=== The update fails with an error of its own after a test failed
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_late_failure.py
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_late_failure.py::test_backfill_fails  FAILED (1.00s)
        -tests/test_late_failure.py::test_breaks_update_after_summary  PASSED (1.00s)
        -
        -FAILURES
        -tests/test_late_failure.py::test_backfill_fails
        -  AssertionError: assert 100 == 150
        -
        -2 tests: 1 passed, 1 failed, 0 skipped, 0 errors
        -Error: update [UUID] is FAILED: ResultDeliveryException: Test results could not be verified.
        -
        -=== Compute terminates before all results are received
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_compute.py
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Waiting for compute...
        -Running tests...
        -tests/test_compute.py::test_small_order  PASSED (1.00s)
        -
        -RUN INCOMPLETE: the update failed before all test results were received.
        -Results received: 1 passed.
        -Error: update [UUID] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.
        +Exit code: 1

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/run-control: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/run-control/output.txt
        +++ /tmp/TestAcceptpipelinestestrun-control3330733693/002/output.txt
        @@ -8,65 +8,55 @@
         
         === --no-wait returns after the update starts
         >>> [CLI] pipelines test my_pipeline tests/test_orders.py --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -=== --resume reads the results of an existing update
        ->>> [CLI] pipelines test my_pipeline --resume [UUID]
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Running tests...
        -tests/test_orders.py::test_valid_order  PASSED (1.00s)
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -1 test: 1 passed, 0 failed, 0 skipped, 0 errors
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== --timeout stops waiting but not the update
        ->>> [CLI] pipelines test my_pipeline tests/test_slow.py --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID] --timeout 3s
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Running tests...
        -Error: stopped waiting for update [UUID] after 3s; the update keeps running, to get its results run: databricks pipelines test my_pipeline --resume [UUID]
         
        -=== The command to read the results keeps the profile
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID] --timeout 3s -p run-control
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -Running tests...
        -Error: stopped waiting for update [UUID] after 3s; the update keeps running, to get its results run: databricks pipelines test my_pipeline --profile run-control --resume [UUID]
        -
        -=== Another update cannot start while the test update is active
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: An active update '[UUID]' already exists for pipeline '[UUID]'.
        -
        -=== Stopping the pipeline cancels the test update
        ->>> [CLI] pipelines stop my_pipeline
        -Stopping my_pipeline...
        -my_pipeline has been stopped.
        -
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID]
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -
        -RUN INCOMPLETE: the update was canceled before all test results were received.
        -Results received: none.
        -Error: update [UUID] is CANCELED
        -
        -=== --resume rejects an update that is not a test update
        ->>> musterr [CLI] pipelines test my_pipeline --resume [UUID]
        -Error: update [UUID] is not a test update; pass the update ID that pipelines test printed
        +Exit code: 1
        
    acceptance_test.go:1131: Test produced unexpected files:
        submit.txt

latest (v1.23.0): same output as main (aed6702)

TestAccept/pipelines/test/selection: branch ✅ | main (aed6702) ❌ | latest (v1.23.0) ❌

main (aed6702):

acceptance_test.go:1341: Diff:
        --- pipelines/test/selection/output.txt
        +++ /tmp/TestAcceptpipelinestestselection26843006/002/output.txt
        @@ -8,118 +8,55 @@
         
         === Run all tests of the only pipeline
         >>> [CLI] pipelines test --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -=== Select tests by relative path and test ID, with pytest arguments
        ->>> [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Shared/tests/test_shared.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg=-m --pytest-arg=smoke,fast --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests",
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_backfill",
        -        "/Workspace/Shared/tests/test_shared.py"
        -      ],
        -      "pytest_args": [
        -        "-k",
        -        "orders and not slow",
        -        "-m",
        -        "smoke,fast"
        -      ]
        -    }
        -  }
        -}
        +Use "databricks pipelines [command] --help" for more information about a command.
         
        -=== --pytest-arg also takes its value as the next argument
        ->>> [CLI] pipelines test my_pipeline --pytest-arg -k --pytest-arg orders, returns --pytest-arg -m --pytest-arg smoke --no-wait
        -Recommendation: This command runs the last deployed version of the code
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "pytest_args": [
        -        "-k",
        -        "orders, returns",
        -        "-m",
        -        "smoke"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Relative paths resolve from the working directory
        ->>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID])
        -Update: [UUID]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_valid_order"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Test selections require the pipeline KEY first
        ->>> musterr [CLI] pipelines test tests/test_orders.py
        -Error: resource with key "tests/test_orders.py" not found; the first argument must be a pipeline KEY when test selections are given
        -
        -=== Test selections must be in the synchronized files
        ->>> musterr [CLI] pipelines test my_pipeline ../test_outside.py
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot resolve test selection "../test_outside.py": [TEST_TMP_DIR]_PARENT/test_outside.py is outside the synchronized bundle files in [TEST_TMP_DIR]
        -
        -=== Relative test selections must exist locally
        ->>> musterr [CLI] pipelines test my_pipeline tests/test_ordrs.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot resolve test selection "tests/test_ordrs.py::test_backfill": [TEST_TMP_DIR]/tests/test_ordrs.py does not exist; pass a local path or an absolute workspace path
        +Exit code: 1

latest (v1.23.0): same output as main (aed6702)

[eng-dev-ecosystem-bot]

Integration test report

Commit: 8542365

Run: 41513935315

	Env	🔄​flaky	✅​pass	🙈​skip	Time
🔄​	aws linux-2core-8gb	2	279	23	6:14
✅​	aws-windows-latest-4core-16gb		283	21	7:09
✅​	azure linux-2core-8gb		280	23	5:58
🔄​	azure-windows-latest-4core-16gb	2	280	21	7:25
✅​	gcp linux-2core-8gb		281	23	6:19
✅​	gcp-windows-latest-4core-16gb		283	21	6:59

	Test Name	aws linux-2core-8gb	azure-windows-latest-4core-16gb
🔄​	TestFsCpFileToFileFileNotOverwritten	✅​p	🔄​f
🔄​	TestFsCpFileToFileFileNotOverwritten/uc-volumes_to_uc-volumes	✅​p	🔄​f
🔄​	TestFilerReadWrite	🔄​f	✅​p
🔄​	TestFilerReadWrite/files	🔄​f	✅​p

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
7:03	aws-windows-latest-4core-16gb	TestAccept
7:02	azure-windows-latest-4core-16gb	TestAccept
6:53	gcp-windows-latest-4core-16gb	TestAccept
3:14	gcp linux-2core-8gb	TestAccept
3:13	aws linux-2core-8gb	TestAccept
3:06	azure linux-2core-8gb	TestAccept