Skip to content

feat: [FME-17261][FME-19301]: add experiment noun (CRUD) and experiment settings/alerts/results - #258

Open
apetruccelli wants to merge 4 commits into
harness:mainfrom
apetruccelli:FME-17261-experiment-v2
Open

apetruccelli wants to merge 4 commits into
harness:mainfrom
apetruccelli:FME-17261-experiment-v2

Conversation

@apetruccelli

@apetruccelli apetruccelli commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • Tickets FME-17261 (experiment noun with full CRUD) and FME-19301 (experiment sub-resources: settings, alerts, results).
  • experiment noun, full CRUD: list, get, create, update, delete against v4 (/fme/api/v4/experiments).
    • Experiments are scoped by parent — --parent-type FEATURE_FLAG|AI_CONFIG plus --parent-name/--parent-id — and by --env.
    • create builds the request from --start-at/--end-at/--baseline-treatment/--comparison-treatment/--key-metric/--supporting-metric/--description/--hypothesis, or accepts -f experiment.json as a full alternative.
    • update narrows the get-then-patch body to UpdateExperimentRequest's accepted fields, keeping read-only fields (id, type, parent, environment, timestamps) out of the PATCH — v4 rejects unknown properties.
  • fme:metric_refs field type (modules/fme/metric_refs.go): keyMetrics/supportingMetrics read as [{id,name}] but write as plain metric id strings — an asymmetric read/write shape core's built-in set type can't handle since it assumes both shapes match. Normalize strips the display name down to a plain id string; Mutate adds/deletes/dedups on the normalized id list.
  • owners is fully mutable, reusing the fme:owners field type already wired onto feature_flag/segment in feat: [FME-17257]: add owners/tags field types, flag_sets, and file (-f) support for feature_flag and segment #257 — no new Go code needed. --add/--del owners.user:<email|id> and owners.group:<identifier> work at both create and update.
  • tags support (modules/fme/tags.go, new fme:tags field type): read shape is [{id,name}], write shape is [{name}] only. --add/--del tags.<name> work at update; list experiment --tag <name> filters by tag name.
  • experiment:settings: get/update/delete the experiment's statistical settings (test type, significance threshold, MCC, minimum sample size, review period, variance reduction). Update is a get-then-patch against PATCH /fme/api/v4/experiments/{id}/settings; id/source are excluded from --set since they're server-derived and echoing them back 400s.
  • experiment:alerts: get/update the experiment's alerting toggle (isEnabled). Update is a get-then-patch against PATCH /fme/api/v4/experiments/{id}/alerting.
  • experiment:results: list metric results for an experiment (GET /fme/api/v4/experiments/{id}/metric-results), with an optional --metric-id filter mapped to the metric_ids query param. Null numeric/string fields render as empty rather than <nil>.

Commands

  • harness create experiment <name> --parent-type feature_flag --parent-name my-flag --env <env-id> --start-at 2026-01-01T00:00:00Z --end-at 2026-02-01T00:00:00Z --baseline-treatment off --comparison-treatment on --add owners.user:<email>
  • harness create experiment <name> -f experiment.json
  • harness list experiment --parent-type FEATURE_FLAG --parent-name my-flag [--env <env-id>] [--status ACTIVE] [--tag <name>]
  • harness get experiment <id>
  • harness update experiment <id> --set description="new desc" --add key_metrics.<metric-id> --add owners.user:<email> --del owners.user:<id> --add tags.<name> --del tags.<name>
  • harness delete experiment <id> --confirm
  • fme get experiment:settings --id <experiment-id>
  • fme update experiment:settings --id <experiment-id> --set <field>=<value>
  • fme delete experiment:settings --id <experiment-id>
  • fme get experiment:alerts --id <experiment-id>
  • fme update experiment:alerts --id <experiment-id> --set is_enabled=<true|false>
  • fme list experiment:results --id <experiment-id> [--metric-id <metric-id>]

Testing

  • go build ./... and go test ./pkg/... ./modules/... pass on the branch rebased onto latest main.
  • Unit tests in pkg/specloader/fme_spec_test.go cover all 5 experiment operations, owners add/del, the tag filter and tag mutations, and all six sub-resource commands (get/update/delete settings, get/update alerts, list results with and without --metric-id).
  • Live-verified against qa0: create/list/get/update/delete round trip, owners add/del, key-metric add on update, read-only field exclusion from PATCH, tags add/del and list experiment --tag.
  • Live-verified that --org/--project scope flags and --format json|table work for list/get experiment.

Acceptance Criteria

  • All 5 experiment operations implemented (list, get, create, update, delete)
  • --org/--project scope flags work
  • --format json|table output works for list/get
  • get/update/delete experiment:settings implemented against the v4 settings endpoint
  • get/update experiment:alerts implemented against the v4 alerting endpoint
  • list experiment:results implemented against the v4 metric-results endpoint, with --metric-id filter
  • Null numeric/string result fields render as empty, not <nil>
  • Unit tests for each operation

🤖 Generated with Claude Code

@apetruccelli
apetruccelli force-pushed the FME-17261-experiment-v2 branch from 2553802 to 39c4940 Compare September 30, 2026 16:54
apetruccelli added a commit to apetruccelli/cli that referenced this pull request Sep 30, 2026
…experiment_results

Adds get/update/delete experiment:settings, get/update experiment:alerts,
and list experiment:results commands, stacked on the experiment noun
introduced in harness#258.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apetruccelli and others added 4 commits October 2, 2026 09:11
- create/list/get/update/delete experiment against v4 (/fme/api/v4/experiments),
  scoped by parent-type (FEATURE_FLAG or AI_CONFIG) and parent-name/parent-id
- fme:metric_refs field type (modules/fme/metric_refs.go): keyMetrics/supportingMetrics
  read as [{id,name}] but write as plain metric id strings, which core's built-in
  "set" type can't handle since it assumes read/write shapes match
- owners mutable via --add/--del owners.user:<email|id> and owners.group:<identifier>,
  reusing the fme:owners field type already wired onto feature_flag/segment
- update narrows the get-then-patch body to UpdateExperimentRequest's accepted fields,
  keeping read-only fields (id/type/parent/environment/timestamps) out of the PATCH
- unit tests for all 5 operations plus owners add/del, in pkg/specloader/fme_spec_test.go

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…gs_builtin

BuiltinFlags has no Add field — set: true already enables --add/--del together.
The extra add: true broke YAML unmarshaling in CI's check:specs task.
…te mutations)

Adds the tags field to the experiment noun (fme:tags field type), wires
tags into update experiment's PATCH allowlist, and adds a --tag filter
to list experiment. Verified end-to-end against qa0 (add/get/list/del).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…experiment_results

Adds get/update/delete experiment:settings, get/update experiment:alerts,
and list experiment:results commands, stacked on the experiment noun
introduced in harness#258.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@apetruccelli
apetruccelli force-pushed the FME-17261-experiment-v2 branch from ccf5d8f to 35d9293 Compare October 2, 2026 12:26
@apetruccelli apetruccelli changed the title feat: [FME-17261]: add experiment noun (CRUD) feat: [FME-17261][FME-19301]: add experiment noun (CRUD) and experiment settings/alerts/results Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant