feat: [FME-17261][FME-19301]: add experiment noun (CRUD) and experiment settings/alerts/results - #258
Open
apetruccelli wants to merge 4 commits into
Open
apetruccelli wants to merge 4 commits into
apetruccelli wants to merge 4 commits into
Conversation
apetruccelli
force-pushed
the
FME-17261-experiment-v2
branch
from
September 30, 2026 16:54
2553802 to
39c4940
Compare
apetruccelli
added a commit
to apetruccelli/cli
that referenced
this pull request
Sep 30, 2026
…experiment_results Adds get/update/delete experiment:settings, get/update experiment:alerts, and list experiment:results commands, stacked on the experiment noun introduced in harness#258. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- create/list/get/update/delete experiment against v4 (/fme/api/v4/experiments),
scoped by parent-type (FEATURE_FLAG or AI_CONFIG) and parent-name/parent-id
- fme:metric_refs field type (modules/fme/metric_refs.go): keyMetrics/supportingMetrics
read as [{id,name}] but write as plain metric id strings, which core's built-in
"set" type can't handle since it assumes read/write shapes match
- owners mutable via --add/--del owners.user:<email|id> and owners.group:<identifier>,
reusing the fme:owners field type already wired onto feature_flag/segment
- update narrows the get-then-patch body to UpdateExperimentRequest's accepted fields,
keeping read-only fields (id/type/parent/environment/timestamps) out of the PATCH
- unit tests for all 5 operations plus owners add/del, in pkg/specloader/fme_spec_test.go
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…gs_builtin BuiltinFlags has no Add field — set: true already enables --add/--del together. The extra add: true broke YAML unmarshaling in CI's check:specs task.
…te mutations) Adds the tags field to the experiment noun (fme:tags field type), wires tags into update experiment's PATCH allowlist, and adds a --tag filter to list experiment. Verified end-to-end against qa0 (add/get/list/del). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…experiment_results Adds get/update/delete experiment:settings, get/update experiment:alerts, and list experiment:results commands, stacked on the experiment noun introduced in harness#258. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apetruccelli
force-pushed
the
FME-17261-experiment-v2
branch
from
October 2, 2026 12:26
ccf5d8f to
35d9293
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
experimentnoun with full CRUD) and FME-19301 (experiment sub-resources: settings, alerts, results).experimentnoun, full CRUD:list,get,create,update,deleteagainst v4 (/fme/api/v4/experiments).--parent-type FEATURE_FLAG|AI_CONFIGplus--parent-name/--parent-id— and by--env.createbuilds the request from--start-at/--end-at/--baseline-treatment/--comparison-treatment/--key-metric/--supporting-metric/--description/--hypothesis, or accepts-f experiment.jsonas a full alternative.updatenarrows the get-then-patch body toUpdateExperimentRequest's accepted fields, keeping read-only fields (id,type,parent,environment, timestamps) out of the PATCH — v4 rejects unknown properties.fme:metric_refsfield type (modules/fme/metric_refs.go):keyMetrics/supportingMetricsread as[{id,name}]but write as plain metric id strings — an asymmetric read/write shape core's built-insettype can't handle since it assumes both shapes match.Normalizestrips the display name down to a plain id string;Mutateadds/deletes/dedups on the normalized id list.ownersis fully mutable, reusing thefme:ownersfield type already wired ontofeature_flag/segmentin feat: [FME-17257]: add owners/tags field types, flag_sets, and file (-f) support for feature_flag and segment #257 — no new Go code needed.--add/--del owners.user:<email|id>andowners.group:<identifier>work at bothcreateandupdate.tagssupport (modules/fme/tags.go, newfme:tagsfield type): read shape is[{id,name}], write shape is[{name}]only.--add/--del tags.<name>work atupdate;list experiment --tag <name>filters by tag name.experiment:settings: get/update/delete the experiment's statistical settings (test type, significance threshold, MCC, minimum sample size, review period, variance reduction). Update is a get-then-patch againstPATCH /fme/api/v4/experiments/{id}/settings;id/sourceare excluded from--setsince they're server-derived and echoing them back 400s.experiment:alerts: get/update the experiment's alerting toggle (isEnabled). Update is a get-then-patch againstPATCH /fme/api/v4/experiments/{id}/alerting.experiment:results: list metric results for an experiment (GET /fme/api/v4/experiments/{id}/metric-results), with an optional--metric-idfilter mapped to themetric_idsquery param. Null numeric/string fields render as empty rather than<nil>.Commands
harness create experiment <name> --parent-type feature_flag --parent-name my-flag --env <env-id> --start-at 2026-01-01T00:00:00Z --end-at 2026-02-01T00:00:00Z --baseline-treatment off --comparison-treatment on --add owners.user:<email>harness create experiment <name> -f experiment.jsonharness list experiment --parent-type FEATURE_FLAG --parent-name my-flag [--env <env-id>] [--status ACTIVE] [--tag <name>]harness get experiment <id>harness update experiment <id> --set description="new desc" --add key_metrics.<metric-id> --add owners.user:<email> --del owners.user:<id> --add tags.<name> --del tags.<name>harness delete experiment <id> --confirmfme get experiment:settings --id <experiment-id>fme update experiment:settings --id <experiment-id> --set <field>=<value>fme delete experiment:settings --id <experiment-id>fme get experiment:alerts --id <experiment-id>fme update experiment:alerts --id <experiment-id> --set is_enabled=<true|false>fme list experiment:results --id <experiment-id> [--metric-id <metric-id>]Testing
go build ./...andgo test ./pkg/... ./modules/...pass on the branch rebased onto latestmain.pkg/specloader/fme_spec_test.gocover all 5 experiment operations, owners add/del, the tag filter and tag mutations, and all six sub-resource commands (get/update/delete settings, get/update alerts, list results with and without--metric-id).list experiment --tag.--org/--projectscope flags and--format json|tablework forlist/get experiment.Acceptance Criteria
experimentoperations implemented (list,get,create,update,delete)--org/--projectscope flags work--format json|tableoutput works forlist/getget/update/delete experiment:settingsimplemented against the v4 settings endpointget/update experiment:alertsimplemented against the v4 alerting endpointlist experiment:resultsimplemented against the v4 metric-results endpoint, with--metric-idfilter<nil>🤖 Generated with Claude Code