Skip to content

refactor: declare each adapter's Service and pass get_ogc_data one Endpoint (ADR 0014, step 3) - #450

Closed
thodson-usgs wants to merge 5 commits into
DOI-USGS:mainfrom
thodson-usgs:refactor/service-endpoint-v2
Closed

thodson-usgs wants to merge 5 commits into
DOI-USGS:mainfrom
thodson-usgs:refactor/service-endpoint-v2

Conversation

@thodson-usgs

Copy link
Copy Markdown
Collaborator

Step 3 of ADR 0014 (proposed), the last step. Stacked on #448 and #449: review only the last commit, 90d5b5c6. I'll rebase as the others merge.

Problem

  • Service names written as literals: each adapter wrote its service's name, such as adapter="waterdata", at every configuration, retry and fan-out call: 18 sites in 10 files, plus its Configuration class.
  • URL resolution repeated: each adapter resolved its URL with its own base_url(adapter=..., default=...) call.
  • Arguments passed separately: get_ogc_data took the URL, the dialect and the adapter name as three arguments that always travel together.

Change

  • dataretrieval._service is a dependency-free leaf with two values:

    • Service: the name settings are resolved by, and the default URL;
    • Endpoint: a service, the URL resolved when a call starts, and a dialect.
  • Each adapter declares its Service once and reads the name from it in:

    • its Configuration class;
    • its retry and fan-out adapter= arguments;
    • configuration.service_url(service), which replaces base_url(adapter=..., default=...).

    Water Data's SERVICE is in waterdata.endpoints, shared by its four APIs.

  • get_ogc_data takes endpoint= in place of base_url=, dialect= and adapter=. The request builder and finalizer it binds keep their URL and dialect arguments; ADR 0014: declare how each service departs from its protocol as data #448 changed the ADR's wording to match.

  • Tests:

    • New test_service_names_are_written_only_where_declared fails if a service name appears as a literal outside its declaration and the configuration roster.
    • The read-site test in configuration_test.py now looks for <declaration>.name or Endpoint(<declaration>, ...).
  • CONTEXT.md defines Service declaration and Endpoint, and broadens Dialect to cover both dialect types.

Not done, by design

  • Endpoint for the STAC ratings search: get_ratings uses STAC_DATES directly.
  • Endpoint for WQP, NLDI, NWDC, StreamStats, Samples and Statistics: no shared protocol code reads a dialect for them. They get a Service only.

Behavior

No change.

  • Offline: the suite (1336 tests) passes under UTC and America/Chicago, coverage is 99.10%, and the pre-commit gates pass.
  • Live, all returned data as expected:
    • Water Data get_daily, get_continuous with an interval string, and get_ratings;
    • get_reference_table with a date in query;
    • NGWMN get_water_level with an open start;
    • WQP get_results;
    • an NgwmnConfiguration(base_url=...) redirect.

Once this merges, ADR 0014 can be marked Accepted.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK

thodson-usgs and others added 5 commits October 12, 2026 15:47
… (ADR 0014)

Step 1 of ADR 0014 (proposed), which replaces the single-branch draft in
DOI-USGS#445.

- dataretrieval._dates (a dependency-free leaf) reads a date argument
  into a DateRange: an instant, a range with None for an open bound, or
  a duration. Every bound is UTC; a naive value is UTC, as in pandas.
- dataretrieval._intervals writes a DateRange in the ISO 8601 interval
  grammar that OGC API Features and STAC share, adjusted by a
  DateDialect(open_start, durations). It sits below ogc because STAC is
  not OGC.
- OgcDialect.open_start becomes OgcDialect.dates. Water Data sets
  durations=True; NGWMN sets open_start=""; get_ratings uses STAC_DATES
  (open_start="").
- ogc/dates.py and get_ratings' separate duration check are removed.

Behavior change: a lone duration passed to an NGWMN getter raises
ValueError before sending; the service rejects one with HTTP 400.
get_ratings' duration message is now the shared one. Otherwise the
values sent are unchanged: a comparison of 1432 inputs (each element
type, every pair, interval strings, error cases) against main's
_format_api_dates found no difference in output or message, under UTC
and America/Chicago.

Live tests check each DateDialect field against the Water Data OGC,
NGWMN, and STAC services.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK
…n entry (ADR 0014)

Step 2 of ADR 0014 (proposed), stacked on step 1 (DOI-USGS#448).

- OgcDialect gains date_params (default {"datetime"}, the standard's one
  date parameter) and timed_params, and date_only_services is renamed
  date_only_collections, since it is keyed by collection. The Water Data
  dialect lists its seven date parameters and timed_params
  {"last_modified"}, which the request builder used to name.
- prepare_request_args(..., dialect=) reads the dialect's date parameters
  into a DateRange when the getter is called and drops an all-open one.
  Water Data's _get_args passes WATERDATA_DIALECT (the Samples and
  Statistics getters pass None, as those APIs are not OGC); NGWMN passes
  NGWMN_DIALECT.
- get_ogc_data reads any date parameter still raw (a verbatim query from
  get_reference_table), and raises TypeError for a DateRange under a key
  the dialect does not list.
- The chunk planner's _NEVER_CHUNK lists no date parameter: a DateRange
  is not a list. _DATE_RANGE_PARAMS is removed.
- tests/architecture_test.py fails if a string a shipped dialect lists
  appears as a literal in ogc/ or _intervals.py. It found one: the
  "value" default of _normalize_str_iterable's param_name, now required.

The values sent are unchanged. An unreadable date now raises when the
getter is called rather than when the first request is built; the
message is the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK
…dpoint (ADR 0014)

Step 3 of ADR 0014 (proposed), stacked on steps 1 (DOI-USGS#448) and 2 (DOI-USGS#449).

- dataretrieval._service (a dependency-free leaf) defines Service (the
  adapter's name, which settings are resolved by, and its default URL)
  and Endpoint (a service, the URL resolved when a call starts, and a
  dialect).
- Each adapter declares its Service once and reads the name from it:
  the Configuration class's adapter attribute, retry and fan-out
  adapter= arguments, and configuration.service_url(service), which
  replaces base_url(adapter=..., default=...) at each URL read site.
  Water Data's SERVICE is in waterdata.endpoints, shared by its four
  APIs.
- get_ogc_data takes endpoint= in place of base_url=, dialect=, and
  adapter=. The request builder and finalizer it binds keep their
  URL and dialect arguments.
- tests/architecture_test.py fails if a service name appears as a
  literal outside its declaration and the configuration roster; the
  read-site test in configuration_test.py looks for <declaration>.name
  or Endpoint(<declaration>, ...).
- CONTEXT.md defines Service declaration and Endpoint, and generalizes
  Dialect.

No behavior change. Checked live: Water Data daily, continuous, ratings,
and a reference table with a date in its query; NGWMN water levels with
an open start; WQP results; and an NgwmnConfiguration base_url redirect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK
@thodson-usgs

Copy link
Copy Markdown
Collaborator Author

Combined into #448, which now holds all three ADR 0014 steps as one commit each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant