Tutorial on building an accesor and site archive by stevehadd · Pull Request #275 · ACCESS-Community-Hub/PyEarthTools

stevehadd · 2026-09-23T22:55:47Z

This PR includes some tutorial notebooks and a site archive for AWS data, which can be run from anywhere.
Changes:

Site Archive tutorial notebook
Acessor tutorial notebook
Inclusion of new notebooks in gallery
AWS site archive package in a new "third party" directory
- Idea is we can include packages for specific other data sources e.g. CDS and also other integrations like mlflow or nvidia earth2 studio in future
tests for site archive

Getting Started

[ X] If there is not an existing issue, raise a new issue
In the issue, state that you are intending to work on a contribution. This gives everyone involved the opportunity to discuss the best way forward.

Docstrings

[X ] Docstrings complete and follow Napoleon (google) style

Test Coverage

[X ] All new code is covered by unit tests

Documentation

[X ] Documentation is updated as required

Final Checks:

[ X] If no generative AI was used, then tick this box

Alternatively, if generative AI was used, then confirm you have:

Attributed any generative AI (such as GitHub Copilot) that was used in this PR. See our contributing guide for more information.
Included the name and version of the tool or system in the pull request
Described the scope of that use

Finally,

Mark the PR as ready to review. Note - we encourage you to ask for feedback at the outset or at any time during the work.

Copilot

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 · 2 · 3

Open (10)

Add missing package LICENSE file · New
Fix invalid variable keyword in MOGlobal10km.sample · New
Fix invalid variable keyword in MOUKV.sample · New
Install site-archive package for test collection · New
Install site-archive package for test collection · New
Align Python requirement with dependency minimum · New
Honor configured root directories in accessors · New
Fix datetime import and retrieval in usage example · New
Rename description hook to desc convention · New
Rename description hook to desc convention · New

What changed in this PR

Adds AWS-backed Met Office UKV and global 10 km accessors, tutorials, packaging metadata, and tests.

Changes:

Added AWS site archive accessors and registration.
Added tutorials and gallery entries.
Added requirements, packaging configuration, and integration tests.

File	Description
packages/third_party/aws_site_archive/tests/test_ukv.py	Updated as part of this pull request.
packages/third_party/aws_site_archive/tests/test_moglobal.py	Updated as part of this pull request.
packages/third_party/aws_site_archive/tests/report.xml	Updated as part of this pull request.
packages/third_party/aws_site_archive/src/site_archive_aws/mo_ukv.py	Updated as part of this pull request.
packages/third_party/aws_site_archive/src/site_archive_aws/mo_global_10km.py	Updated as part of this pull request.
packages/third_party/aws_site_archive/src/site_archive_aws/__init__.py	Updated as part of this pull request.
packages/third_party/aws_site_archive/requirements.txt	Updated as part of this pull request.
packages/third_party/aws_site_archive/requirements-dev.txt	Updated as part of this pull request.
packages/third_party/aws_site_archive/README.md	Updated as part of this pull request.
packages/third_party/aws_site_archive/pyproject.toml	Updated as part of this pull request.
packages/third_party/aws_site_archive/MANIFEST.in	Updated as part of this pull request.
docs/notebooks/Gallery.ipynb	Updated as part of this pull request.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🟡 Changes recommended

Critical accessor behavior and test assertions remain unresolved, along with configuration inconsistencies.

Review effort: Lite
Findings: 6 · 1 · 3

Open (10)

Normalize scalar variables before filesystem iteration · New
Normalize scalar variables before filesystem iteration · New
Assert global accessor dataset dimensions · New
Install site-archive package for test collection
Install site-archive package for test collection
Add missing package LICENSE file
Honor configured root directories in accessors
Rename description hook to desc convention
Rename description hook to desc convention
Fix datetime import and retrieval in usage example

Resolved since last review (3)

Fix invalid variable keyword in MOUKV.sample
Fix invalid variable keyword in MOGlobal10km.sample
Align Python requirement with dependency minimum

Previously missed (2)

In code that hasn't changed since last review

Fix cache recommendation spelling

packages/third_party/aws_site_archive/README.md:5

The new documentation says it is “recommnded” to use a cache; correct the spelling in this user-facing guidance.

Correct global archive documentation URL

packages/third_party/aws_site_archive/src/site_archive_aws/mo_global_10km.py:103

The global accessor's documentation URL points to the UK deterministic archive, so users following the metadata are sent to the wrong dataset documentation. Use the global deterministic archive URL instead.

Copilot

Copilot review overview

🟡 Changes recommended

Moderate implementation, dependency, registration, and test reliability issues remain unresolved.

Review effort: Lite
Findings: 3 · 1 · 1

Open (5)

Assert global accessor dataset dimensions
Normalize scalar variables before filesystem iteration
Normalize scalar variables before filesystem iteration
Honor configured root directories in accessors
Link global accessor to global deterministic documentation · New

Resolved since last review (6)

Install site-archive package for test collection
Install site-archive package for test collection
Add missing package LICENSE file
Rename description hook to desc convention
Rename description hook to desc convention
Fix datetime import and retrieval in usage example

Previously missed (7)

In code that hasn't changed since last review

Load configured archive roots during package initialization

packages/third_party/aws_site_archive/src/site_archive_aws/__init__.py:33

The README says that ~/.pyearthtoolsconfig can override these defaults, but this module never calls the imported load_root_directories_from_config (unlike site_archive_met_office). As a result, configured roots are ignored whenever this package is imported; load the config after registering the defaults.

Normalize string variables to a one-element list

packages/third_party/aws_site_archive/src/site_archive_aws/mo_global_10km.py:46

variables is documented and registered as accepting either a string or a list, and the registered .sample() path passes a string. Keeping the string here makes filesystem() iterate over its characters, so MOGlobal10km.sample() generates one invalid S3 path per character instead of one variable path. Normalize a string to a one-element list before storing it.

Reject empty variable lists in MOGlobal accessor

packages/third_party/aws_site_archive/src/site_archive_aws/mo_global_10km.py:74

Unlike MOUKV, this accessor does not reject an empty variable list. The committed test report shows test_moglobal_novar as an XPASS, so the xfail is not exercising the documented failure behavior and an empty request can proceed to an empty merge. Add the same early DataNotFoundError guard used by UKV.

Normalize string variables to a one-element list

packages/third_party/aws_site_archive/src/site_archive_aws/mo_ukv.py:46

variables is documented and registered as accepting either a string or a list, and the registered .sample() path passes a string. Keeping the string here makes filesystem() iterate over its characters, so MOUKV.sample() generates one invalid S3 path per character instead of one variable path. Normalize a string to a one-element list before storing it.

Correct global archive dimensions in slow-load test

packages/third_party/aws_site_archive/tests/test_moglobal.py:63

The global archive output is not a UKV-sized grid: the site-archive notebook shows global data with dimensions (pressure=37, latitude=1920, longitude=2560), so running this marked-slow test will fail against the implementation's actual dataset shape (1, 37, 1920, 2560). Update the assertion to the global grid dimensions.

Notebook Python version is unsupported by project bounds

docs/notebooks/Gallery.ipynb:219

This notebook metadata now records Python 3.14.6, while the repository and the new AWS package both declare requires-python = ">=3.11, <3.14". The gallery therefore claims an execution environment that cannot install the documented project; keep the metadata on a supported Python version or update the package bounds after validating 3.14 support.

Fix misspelling in README installation guidance

packages/third_party/aws_site_archive/README.md:5

The new README contains the misspelling recommnded; correct it so the installation guidance is rendered professionally.

This issue also appears on line 23 of the same file.

tennlee · 2026-10-29T23:24:09Z

            if issubclass(obj_reference, pyearthtools.data.indexes.Index):
-                project_code = getattr(obj_reference, 'project_code', None)
-                print(f'{obj_reference} is a registered Index in the archive as {__name__}.{obj_name}')
+                project_code = getattr(obj_reference, "project_code", None)


This is fine but generally it's better to just not touch files which aren't in the scope of the change. Don't back this one out though because it's so simple, but this is just a general request not to touch (even for quote replacements) which aren't in scope of the PR.

tennlee · 2026-10-29T23:47:08Z

+    MO_GLOBAL_AWS_DIR_TEMPLATE = "{root_dir}/{vt_str}"
+    MO_GLOBAL_FNAME_TEMPLATE = "{vt_str}-PT0000H00M-{var_name}.nc"
+
+    def __init__(


If you look at e.g. pyearthtools.data.download.weatherbench.py, the constructor takes license_url and license_ok variables so that there is runtime alerting of the data license conditions unless the user passes an argument. I think that's less relevant for local data sets where the user has already downloaded the data, but for online / streaming access I think we should alert the user to the existence of data licenses and recommend they read the Ts and Cs.

tennlee · 2026-10-29T23:47:31Z

+    def __init__(
+        self,
+        variables: list[str],
+        *,


See previous comment on license terms alert

stevehadd and others added 15 commits September 7, 2026 17:12

first draft of a building an accessor tutorial.

60075c8

start of accessor tutorials.

75c75e1

start of accessor tutorials.

f490dac

further info on tutorials.

bc14ee5

start for a remote accessor for CMIP data.

66e576d

added intial aws site archive.

a9d3d98

now working.

f1e7471

accessor notebook complete, site archive in development.

6b068f1

working versions of building accessor and site archvie tutorials.

cdbe66a

preparing for code review.

f016f4f

Adding additional text, testing notebooks and updating the gallery.

ce61556

adding some tests

762f163

ran black formatter.

e1c10ef

removing old files.

e0ba891

removing old files.

c86c506

stevehadd requested review from tennlee and a lite review from Copilot September 23, 2026 22:55

Copilot started reviewing on behalf of stevehadd September 23, 2026 22:56

View session

Copilot AI reviewed Sep 23, 2026

View reviewed changes

stevehadd added 2 commits October 1, 2026 13:45

added a license file.

60ca7ce

changes from copilot review.

659e7fd

stevehadd requested a lite review from Copilot October 1, 2026 12:53

Copilot started reviewing on behalf of stevehadd October 1, 2026 12:54

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread packages/third_party/aws_site_archive/src/site_archive_aws/mo_global_10km.py Resolved

Comment thread packages/third_party/aws_site_archive/src/site_archive_aws/mo_ukv.py Resolved

Comment thread packages/third_party/aws_site_archive/tests/test_moglobal.py Resolved

stevehadd added 2 commits October 1, 2026 14:48

removed reference to a license file.

a442706

further fixes for testing.

bf8a8e3

stevehadd requested a lite review from Copilot October 1, 2026 15:41

Copilot started reviewing on behalf of stevehadd October 1, 2026 15:42

View session

stevehadd added 2 commits October 1, 2026 16:44

fixing formatting for black.

c72bdbb

more formatting fixes

b0f8294

fixed readme from copilot review suggestion

802ccda

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread packages/third_party/aws_site_archive/src/site_archive_aws/mo_global_10km.py Resolved

stevehadd added 2 commits October 2, 2026 15:38

further fixed from copilot review

3463369

correcting docs URL.

d19d5b8

tennlee reviewed Oct 29, 2026

View reviewed changes

Conversation

stevehadd commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Getting Started

Docstrings

Test Coverage

Documentation

Final Checks:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

tennlee Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

tennlee Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

tennlee Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stevehadd commented Sep 23, 2026 •

edited

Loading