Operator integration tests still depend on removed SeaweedFS NodePorts #281

Summary

The OpenServerless operator integration suite still assumes that the SeaweedFS control API is exposed through a fixed NodePort. This assumption became invalid after openserverless-operator PR #155 correctly changed the SeaweedFS Service from NodePort to ClusterIP and restricted its control ports.

As a result, several integration scripts fail with Connection reset by peer. The two Milvus scripts are reported as Milvus failures even though execution stops in seaweedfs.create() before milvus.create() is reached.

This issue proposes updating the integration access path while retaining the private ClusterIP Service and the NetworkPolicy. It does not propose restoring a NodePort.

Affected component and branch

• Component: apache/openserverless-operator
• Branch: 0.9.0
• Confirmed affected integration scripts:
  • tests/kind/milvus_standalone_test.ipy
  • tests/kind/milvus_standalone_slim_test.ipy
  • tests/kind/seaweedfs_test.ipy
  • tests/kind/seaweedfs_util_test.ipy
  • tests/kind/storage_static_test.ipy

Root cause

The Kind configuration maps host port 9090 to node container port 32091, which was the former SeaweedFS filer NodePort.

The Milvus integration scripts still do the following before creating Milvus:

os.environ['SEAWEEDFS_API_HOST'] = 'localhost'
os.environ['SEAWEEDFS_API_PORT'] = '9090'

assert(etcd.create())
assert(seaweedfs.create())

PR #155 removed NodePort 32091 and changed the Service to ClusterIP. Therefore, localhost:9090 no longer reaches the SeaweedFS filer API.

seaweedfs.create() retries the readiness request for approximately 30 seconds and logs repeated connection resets. The shared wait_for_http() helper then returns without raising an exception when its deadline expires, despite its docstring declaring TimeoutError. Execution continues until make_public_bucket() performs another request and raises ConnectionError.

The resulting traceback points to the Milvus integration file, but the failing call is seaweedfs.create() and Milvus has not yet been created.

Evidence

• The final CI run for operator PR #141, with Milvus 2.5.2 and the private Milvus tunnel changes, passed the full and slim Milvus scenarios: run 39257637756.
• Operator PR #155 failed both Milvus integration scripts while still using Milvus 2.5.2: run 40344816581.
• Operator PR #154 reports the same failure set after updating Milvus to 2.5.27: run 40479000340.
• The Milvus full and slim integration scripts are unchanged between the successful reduce runtimes and set slim as the defalt for devcluster #141 run and the [fabio ziviello] Deploy a local registry #155 failure.

This shows that the failure is not caused by the Milvus 2.5.27 image update.

Proposed solution

1. Keep the private service boundary

Retain:

• the SeaweedFS ClusterIP Service;
• the removal of filer/master NodePorts;
• the current NetworkPolicy restrictions.

The tests must adapt to the private endpoint rather than reopening it.

2. Use a reusable private endpoint for out-of-cluster execution

Use the existing loopback-only service_endpoint() mechanism for the SeaweedFS filer control port after the Service has been applied and before readiness/bootstrap operations run.

The cleanest implementation would separate SeaweedFS creation into two internal phases:

1. render and apply the StatefulSet, Service and NetworkPolicy, then wait for the pod;
2. obtain the control endpoint and perform readiness checks and bootstrap operations.

For phase 2:

• in-cluster execution should continue to use seaweedfs:9090;
• out-of-cluster integration execution should use an RBAC-authorized kubectl port-forward bound only to 127.0.0.1 and an ephemeral local port;
• endpoint overrides should be scoped to the operation and restored afterwards.

An alternative test-only post-apply hook, similar to the Milvus integration hook, could be considered if maintainers prefer not to refactor the creation flow. The important property is that the tunnel must be established only after the Service exists and before any external SeaweedFS HTTP request.

3. Remove fixed NodePort assumptions from the tests

Update the affected integration scripts to stop relying on:

• fixed host port 9090;
• fixed node port 32091;
• direct access to a private control API.

The tests should consume the endpoint returned by the shared private connection helper.

4. Make readiness timeout behavior explicit

Update wait_for_http() to raise TimeoutError after the deadline, as documented. This will make the first inaccessible endpoint fail at the readiness step instead of producing a later and misleading bootstrap traceback.

Callers that intentionally tolerate an unavailable endpoint should handle the timeout explicitly.

Acceptance criteria

• SeaweedFS filer and master ports remain unavailable through NodePort.
• The full and slim Milvus integration scripts pass without fixed host or node ports.
• The direct SeaweedFS and static-storage integration scripts pass through the same private endpoint mechanism.
• The local tunnel binds only to 127.0.0.1, uses an ephemeral port and is always terminated during cleanup.
• In-cluster operator execution continues to use the Kubernetes Service directly.
• wait_for_http() raises TimeoutError when no accepted response is received before the deadline.
• A SeaweedFS connectivity failure is reported as a SeaweedFS readiness/bootstrap failure, not as a Milvus image regression.
• No production Service exposure or NetworkPolicy relaxation is introduced.

Scope

This proposal is limited to correcting the integration access path and timeout reporting. It does not change SeaweedFS tenant isolation, credentials, data, PVCs, or the Milvus image version.