Skip to content

ci: the PyPy protocols shard stalls past the 30s case timeout since the checksum wave #2600

Closed

Description

@JarryShaw

main is red at f4aa57742. The "Python pypy3.11" job failed test_transport_edge_roundtrip_unit's tcp/min-header case with REBUILD: TimeoutError: did not finish within 30s (Unit Tests run 41340399393).

Root cause: a regression from the checksum wave (#2543, 47fac072e), not a flake.

  • Not the rebuild: the TCP rebuild path did not change. It takes about 1 ms warm and 0.2 s cold at every commit measured.
  • The shard is heavier: the wave added 71 tests to the PyPy tests/protocols tests/corekit shard. Measured locally at -n 4, peak RSS rose from 15.6 GB to 18.1 GB on a 16 GB runner. CI memory itself could not be observed, so memory pressure is the working hypothesis.
  • The stall lands at the tail: the transport tests sort last, so they run at peak heap. In 5 of the 6 PyPy legs since the wave, one UDPChecksumTests test stalls for 85–146 s, against a pre-wave maximum of 33 s. The 30 s time_limit counts wall clock, so a stalled worker fails it.

Fix: split the PyPy shard. Set SHARDS in unit-tests.yml to tests/protocols/internet tests/protocols/application tests/protocols/misc tests/protocols tests/corekit, and update the SHARDS pin in test_pypy_worker_memory_2005 and the PyPy paragraph in workflows.rst.

  • Emulated locally: peak RSS 5.4 GB, slowest test 6.9 s, total time 3 m 51 s against 4 m 41 s.
  • Raising the timeout would only hide the stall. A smaller -n only hands each worker more tests.

Activity

  1. added
    ciPull requests that change CI or workflow configuration (ci: subject prefix)
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Nov 1, 2026
  2. added this to the PyPCAPKit v1.6 milestone on Nov 2, 2026
  3. removed
    wipWork in flight - a covering PR is open or an agent is actively on it
    on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciPull requests that change CI or workflow configuration (ci: subject prefix)

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions