Skip to content

feat!: iterate only the current page when looping over a list response #744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tribble wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat!: iterate only the current page when looping over a list response #744

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/V6_MIGRATION_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ List endpoints now return typed page wrappers with cursor metadata and built-in
```python
page = client.organizations.list_organizations()

for organization in page:
for organization in page.auto_paging_iter():
print(organization.id)

assert page.before is None or isinstance(page.before, str)
Expand All @@ -273,12 +273,14 @@ assert page.after is None or isinstance(page.after, str)
```python
page = await async_client.organizations.list_organizations()

items = [organization async for organization in page]
items = [organization async for organization in page.auto_paging_iter()]
```

**Affected users:** Any code that expected a handwritten list wrapper or manually handled pagination state differently.

**Migration:** Update pagination code to work with `SyncPage` or `AsyncPage`, and use `page.data`, `page.before`, `page.after`, or iteration over the page as needed.
**Migration:** Update pagination code to work with `SyncPage` or `AsyncPage`, and use `page.data`, `page.before`, `page.after`, or `auto_paging_iter()` as needed.

**Breaking change (v11):** Iterating a page directly (`for organization in page`, `async for`) yields only the current page's `data` and makes no further requests. In v6–v10 it auto-paginates; in v11, call `page.auto_paging_iter()` to iterate across all pages.

@greptile-apps greptile-apps Bot Oct 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 V11 guidance is hard to find The instruction for upgrading page iteration to v11 appears only in this v5-to-v6 migration guide. Developers upgrading from v10 may miss the need to use auto_paging_iter() and silently process only the first page. Please put the instruction somewhere v11 users are likely to look.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: docs/V6_MIGRATION_GUIDE.md
Line: 283

Comment:
**V11 guidance is hard to find** The instruction for upgrading page iteration to v11 appears only in this v5-to-v6 migration guide. Developers upgrading from v10 may miss the need to use `auto_paging_iter()` and silently process only the first page. Please put the instruction somewhere v11 users are likely to look.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!


### Requests now retry by default

Expand Down
11 changes: 6 additions & 5 deletions src/workos/_pagination.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ def auto_paging_iter(self) -> Iterator[T]:
page = page._fetch_page(after=page.after)

def __iter__(self) -> Iterator[T]:
"""Iterate through all items across all pages."""
return self.auto_paging_iter()
"""Iterate this page's items only; use auto_paging_iter() to cross pages."""
return iter(self.data)


@dataclass
Expand Down Expand Up @@ -109,6 +109,7 @@ async def auto_paging_iter(self) -> AsyncIterator[T]:
break
page = await page._fetch_page(after=page.after)

def __aiter__(self) -> AsyncIterator[T]:
"""Iterate through all items across all pages."""
return self.auto_paging_iter()
async def __aiter__(self) -> AsyncIterator[T]:
"""Iterate this page's items only; use auto_paging_iter() to cross pages."""
for item in self.data:
yield item
4 changes: 2 additions & 2 deletions tests/test_http_backends.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,13 +374,13 @@ def handler(request: Any) -> Any:
page = await client.user_management.list_users(
email="alice@example.com", limit=7
)
users = [user async for user in page]
users = [user async for user in page.auto_paging_iter()]
else:
sync_client = WorkOSClient(api_key=API_KEY, http_client=http_client)
sync_page = sync_client.user_management.list_users(
email="alice@example.com", limit=7
)
users = list(sync_page)
users = list(sync_page.auto_paging_iter())
finally:
if asynchronous:
await http_client.aclose()
Expand Down
65 changes: 55 additions & 10 deletions tests/test_pagination.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,24 +108,26 @@ async def _fetch(after=None):
assert [i.id for i in items] == ["1", "2", "3"]


ORG_BASE = {
"object": "organization",
"domains": [],
"metadata": {},
"external_id": None,
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:00:00Z",
}


class TestPaginationHTTPIntegration:
"""Integration test verifying auto_paging_iter fetches multiple pages via httpx."""

def test_auto_paging_iter_fetches_two_pages(self, workos, httpx_mock):
org_base = {
"object": "organization",
"domains": [],
"metadata": {},
"external_id": None,
"created_at": "2024-01-01T00:00:00Z",
"updated_at": "2024-01-01T00:00:00Z",
}
page1_json = {
"data": [{"id": "org_1", "name": "Org 1", **org_base}],
"data": [{"id": "org_1", "name": "Org 1", **ORG_BASE}],
"list_metadata": {"after": "cursor_page2"},
}
page2_json = {
"data": [{"id": "org_2", "name": "Org 2", **org_base}],
"data": [{"id": "org_2", "name": "Org 2", **ORG_BASE}],
"list_metadata": {},
}
httpx_mock.add_response(json=page1_json)
Expand All @@ -140,3 +142,46 @@ def test_auto_paging_iter_fetches_two_pages(self, workos, httpx_mock):
requests = httpx_mock.get_requests()
assert len(requests) == 2
assert "after=cursor_page2" in str(requests[1].url)


class TestPageLocalIteration:
"""Iterating a page yields only that page's data and makes no extra request."""

def test_iter_yields_current_page_only(self, workos, httpx_mock):
httpx_mock.add_response(
json={
"data": [
{"id": "org_1", "name": "Org 1", **ORG_BASE},
{"id": "org_2", "name": "Org 2", **ORG_BASE},
],
"list_metadata": {"after": "cursor_page2"},
}
)

page = workos.organizations.list_organizations(
request_options={"max_retries": 0}
)
items = list(page)

assert [item.id for item in items] == ["org_1", "org_2"]
assert len(httpx_mock.get_requests()) == 1

@pytest.mark.asyncio
async def test_aiter_yields_current_page_only(self, async_workos, httpx_mock):
httpx_mock.add_response(
json={
"data": [
{"id": "org_1", "name": "Org 1", **ORG_BASE},
{"id": "org_2", "name": "Org 2", **ORG_BASE},
],
"list_metadata": {"after": "cursor_page2"},
}
)

page = await async_workos.organizations.list_organizations(
request_options={"max_retries": 0}
)
items = [item async for item in page]

assert [item.id for item in items] == ["org_1", "org_2"]
assert len(httpx_mock.get_requests()) == 1
Loading