Python API Pagination and Retries: Build a Reliable Requests Client

Python course module guide: Automation, APIs and error handling

A useful API client must handle more than one successful response. It needs explicit timeouts, status checks, pagination, bounded retries, rate-limit awareness and validation before downstream code trusts the data.

Begin with a defined API contract

Before writing a loop, identify the endpoint, authentication method, pagination fields, rate limits, response schema and error format. Never guess whether page numbering starts at zero or one. Use the provider’s documentation and a small inspected response.

If requests, JSON and status codes are new, begin with Python APIs for beginners. This guide focuses on the reliability controls needed after the first call works.

Create a session with retry policy

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

def build_session() -> Session:
    retry = Retry(
        total=4,
        connect=4,
        read=4,
        status=4,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET"}),
        respect_retry_after_header=True,
    )
    session = Session()
    session.mount("https://", HTTPAdapter(max_retries=retry))
    session.headers.update({"Accept": "application/json"})
    return session

Bound the retry count and restrict automatic retries to operations that are safe for your API. Retrying a write can create duplicate work unless the endpoint supports idempotency. Exponential backoff reduces pressure on an unavailable service.

Always set timeouts and check status

def fetch_json(session, url, params=None):
    response = session.get(url, params=params, timeout=(3.05, 20))
    response.raise_for_status()
    payload = response.json()
    if not isinstance(payload, dict):
        raise TypeError("Expected a JSON object")
    return payload

The two timeout values control connection and read waiting. They are not necessarily a wall-clock deadline for the complete download. raise_for_status() prevents an error page from being handled like valid data.

Page-number pagination

def iter_page_items(session, url, page_size=100, max_pages=1000):
    if page_size < 1 or max_pages < 1:
        raise ValueError("page_size and max_pages must be positive")
    for page in range(1, max_pages + 1):
        payload = fetch_json(
            session, url,
            params={"page": page, "per_page": page_size},
        )
        items = payload.get("items")
        has_more = payload.get("has_more")
        if not isinstance(items, list):
            raise TypeError("items must be a list")
        if not isinstance(has_more, bool):
            raise TypeError("has_more must be a boolean")
        yield from items
        if not has_more:
            return
    raise RuntimeError("Maximum page count reached")

Stopping on an explicit has_more, page count or next link is safer than assuming a short page always means completion. Add a maximum-page guard when a broken API could repeat the same page forever.

Cursor pagination

def iter_cursor_items(session, url, max_pages=1000):
    if max_pages < 1:
        raise ValueError("max_pages must be positive")
    cursor = None
    seen_cursors = set()
    for _ in range(max_pages):
        params = {"limit": 100}
        if cursor is not None:
            params["cursor"] = cursor
        payload = fetch_json(session, url, params=params)
        items = payload.get("items")
        next_cursor = payload.get("next_cursor")
        if not isinstance(items, list):
            raise TypeError("items must be a list")
        if next_cursor is not None and (
            not isinstance(next_cursor, str) or not next_cursor
        ):
            raise TypeError("next_cursor must be a nonempty string or null")
        if next_cursor is not None and next_cursor in seen_cursors:
            raise RuntimeError("API repeated a cursor")
        yield from items
        if next_cursor is None:
            return
        seen_cursors.add(next_cursor)
        cursor = next_cursor
    raise RuntimeError("Maximum page count reached")

A cursor is opaque: store and send it without modifying it. Detect repetition so a server defect cannot create an endless loop.

Close the session you own

def download_all(url):
    with build_session() as session:
        return list(iter_page_items(session, url))

The with block makes session ownership explicit. See Python context managers for cleanup patterns around files, connections and other resources.

Validate and deduplicate records

Check required identifiers and types before saving data. APIs can return overlapping pages when records change during pagination. Track stable IDs and decide whether the newest record should replace an older copy. Log rejected records with enough context to investigate, but never log access tokens.

Checkpoint long downloads

For a large export, write completed pages or the last safe cursor to durable storage. On restart, resume only if the provider guarantees the cursor remains valid. Otherwise restart the download and use idempotent upserts keyed by a stable record ID. The Python automation projects guide explains repeatable file and report workflows.

Rate limits and Retry-After

A 429 response means the client should slow down. Respect the provider’s retry guidance and headers. Do not create many parallel workers to bypass a limit. Rate limiting protects shared services, and violating it can block credentials.

Common API client mistakes

• Making requests without a timeout.
• Retrying every status code or retrying writes blindly.
• Assuming one response contains every record.
• Trusting JSON shape without validation.
• Following a repeated cursor forever.
• Logging tokens or personal data.
• Loading a huge export into memory when streaming would work.

Practice project

Build a client for a documented public API that supports pagination. Add a session, timeout tuple, status checks, bounded retries and schema validation. Save records by stable ID, write a progress checkpoint and create tests with fake responses. Use the output in the pandas GroupBy and merge project to turn reliable collection into analysis.

API pagination FAQs

Should POST requests be retried?

Only when the API defines safe idempotency behavior. Blindly retrying a write can create duplicate records or payments.

What is better: page or cursor pagination?

That depends on the provider. Cursors often handle changing datasets better, while numbered pages are easier to inspect and resume.

Why use a Session?

A session reuses connections and can hold shared headers, authentication and adapter policy across related requests.

Make the stopping rule part of the contract

The examples now enforce maximum page counts as well as normal completion. Their numbered-page contract requires an items list and an explicit boolean has_more. Their cursor contract treats a missing or null next_cursor as completion and otherwise requires a nonempty string. Adapt these rules to the provider’s documented response; do not apply them blindly to an API that uses links or numeric cursors.

Validate the pagination metadata before yielding a page. A missing items field should not quietly become an empty successful export. A repeated next cursor should cause a visible failure rather than an endless request cycle. A page limit also protects against an API that continually produces new cursors while never finishing.

Bound retries and the whole export separately

In the retry policy above, total=4 allows four retries after the initial attempt, not four requests overall. That limit applies to a request’s retry sequence, not the complete pagination job. With many pages, total network attempts can still be substantial. Track pages requested and job elapsed time independently.

Requests’ connection and read timeouts do not establish a complete export deadline. Retry delays, repeated calls and data processing add time. Define an operational deadline outside the request helper and stop with a resumable failure when the job cannot continue within it. A cooperative check between pages does not interrupt a request already in progress; strict deadlines require an execution mechanism that can enforce cancellation.

Do not assume a backoff setting alone bounds all server-directed waiting. A Retry-After response can request a different delay. If the required delay exceeds your available job window, defer the export rather than hammering the service with shorter waits. Check the installed urllib3 version before relying on optional retry configuration parameters.

A recovery example with overlapping pages

Imagine a fictional course catalogue export. Page one contains course IDs 101, 102 and 103. Page two contains 103, 104 and 105 because the source changed during collection. Six rows arrived, but only five distinct IDs exist. Preserve an explicit policy for ID 103: ignore an identical duplicate, or update the record when a trustworthy version field establishes which representation is newer.

For a durable export, save a completed page and its progress marker together where the storage system supports a transaction. Advancing the marker first can lose records after a crash. Saving records first without an atomic marker update can repeat work on restart, so use an idempotent write keyed by the stable ID. A checkpoint describes committed progress, not merely a response received.

Deduplication does not prove completeness. If an offset-based dataset changes while you walk it, a record can be skipped entirely. Ask whether the API offers stable sorting, a snapshot or a documented incremental export boundary. Capture those assumptions in the job summary along with accepted, rejected and duplicate counts.

Once records pass validation, the Python data pipeline guide for CSV validation and reports shows a useful next step: persist clean records and produce an auditable report. Carry the stable IDs and rejection counts into that downstream stage so collection problems remain traceable.

Test the client without calling a real service

Use a fake session that returns two known response objects and records requested parameters. Verify the second page or cursor is sent correctly and that the collected IDs match expectations. Add cases for malformed items, non-boolean has_more, a repeated cursor and a maximum-page failure. Confirm that an invalid page is not mistaken for successful completion.

Also distinguish validation tests from transport tests. A fake session can prove pagination control flow and timeout arguments; it cannot establish that a real server honours rate limits or that a particular network failure triggers the adapter as expected. Keep those claims separate in test documentation.

What if a write times out?

A read timeout does not prove that the server rejected the operation. For a non-idempotent write, first use the provider’s documented status lookup or idempotency mechanism. Repeating a create request simply because no response arrived can create a duplicate business action.

Further official references

• Requests timeouts and response handling
• urllib3 Retry configuration
• Requests sessions and adapters

Official reference: Requests advanced usage documentation.