Python API Pagination and Retries: Build a Reliable Requests Client

Python course module guide: Automation, APIs and error handling

A useful API client must handle more than one successful response. It needs explicit timeouts, status checks, pagination, bounded retries, rate-limit awareness and validation before downstream code trusts the data.

Begin with a defined API contract

Before writing a loop, identify the endpoint, authentication method, pagination fields, rate limits, response schema and error format. Never guess whether page numbering starts at zero or one. Use the provider’s documentation and a small inspected response.

If requests, JSON and status codes are new, begin with Python APIs for beginners. This guide focuses on the reliability controls needed after the first call works.

Create a session with retry policy

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

def build_session() -> Session:
    retry = Retry(
        total=4,
        connect=4,
        read=4,
        status=4,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET"}),
        respect_retry_after_header=True,
    )
    session = Session()
    session.mount("https://", HTTPAdapter(max_retries=retry))
    session.headers.update({"Accept": "application/json"})
    return session

Bound the retry count and restrict automatic retries to operations that are safe for your API. Retrying a write can create duplicate work unless the endpoint supports idempotency. Exponential backoff reduces pressure on an unavailable service.

Always set timeouts and check status

def fetch_json(session, url, params=None):
    response = session.get(url, params=params, timeout=(3.05, 20))
    response.raise_for_status()
    payload = response.json()
    if not isinstance(payload, dict):
        raise TypeError("Expected a JSON object")
    return payload

The two timeout values control connection and read waiting. They are not necessarily a wall-clock deadline for the complete download. raise_for_status() prevents an error page from being handled like valid data.

Page-number pagination

def iter_page_items(session, url, page_size=100):
    page = 1
    while True:
        payload = fetch_json(
            session,
            url,
            params={"page": page, "per_page": page_size},
        )
        items = payload.get("items")
        if not isinstance(items, list):
            raise TypeError("items must be a list")
        yield from items
        if not payload.get("has_more"):
            break
        page += 1

Stopping on an explicit has_more, page count or next link is safer than assuming a short page always means completion. Add a maximum-page guard when a broken API could repeat the same page forever.

Cursor pagination

def iter_cursor_items(session, url):
    cursor = None
    seen_cursors = set()
    while True:
        params = {"limit": 100}
        if cursor:
            params["cursor"] = cursor
        payload = fetch_json(session, url, params=params)
        yield from payload.get("items", [])
        cursor = payload.get("next_cursor")
        if not cursor:
            break
        if cursor in seen_cursors:
            raise RuntimeError("API repeated a cursor")
        seen_cursors.add(cursor)

A cursor is opaque: store and send it without modifying it. Detect repetition so a server defect cannot create an endless loop.

Close the session you own

def download_all(url):
    with build_session() as session:
        return list(iter_page_items(session, url))

The with block makes session ownership explicit. See Python context managers for cleanup patterns around files, connections and other resources.

Validate and deduplicate records

Check required identifiers and types before saving data. APIs can return overlapping pages when records change during pagination. Track stable IDs and decide whether the newest record should replace an older copy. Log rejected records with enough context to investigate, but never log access tokens.

Checkpoint long downloads

For a large export, write completed pages or the last safe cursor to durable storage. On restart, resume only if the provider guarantees the cursor remains valid. Otherwise restart the download and use idempotent upserts keyed by a stable record ID. The Python automation projects guide explains repeatable file and report workflows.

Rate limits and Retry-After

A 429 response means the client should slow down. Respect the provider’s retry guidance and headers. Do not create many parallel workers to bypass a limit. Rate limiting protects shared services, and violating it can block credentials.

Common API client mistakes

• Making requests without a timeout.
• Retrying every status code or retrying writes blindly.
• Assuming one response contains every record.
• Trusting JSON shape without validation.
• Following a repeated cursor forever.
• Logging tokens or personal data.
• Loading a huge export into memory when streaming would work.

Practice project

Build a client for a documented public API that supports pagination. Add a session, timeout tuple, status checks, bounded retries and schema validation. Save records by stable ID, write a progress checkpoint and create tests with fake responses. Use the output in the pandas GroupBy and merge project to turn reliable collection into analysis.

API pagination FAQs

Should POST requests be retried?

Only when the API defines safe idempotency behavior. Blindly retrying a write can create duplicate records or payments.

What is better: page or cursor pagination?

That depends on the provider. Cursors often handle changing datasets better, while numbered pages are easier to inspect and resume.

Why use a Session?

A session reuses connections and can hold shared headers, authentication and adapter policy across related requests.

Official reference: Requests advanced usage documentation.