Define when a paginated API read is complete
Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.
A page containing fewer records than requested does not, by itself, prove that an API read is finished. Neither does an empty page. A traversal is finished when the API’s documented terminal condition has been reached, every required page has been successfully processed, and the result is described within an explicit consistency boundary.
That last condition matters. Reaching the end of a live collection does not necessarily mean you captured every record that existed when the read began, every record that existed when it ended, or a single consistent version of those records.
Useful API pagination completeness testing therefore separates two questions: Did the client finish following the pagination protocol? And what dataset can the resulting records legitimately represent? Define both before building the loop.
1. Write two completion statements, not one
Start with a protocol completion statement. It should identify the exact response condition that ends traversal and the processing work that must already be durable.
Protocol completion means the documented end signal was received, every preceding continuation was followed successfully, and all accepted page results were durably recorded.
Then write a dataset coverage statement. This specifies the population you intended to read, the ordering assumptions, and how changes during traversal affect the claim.
Dataset coverage means the collected records satisfy the agreed population and consistency contract. If the API does not establish a stable population, coverage remains qualified even after protocol completion.
These statements produce useful outcomes beyond a single success flag:
- Protocol complete, coverage supported: the traversal ended correctly and evidence supports the stated dataset contract.
- Protocol complete, coverage qualified: the traversal ended correctly, but concurrent changes prevent a stronger coverage claim.
- Incomplete: an error, expired continuation, processing failure, or safety stop prevented completion.
- Indeterminate: the response contradicts the expected pagination contract, so the client cannot establish whether it reached the end.
Use these distinctions in job reports. A green status labelled only “complete” invites downstream users to assume more than the read established.
2. Identify the documented end signal
Read the pagination contract for the specific operation. Do not infer it from a different operation supplied by the same provider. Record the response field, its terminal value, and whether that value is meaningful only on a successful response.
Possible contract designs include an absent continuation token, a null next link, an explicit end flag, or an empty result page. These are alternative designs, not interchangeable rules. An empty page is terminal only when the operation’s documentation says it is.
A short page is especially easy to misread. The requested page size may be a maximum rather than an exact count. Unless the contract establishes otherwise, a response containing fewer records than requested must still be checked for continuation.
Set explicit handling rules for ambiguous cases:
- A nonterminal response with no usable continuation is a contract failure, not successful completion.
- A repeated token or repeated request position triggers investigation or a safety stop, not an end-of-data result.
- A request budget or elapsed-time limit stops the job as incomplete.
- A failed request with no records is not an empty successful page.
- A reported total is supporting evidence only under its documented meaning; it is not a substitute for the terminal condition.
Treat continuation tokens as opaque unless documentation explicitly defines their structure. Do not increment, decode, or reconstruct them to guess the next position.
3. Define the population and ordering before the first request
A reproducible traversal needs a written scope: account or tenant, authorisation context, filters, sort order, requested fields, and any supported time boundary. Keep those inputs unchanged while following a continuation unless the contract explicitly permits changes.
Ordering needs particular attention. A timestamp alone may not provide a unique position when multiple records share it. Where supported, an immutable unique identifier can provide a tie-breaker. Record both the requested order and the provider’s documented guarantees; requesting an order does not prove that it remains stable during concurrent writes.
Different pagination designs expose different risks:
- Offset pagination: additions or removals before the current offset can shift later positions, creating skips or repeats.
- Keyset pagination: advancing beyond a key can miss records that later become visible behind that key. Mutable sort keys can also move records across the boundary.
- Opaque cursors: the token may preserve a position, a server-side session, or something else. Its presence alone says nothing about snapshot consistency.
A fixed upper boundary can make a task finite, but it does not automatically freeze membership. For example, filtering by a creation timestamp does not prevent an older record from becoming visible later. Nor does it preserve the earlier values of records updated during the read.
If the business requirement is “all qualifying records as they existed at the starting instant,” confirm whether the API documents a mechanism that supports that requirement. If it does not, change the requirement or qualify the result rather than implying snapshot isolation.
4. Filled traversal worksheet: an empty page and a changing collection
Fictional worked example: every record, time, token, response rule, and count below is hypothetical. This is a proposed test fixture, not an Alactic capability or customer result.
A team wants a list of open business records created before a fixed cutoff. Its hypothetical API documents that a successful response with a null next token is terminal. It also explicitly permits empty pages with a continuation token. Reads evaluate the live collection; no stable dataset version is documented.
Traversal worksheet — hypothetical run R-17
Purpose:
Read visible open records created before 10:00 UTC.
Fixed request scope:
Workspace: fictional workspace W
Status filter: open
Creation boundary: created_at < 10:00 UTC
Order: created_at ascending, then immutable ID ascending
Requested page size: 3
Authorisation context: unchanged throughout run
Documented hypothetical pagination rules:
Follow each non-null next_token on a successful response.
next_token = null means terminal.
Empty pages may carry a next_token.
Tokens are opaque and must be used with the same scope.
No stable dataset version is documented.
Initial visible qualifying records at 10:00:00:
A — created 09:00, status open
B — created 09:10, status open
C — created 09:20, status open
D — created 09:30, status open
Page 1 — 10:00:01
Request: initial scope, no token
Response IDs: A, B
next_token: token-alpha
Action: persist A and B; checkpoint token-alpha
Decision: continue despite short page
Changes after page 1:
B changes from open to closed.
E becomes visible with created_at 09:05 and status open.
D remains open but its descriptive value changes.
F is created at 10:00:02; it is outside the creation boundary.
Page 2 — 10:00:03
Request token: token-alpha
Response IDs: none
next_token: token-beta
Action: persist successful empty-page receipt;
checkpoint token-beta
Decision: continue despite empty page
Page 3 — 10:00:04
Request token: token-beta
Response IDs: C, D
next_token: null
Action: persist C and D; persist terminal receipt
Decision: protocol traversal complete
Final evidence:
Successful pages: 3
Raw record occurrences: 4
Unique IDs collected: 4 — A, B, C, D
Duplicate occurrences: 0
Outstanding page-processing failures: 0
Documented terminal signal observed: yes
Coverage assessment:
E was not returned; it became visible behind the cursor.
B was collected before it stopped qualifying.
D was collected with its later descriptive value.
F was correctly excluded by the creation boundary.
Outcome:
Protocol complete; dataset coverage qualified.
Output is not a demonstrated start-time or end-time snapshot.The worksheet exposes two independent defects a simplistic client could introduce. Stopping after page one would miss later results because the page was short. Stopping after page two would miss them because the page was empty. Following all tokens fixes those client errors, but it does not recover E or make the collected versions mutually consistent.
5. Choose how changing records will be treated
The worksheet needs a policy for each kind of change, not just a general note that data may change.
Additions and newly visible records
State whether records appearing after the start belong in the target population. Distinguish newly created records from older records that become visible later. A creation cutoff can exclude the former while still leaving uncertainty about the latter.
Updates and records leaving the filter
Decide whether the output preserves the version observed on each page or attempts to obtain a later version. In the hypothetical worksheet, B remains in the collected output even after its status changes. Removing it would require additional evidence and would change the meaning of the output.
If a repeated ID has different values, define the selection rule. “Last observed” is an observation-order rule, not proof of the latest source version. Use documented version semantics if available; do not invent them from timestamps.
Deletions and permission changes
A record disappearing before its page is read may leave no evidence in the traversal. A change in access can similarly alter the visible population. Preserve the authorisation context where possible and report known changes that could affect visibility.
Deduplication solves repeated processing of an ID. It does not solve omissions, recover deleted records, or establish that every collected value belongs to the same instant.
6. Make retries preserve progress without claiming success early
Design page processing so a retry does not silently skip data. One recommended pattern is to persist the page result and its next-position checkpoint atomically. Another is to make page processing safely repeatable and advance the checkpoint only after processing succeeds.
If the process stops after storing records but before recording progress, it may replay a page. If it records progress before storing records, it may skip them. Choose the recoverable failure deliberately and test it.
Rate limiting is a pause or failure condition, not a terminal page. HTTP 429 indicates rate limiting, and a Retry-After header may indicate when to retry; providers differ in how limits apply. The MDN HTTP 429 reference explains those response semantics. Apply the provider’s documented retry and token-lifetime rules.
If a token expires, do not replace it with an estimated offset or silently begin a new traversal inside the same success claim. Mark the attempt incomplete and start a separately identified attempt if appropriate. Preserve enough evidence to explain which attempt produced each result.
7. Retain evidence that supports the exact claim
A record count alone cannot prove completeness. The same count can conceal both an omitted record and a duplicate, and a live total may describe a different moment from the pages.
Keep a compact traversal manifest containing:
- The run identifier, start and finish times, fixed request scope, and authorisation-context reference without secrets.
- The documented terminal rule and the consistency assumptions used for the assessment.
- A page sequence with request-position references, outcomes, returned counts, and continuation presence.
- Raw occurrences, unique IDs, duplicates, and any conflicting versions observed.
- Retry history, processing failures, safety stops, and the terminal receipt.
- The final protocol status and a separate dataset coverage assessment.
Continuation values may be sensitive. Restrict any raw values needed for resumption; use safe references or fingerprints in ordinary operational reports.
Counts are still useful for detecting anomalies. Compare them only against a baseline whose scope, timing, and consistency meaning are understood. An unexplained mismatch should qualify or block the coverage claim, not be explained away because the loop ended normally.
8. Turn the worksheet into acceptance tests
Before the next production read, fill this worksheet for one actual operation and run controlled pagination fixtures. Include a short nonterminal page, an empty nonterminal page, a terminal page, a repeated continuation, a transient request failure, a processing interruption, and records changing around the traversal boundary.
Make the expected outcomes observable:
- Every valid continuation is followed, regardless of page length.
- No failed request or safety stop is reported as successful completion.
- Interrupted processing can resume or replay without silently losing accepted records.
- The terminal decision is backed by a retained successful response.
- Concurrent changes produce the coverage qualification specified in advance.
The next action is concrete: have the integration owner and the business owner approve the same completed worksheet. The integration owner confirms the documented mechanics; the business owner confirms whether the resulting coverage is sufficient for the intended decision. A traversal is ready to rely on when its evidence supports that agreed claim—not merely when its loop stops.
Put the next conversation to work.
Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.
Explore Communications