A supplier-feed job requests 400 part records, writes them to a database and then loses the response that would have confirmed success. The scheduler sees a timeout and runs the job again. If the second run creates another 400 records, the buying agent now sees two apparent observations for every part. If the system collapses all matching rows into one current value, it may erase a real price change that arrived between the two attempts. The transport failure has become a commercial claim.
HTTP cannot promise that a response reaches the client, even when the server completed the request. RFC 9110 defines an idempotent method by its intended server effect and explains why such requests can be retried after a communication failure. Supplier feeds often arrive through scheduled GET requests, paginated exports, webhooks or file drops, then trigger several internal writes. Safe retrieval does not make those downstream writes replay-safe. The application has to define which repeated work is harmless.
Separate four records that answer different questions
Many ingestion bugs start with one table being asked to represent transport, evidence and current state. Split the workflow into four records. An ingestion run answers when the job started, which connector and credentials it used, what pages it requested and whether it completed. A raw artifact preserves the literal response or file in private storage, together with a checksum and retrieval time. A normalized observation states what one source claimed about one identified item at one observed time. A current projection selects the latest eligible observation for a read path.
Those records have different retention and access needs. Operators need run logs to diagnose a failed page. Auditors may need the raw artifact to show what the source returned. A buyer-facing agent usually needs the normalized observation and its freshness, without receiving the entire supplier payload. The current projection can change when a late observation arrives or a source is quarantined; the underlying observations should remain available so that the change is explainable.
Give every delivery a stable replay identity
The replay key should come from the strongest identity the source provides. A webhook event ID, export ID, file name plus version, or supplier snapshot token can name the delivery. Scope it by tenant, connector and source account so that two suppliers using the same counter do not collide. CloudEvents provides a useful convention: a producer must make the combination of source and event ID unique for each distinct event, and a resent duplicate may keep the same ID. The same shape works even when the feed does not use CloudEvents.
Store a payload fingerprint beside the replay key. The key expresses intent; the fingerprint checks whether the repeated delivery is byte-for-byte or semantically the same after a declared canonicalization step. AWS describes the same client request ID with different parameters as a validation error, because treating different intent as a retry is unsafe. Supplier ingestion needs the same hard conflict. If snapshot `S-1042` arrives twice with the same payload, return the recorded result. If `S-1042` arrives with a different price or row count, quarantine it for review. Do not silently choose one version.
A content hash alone cannot express intent. Two consecutive exports can legitimately contain identical stock and price. They are separate observations if the supplier says they describe different snapshots. Conversely, a retry can acquire a different retrieval timestamp while carrying the same source event. Deduplication therefore needs both the source-scoped replay identity and a fingerprint. When the source offers no stable ID, create a documented fallback from stable fields such as source account, endpoint, page cursor, source timestamp and payload hash. Mark that fallback as weaker evidence.
Commit the key and the observation together
The replay check and the business write belong in one atomic transaction. Otherwise the process can record the key and crash before the observation exists, causing every retry to return a success that never happened. The reverse order can create the observation and crash before the key is saved, allowing the retry to create another. AWS's idempotent API guidance calls out this exact requirement: recording the request token and the related mutations must be all or nothing.
Keep raw-archive success in the contract too. One defensible sequence is to write the encrypted raw artifact first with a deterministic object name, verify the stored checksum, then transact the artifact reference, replay key, normalized observations and run state. If the transaction fails, the deterministic object can be reused on retry. If the raw archive fails, do not publish normalized observations that cannot be traced back to their literal source. The right ordering depends on the storage systems, but every crash point needs a declared recovery path.
Preserve genuine market changes
A supplier record is not the product. Keep canonical item identity separate from the source's offer observation. The item may be a manufacturer part number with a revision, packaging unit and condition. The observation belongs to a supplier account, source record, currency, quantity basis, observed time and evidence artifact. Matching the item allows comparison; it does not make two offers interchangeable or prove that either supplier holds stock.
Create a new observation when the source identifies a new event or snapshot, even if the value is unchanged. This preserves the evidence that the source repeated its claim at a later time. A current view can select the latest observation that passes identity and policy checks. It should expose both `observed_at` and `retrieved_at`: the first says when the claim applied according to the source, while the second says when your system received it. Sorting only by arrival time lets a late backfill overwrite a newer market claim.
Illustrative test case: a timeout between pages
This example is synthetic. Connector `acme-eu` requests a three-page export. The supplier labels the export `EXP-8821` and each page `1`, `2` or `3`. Page 2 contains an offer for part `AX-440-R2`: quantity 6 at EUR 820, source time 08:00 UTC. The system archives the page, creates observation `OBS-77` and commits replay key `acme-eu/EXP-8821/2`. The HTTP response is lost, so the scheduler sends page 2 again.
The second delivery has the same scoped key and payload fingerprint. The handler returns the stored result pointing to `OBS-77`; it records another attempt but creates no observation. At 09:00 UTC, export `EXP-8822` reports quantity 4 at the same price. Its new export identity creates `OBS-78`. The current projection now shows quantity 4, while the history retains the earlier quantity 6 claim. Neither quantity is labelled confirmed stock unless a separate verification step supports that statement.
Now change the replay. Suppose the repeated page claims quantity 5 but keeps key `acme-eu/EXP-8821/2`. The handler must stop with `replay_payload_mismatch`, keep the original observation unchanged and quarantine the later payload. Possible causes include a supplier mutating an export, a connector computing the key incorrectly or a page cursor being reused. Accepting the later value would erase the evidence needed to diagnose which event was real.
Failure modes the acceptance test should force
- Lost acknowledgement: the write commits and the response disappears. The retry returns the original result without a second observation.
- Concurrent duplicate: two workers receive the same delivery at once. A unique transactional key allows one commit and one replay result.
- Same key, changed payload: the handler rejects and quarantines the mismatch instead of overwriting history.
- Late backfill: an old source timestamp arrives after a newer snapshot. History accepts it, while the current projection remains on the newer eligible observation.
- Partial pagination: pages 1 and 2 succeed and page 3 fails. The run stays incomplete, and policy decides whether page-level observations remain private or become readable.
- Unstable ordering: the supplier returns identical rows in a different order. The canonicalization rule either proves semantic equivalence or treats the payload as different; it never changes informally between runs.
- Identity correction: a manufacturer suffix is resolved after ingestion. The system links or supersedes the earlier normalization without rewriting the raw artifact.
- Replay after retention expiry: an old key returns after its deduplication record has expired. The handler applies a stated horizon and flags the lower confidence instead of promising unlimited exactly-once behavior.
Keep read paths free of ingestion side effects
A search or quote-comparison request should not refresh a supplier feed as a hidden side effect. That design makes a read's latency and result depend on an external service, mixes user traffic with ingestion authority and can turn repeated searches into repeated writes. Schedule or explicitly trigger ingestion behind its own credentials and policy. Let the read path report the observation time, retrieval time, source and verification state already held. If the evidence is too old for the decision, return a freshness gap and the action needed to refresh it.
Replay safety also stops at the evidence boundary. An idempotent ingestion proves that one delivery produced one stored observation. It does not prove that the supplier owns the goods, that the quantity remains available, that the price includes the required terms, or that anyone is authorised to contact, reserve or buy. Those claims need their own current evidence and approval gates. Technical cleanliness cannot upgrade a source claim into execution authority.
The 15-minute move
Take one supplier connector and draw four boxes: run, raw artifact, observation and current projection. Write the identifier for each box and the transaction boundary between them. Then mark what happens after a timeout at every arrow. If any retry can create a second observation for the same source event, or any crash can leave a recorded success without its evidence, you have found the first replay test to add.