Freeze retries and reconcile the destination
After a partial import, pause automatic retries for that batch. Keep the exact input and import logs. Compare each source record with the destination using a stable record ID. Confirm what already landed, correct definite failures, and investigate records whose outcome is unknown. Build the recovery file from rows that are safe to write.
Record the batch ID, source version, intended operation, and target account or table. If someone already started a second attempt, include it in the same investigation. Check which records may have changed since the import. Agree who can pause jobs and who can approve a recovery file.
A single database transaction can commit or roll back all its changes. PostgreSQL describes this behavior in its transaction guide. Check whether your importer used one transaction, a transaction per chunk, or separate requests to another service. A failed batch status cannot tell you which of those writes committed.
Build a ledger with one row per intended change
Choose an ID that still identifies the operation after sorting, filtering, or correcting the source. If one source record intentionally creates several outputs, include the output type or line ID in the key. Check that the source keys are unique before comparing them with the destination. If the destination already has duplicates for a key, stop and resolve the collision.
| Status | Evidence needed | Recovery decision |
|---|---|---|
| Confirmed | One destination record with the correct stable ID and expected values or version | Exclude it from the recovery file. Keep the destination record ID as evidence. |
| Failed | Definite rejection or rollback and evidence that the intended write did not land | Correct the cause. Approve the row and check the destination again before retry. |
| Unknown | No reliable write result. A timeout, incomplete log, delayed destination read, or conflicting values can leave this state. | Hold the row. Check the authoritative destination or the provider's operation status before retry. |
For an update, compare the fields the import intended to change. A matching record ID with the wrong price or quantity is an exception. Hold a row if you cannot tell whether another worker or a person made the change. Use a destination export that covers the whole batch, including pages or archived rows where applicable. A partial or delayed export cannot prove that a row is absent.
Fictional example: recover a warehouse slot import
This example is fictional. Cedar Finch Supply imports four storage locations into a warehouse tool. The stable key is the warehouse plus the slot ID. The importer stops after a timeout. A supervisor keeps the original input and checks the warehouse tool by key before any retry.
| Key | Observed evidence | Status and action |
|---|---|---|
| W1-A14 | One destination slot with the expected capacity of 12. | Confirmed. Leave it out of the recovery file. |
| W1-B08 | Validation rejected a blank zone before writing. A lookup shows no slot for this key. | Failed. Approve the corrected zone, then recheck and include this row. |
| W1-C02 | The request timed out. The destination's read view may still be behind. | Unknown. Hold until an authoritative check returns a result. |
| W1-D03 | The key exists, but capacity is 24. The input expected 20. | Unknown. Ask the owner to resolve the difference before changing this slot. |
The first recovery file contains only W1-B08. After the run, the supervisor checks its zone and confirms exactly one record for that key. A later authoritative lookup finds W1-C02 with the expected values. Mark it confirmed without a retry. Leave W1-D03 open with the owner. The ledger then shows three confirmed slots and one unresolved difference. The stopped batch still needs an exception owner.
Limit the retry to approved rows
- Keep a new recovery input linked to the original batch and source version. Record the correction for each included row.
- Recheck the destination immediately before the write. A second run or a manual edit can invalidate an earlier lookup.
- Use a write method that enforces the agreed key where the destination supports it. Make the intended behavior explicit for an existing record.
- Start with the smallest useful subset. Stop if the returned destination IDs or field values differ from the ledger.
- Read the destination after the attempt. Record counts for each status and compare them with the total number of intended changes.
For a PostgreSQL destination, INSERT with ON CONFLICT can handle a conflict on a suitable unique constraint. A skipped insert still needs a value check. If an update is intended, define which fields it may change and how it handles a newer version. For an external API, check its retry rules before making another request. For example, Stripe documents key reuse, parameter checks, and key retention. Apply the destination's own rules. A key from one system cannot protect a write in another.
If a repeat attempt has a different outcome from the ledger, stop the job and preserve the new evidence. Keep the affected row open. If the recovery also sends alerts or changes another system, reconcile those effects separately before repeating them.
Use this checklist before resuming the normal import
- The original batch input and logs have identifiers and can still be retrieved.
- Every intended operation has one ledger row and one current status.
- Duplicate keys, delayed lookups and wrong values have an owner and a hold decision.
- The recovery file contains only approved rows with a recent destination check.
- Confirmed rows are excluded. For recovered rows, exact values and record counts match.
- Any remaining differences are documented and assigned. Automatic retries follow the agreed exception rules.
PARTIAL IMPORT RECOVERY — SYNTHETIC WORKSHEET Original batch ID and input version: Destination and expected operation: Paused at / paused by: Source count and duplicate-key check: Destination view and lookup time: Key | Intended value/version | Destination ID/value | Status | Evidence | Owner | Decision Recovery file ID and included keys: Approver and pre-write lookup time: Post-write IDs and value checks: Confirmed + failed + unknown = intended operations: Open differences, owner and follow-up: Conditions to resume automatic runs:
See the duplicate actions guide when designing new retry behavior. To scope recovery for an existing import, bring the field names, the type of destination, and a small fictional example to a fit call. ZenOut's workflow automation service can help define the recovery rules and test them.