Courier booking that handles “did it actually book?”

When a backend asks a courier to create a shipment and the response is lost, the shipment may or may not exist. Retrying can create a duplicate booking and a second charge; assuming failure can lose the order. I designed the booking and webhook layer of a social-commerce backend around that uncertainty.

Role

Architecture, specifications, integration design, and engineering review

Stack

Python, FastAPI, PostgreSQL, arq workers, three courier APIs

Type

Backend design

Status

Backend architecture and courier integrations.

orderseller dashboardbooking workerarq · idempotency keycourier APIcreate shipmenttimeout: booked?uncertain statelook up, never blind-retryreconcilerconfirm or retry oncewebhooks → dedupe → statusrepeated deliveries collapse to one updateFastAPI · PostgreSQL · 3 couriers
  1. Problem

    Book courier deliveries for online orders through three courier APIs.

  2. Constraints

    A booking request can time out after the courier has accepted it, so "retry" may book twice.

  3. Approach

    An explicit booking_unknown state, idempotent operations, reconciliation instead of blind retries, and tests for each failure case.

  4. Result

    The create-parcel call is never retried blindly; uncertain bookings are reconciled and surfaced instead of risking a second booking.

My role covered the architecture, specifications, courier integration design, and engineering review. This case study focuses on the booking and webhook layer: how the backend handles uncertain API responses, repeated requests, and delivery updates.

The setting

Small sellers in Bangladesh take orders through Facebook and chat messages, then book each parcel with a courier (Steadfast, Pathao, or RedX) by hand. The backend reads orders out of chat text, verifies them, books couriers, and tracks delivery from courier webhooks. Money is involved at every step: courier charges, cash on delivery, advance payments.

The core problem: lost responses

The backend sends “create parcel” to the courier. The courier creates it. The response never arrives: a read timeout, a dropped connection, a garbled body. Now the backend doesn’t know.

  • Retry, and the courier may create a second booking, with a second charge and a second pickup request.
  • Assume it failed, and the seller thinks the order isn’t booked when it is.

Neither guess is safe. So the design makes “we don’t know” an explicit state that blocks retries until someone reconciles it.

Booking with a lost response
Seller        API                     orders row              Courier
  |--book----->| lock (merchant, order)  |                       |
  |            |--SELECT ... FOR UPDATE->| status=approved       |
  |            |--create parcel (1 try)------------------------->| created
  |            |   X  read timeout / bad body  <-----------------|
  |            |--mark booking_unknown, audit, COMMIT->|         |
  |<--502------|                         |                       |
  |--retry---->|--locked row: booking_unknown -> 409 "reconcile first"
  |            |  dashboard + readiness check show the order as needing attention

Decisions

Classify failures by whether the request could have arrived. A connection that never opened is safe to retry. A timeout or bad response after sending is not; it becomes booking_unknown. The create-parcel call is never retried automatically.

Serialize booking requests. The design combines an in-process lock per merchant and order with a database row lock to control concurrent booking attempts. An order that’s already booked with the same courier returns the existing booking without calling the courier again.

Store webhooks first, process second. Every courier callback is saved raw before anything acts on it, deduplicated on the courier and its event ID, and processed by a worker. Older timestamps can’t move an order backwards, and delivered, returned, or cancelled orders can’t leave that state.

Rejected alternative: retry with backoff. It’s the default in most HTTP clients and it’s exactly wrong for a non-idempotent create call. None of the three courier APIs offer an idempotency key, so the only safe automatic action after an ambiguous failure is to stop and flag it.

Failure cases and the tests that cover them

Failure handling, mapped to tests
Scenario Handling Test
Two identical booking requests at once Lock per order; one courier call concurrent identical confirmation returns one booking
Booking request replayed Already booked: return existing, no courier call replay does not call provider twice
Response lost after sending Mark booking_unknown, block retries indeterminate result blocks unsafe retry
Courier rejects the booking Order stays approved, safe to retry provider error leaves order approved
Same webhook delivered twice Unique (courier, event ID) raw event captured, deduplicated, processed
Webhooks out of order Older timestamp ignored; terminal states final out-of-order replay does not regress
Forged callback claims a real event ID Invalid callbacks never reserve the ID invalid signature cannot poison later event
Bulk worker dies mid-booking Stale lease marks the item unknown stale running lease becomes unknown

The suite has about 445 tests. The same “nothing lost silently” rule applies to order extraction from chat messages: every non-blank line of a message must belong to an order or be flagged for review, and there are 70 tests built around messy and adversarial messages.

Integrating three courier APIs

The adapters follow each courier’s authentication model: bearer token, header, or URL token, checked using constant-time comparison. Provider-specific status mappings keep the rest of the backend independent of each courier’s response format.

Proof

Backend architecture and courier integrations

  • Backend with tests covering each failure case, including a timeout after a courier has accepted a booking.
  • Failure handling is described in the case study above.

Good to know

  • Courier APIs confirm bookings differently; reconciliation relies on each courier offering a lookup.

Integrations that handle uncertain outcomes →

Next step

Integrating with an API that must not double-book?

Describe the integration and what goes wrong when a call fails halfway. I'll reply with how I'd make it safe to retry.