My role covered the architecture, specifications, courier integration design, and engineering review. This case study focuses on the booking and webhook layer: how the backend handles uncertain API responses, repeated requests, and delivery updates.
The setting
Small sellers in Bangladesh take orders through Facebook and chat messages, then book each parcel with a courier (Steadfast, Pathao, or RedX) by hand. The backend reads orders out of chat text, verifies them, books couriers, and tracks delivery from courier webhooks. Money is involved at every step: courier charges, cash on delivery, advance payments.
The core problem: lost responses
The backend sends “create parcel” to the courier. The courier creates it. The response never arrives: a read timeout, a dropped connection, a garbled body. Now the backend doesn’t know.
- Retry, and the courier may create a second booking, with a second charge and a second pickup request.
- Assume it failed, and the seller thinks the order isn’t booked when it is.
Neither guess is safe. So the design makes “we don’t know” an explicit state that blocks retries until someone reconciles it.
Seller API orders row Courier
|--book----->| lock (merchant, order) | |
| |--SELECT ... FOR UPDATE->| status=approved |
| |--create parcel (1 try)------------------------->| created
| | X read timeout / bad body <-----------------|
| |--mark booking_unknown, audit, COMMIT->| |
|<--502------| | |
|--retry---->|--locked row: booking_unknown -> 409 "reconcile first"
| | dashboard + readiness check show the order as needing attention
Decisions
Classify failures by whether the request could have arrived. A connection that never opened is safe to retry. A timeout or bad response after sending is not; it becomes booking_unknown. The create-parcel call is never retried automatically.
Serialize booking requests. The design combines an in-process lock per merchant and order with a database row lock to control concurrent booking attempts. An order that’s already booked with the same courier returns the existing booking without calling the courier again.
Store webhooks first, process second. Every courier callback is saved raw before anything acts on it, deduplicated on the courier and its event ID, and processed by a worker. Older timestamps can’t move an order backwards, and delivered, returned, or cancelled orders can’t leave that state.
Rejected alternative: retry with backoff. It’s the default in most HTTP clients and it’s exactly wrong for a non-idempotent create call. None of the three courier APIs offer an idempotency key, so the only safe automatic action after an ambiguous failure is to stop and flag it.
Failure cases and the tests that cover them
| Scenario | Handling | Test |
|---|---|---|
| Two identical booking requests at once | Lock per order; one courier call | concurrent identical confirmation returns one booking |
| Booking request replayed | Already booked: return existing, no courier call | replay does not call provider twice |
| Response lost after sending | Mark booking_unknown, block retries |
indeterminate result blocks unsafe retry |
| Courier rejects the booking | Order stays approved, safe to retry | provider error leaves order approved |
| Same webhook delivered twice | Unique (courier, event ID) | raw event captured, deduplicated, processed |
| Webhooks out of order | Older timestamp ignored; terminal states final | out-of-order replay does not regress |
| Forged callback claims a real event ID | Invalid callbacks never reserve the ID | invalid signature cannot poison later event |
| Bulk worker dies mid-booking | Stale lease marks the item unknown | stale running lease becomes unknown |
The suite has about 445 tests. The same “nothing lost silently” rule applies to order extraction from chat messages: every non-blank line of a message must belong to an order or be flagged for review, and there are 70 tests built around messy and adversarial messages.
Integrating three courier APIs
The adapters follow each courier’s authentication model: bearer token, header, or URL token, checked using constant-time comparison. Provider-specific status mappings keep the rest of the backend independent of each courier’s response format.
Proof
Backend architecture and courier integrations
- Backend with tests covering each failure case, including a timeout after a courier has accepted a booking.
- Failure handling is described in the case study above.
Good to know
- Courier APIs confirm bookings differently; reconciliation relies on each courier offering a lookup.
Integrations that handle uncertain outcomes →