How the hard parts work.

Four load-bearing decisions, pulled out of shipped code and drawn as schematics — the edge cases where data sync, money, and real-time state go wrong, and the single invariant that keeps each one correct.

Every exhibit below is read from production code. Each carries one highlighted invariant — the property the system is built to never violate.

erp-sync --keyset-drain

§V · FIG.01

FIG. 01

No-silent-loss ERP sync

The hard part: the ERP’s change-cursor is a minute-resolution timestamp, so a burst of >1000 rows in one minute would silently strand the overflow.

watermark = MINUTE-resolution timestamp>1000 rows in one minute → overflow stranded silently

composite keyset cursor(timestamp, primary-key)
drain a saturated minutepage-by-page
restart at a 1-minute overlap floorre-reads in-place edits
idempotent upsert

Never advance the watermark unless accepted + rejected == rows_sent — silent drops are caught, not skipped.

payouts --exactly-once

§V · FIG.02

FIG. 02

Exactly-once payouts

The hard part: a payout API call can time out after the money has already moved — so “no response” is not the same as “it failed.”

provably rejected

DEFINITIVE failuremoney never moved

RELEASE fundssafe to re-claim

might have succeeded

AMBIGUOUSkeep PENDING

reconcilewebhook · status-check · admin

resolvepaid|failed

Release funds only on a provable failure — a maybe-succeeded transfer is never double-paid.

realtime --server-authoritative

§V · FIG.03

FIG. 03

Cheat-resistant real-time6 → 100+ players

The hard part: in a live multiplayer round, the client can’t be trusted — so the correct answer can never reach it, and the score can’t be set by it.

token

client requests join
gateauth + payment + room membership
scoped tokensubscribe / presence only

the correct answer is never sent to clients during a live question

answer

answer submit
atomic compare-and-setfirst-answer-wins guard
server-authoritative scoring
broadcast6 → 100+ subscribers

round

round advance
coordinator electiondeterministic leader + liveness takeover
idempotent phase claim

Scoring is server-authoritative and answers aren’t leaked — you can’t fake a correct answer. Scales 6-player rooms → 100+ broadcast.

locks --one-global-order

§V · FIG.04

FIG. 04

Deadlock-free money pathsLogistics · Postgres

The hard part: two money flows that lock the same rows in opposite orders will eventually deadlock — Postgres aborts one with 40P01.

flow Abillspayments

flow Bpaymentsbills

each holds what the other wants → 40P01

billssorted by bill number
then the payment row

unlocked discovery readlock in global orderrevalidate under lock

A single global lock ordering across every money path — the cycle that caused 40P01 can’t form.

These are the load-bearing parts, drawn small. The full case studies — context, stack, and what shipped — live in ~/work.