Reason about overlapping backend work by identifying invariants, race conditions, atomic boundaries, critical sections, coordination scope, optimistic and pessimistic strategies, contention, and deadlocks.

Backend Concurrency: Preserve Invariants When Requests Overlap

Illustrative concurrency failure (not a reported financial incident): Suppose an account starts with a $100 balance. Two $50 withdrawal requests on different application replicas both read balance = 100, both approve their own withdrawal, and each later writes the previously calculated value 50. If both payouts were actually executed, the account would have paid out $100 but the stored balance would incorrectly remain $50 instead of $0. The system would under-record the debit by $50; the funds did not mysteriously “vanish”. This constructed example assumes both operations are accepted, no cross-request idempotency protection is in place, and the database permits both stale computed writes to commit (for example under PostgreSQL READ COMMITTED). It is not evidence of a real breach or of transaction behavior under every isolation level.

Concurrency bugs arise when overlapping work reads state, makes decisions and commits effects without a mechanism that preserves the full business invariant. BEGIN/COMMIT alone does not mean a read-check-write sequence is serializable; transaction isolation, constraints, atomic conditional updates or shared coordination determine which interleavings can succeed. Two asynchronous tasks can race even on one CPU core; separate replicas widen the coordination scope.

TL;DR

💡 Rule of thumb: A process-local lock cannot enforce a cross-replica invariant. Prefer a datastore-owned atomic conditional update or constraint, or use transactions with the appropriate isolation and locking. If the workflow includes external side effects, separately design durable intent, retry and deduplication boundaries.

Invariants define coordination boundaries: Concurrency is not about whether two CPU cores execute simultaneously, but whether overlapping operations violate business invariants (such as non-negative balances or strict inventory caps).
Process-local locks stop at the process boundary: An in-memory mutex (sync.Mutex, language locks) cannot coordinate across multiple container instances, serverless functions, or background workers sharing a database.
Coordination primitives by contention: Use atomic conditional writes (UPDATE ... WHERE balance >= 50) for simple numeric transitions, optimistic concurrency (version = version + 1) for low-contention document edits, and pessimistic locking (SELECT ... FOR UPDATE) when conflict retries would cause disastrous retry storms.
Contention is an unmodeled queue: When hundreds of requests fight over the same critical section, lock wait times spike, connection pools saturate, and latency degrades; shrink critical sections rather than weakening invariants.
High-impact pitfall: A stale read-check-write pattern (SELECT balance → app check → write a previously computed balance) can approve both operations and overwrite one debit under permissive isolation. Preserve both the database invariant and the separately committed business effects.

The goal is not “use locks.” The goal is make invalid interleavings impossible or detect and reject them before they commit.

1. Start from the invariant

Concurrency bugs are easier to reason about when you stop asking “can these two lines run at the same time?” and instead ask:

What must still be true after every allowed overlap?

Examples of invariants:

inventory never becomes negative;
a username is unique within one tenant;
at most one active payout exists for a settlement batch;
an account balance cannot cross a configured floor;
one scheduled job owns a lease at a time;
an order cannot move from cancelled back to paid.

The invariant determines the protection boundary. A mutex around an in-memory map cannot enforce a uniqueness rule whose true state lives in a shared database. A database row lock cannot protect an external payment that already succeeded elsewhere.

2. A race condition is an interleaving problem

Consider a naive quota check:

read remaining = 1
if remaining > 0:
  write remaining = remaining - 1

Two requests can interleave like this:

The final number may look harmless, but the business effect is wrong: two operations were accepted from one unit of capacity.

A race condition exists when correctness depends on which concurrent operation wins a timing race. The dangerous pattern is often not a low-level torn memory write. It is a multi-step decision such as read → check → act.

A related failure is the lost update:

A reads version 7
B reads version 7
A writes change X as version 8
B writes change Y as version 8

If B overwrites A without detecting the conflict, A's successful update is lost.

3. Atomicity closes the read-check-write gap

An operation is atomic with respect to an observer when it behaves as one indivisible state transition: conflicting work cannot observe or commit a forbidden intermediate interleaving.

Instead of:

read remaining
check remaining > 0
write remaining - 1

prefer an atomic conditional state transition when the datastore supports it:

UPDATE inventory
SET remaining = remaining - 1
WHERE sku = $1
  AND remaining > 0
RETURNING remaining;

Now the datastore decides whether the transition is valid at the same boundary where it applies the write. Check the returned row count: zero rows means the attempted decrement was not applied (the SKU may be missing or its remaining count may be zero). Do not report success or schedule a side effect merely because the SQL statement returned without throwing.

For the opening bank-balance example, a PostgreSQL-style single-row transition is:

UPDATE accounts
SET balance = balance - $1
WHERE id = $2
  AND balance >= $1
RETURNING balance;

Bind $1 to the positive withdrawal amount and $2 to the account ID; authorize the account separately. Exactly one returned row means the balance change was applied in the transaction, not that an external payout has succeeded. Persist the withdrawal intent and any outbox work atomically with the balance update where possible; a remote payment service cannot be rolled back merely by rolling back the database transaction.

Atomicity is always relative to a boundary. A language-level atomic increment can protect one memory location but not automatically protect:

two related fields that must change together;
multiple rows that form one business invariant;
state split across a database and a remote API;
multiple service replicas each holding their own memory.

4. First choice: remove unnecessary shared mutable state

The cheapest lock is often the one you do not need.

Before coordinating, ask whether ownership can be simplified:

make request-local data immutable;
partition work so one owner handles one key at a time;
use append-only events instead of in-place mutation where appropriate;
compute pure values outside the critical section;
move durable uniqueness or bounds into datastore constraints;
avoid global in-memory registries for state that belongs to users, tenants, or jobs.

For example, if each worker owns exactly one partition, operations on different partitions do not need a global mutex. If uniqueness is a database rule, a unique constraint can be a stronger and simpler concurrency primitive than “check first, insert later.”

Do not treat “lock-free” as automatically better. A design that removes shared state is valuable because it reduces coordination and makes the correctness model clearer.

5. Optimistic concurrency: detect conflict, then retry or reject

Optimistic concurrency assumes conflicts are possible but not common enough to justify blocking every operation up front. The writer proves that the state it based its decision on is still current.

A typical version check looks like:

UPDATE documents
SET body = $1,
    version = version + 1
WHERE id = $2
  AND version = $3;

If zero rows are updated, the version did not match or the document was absent; distinguish those cases before returning the API's documented 404 or conflict response. The application can then:

re-read and retry when the operation is safely recomputable;
surface a conflict so a human can reconcile edits;
merge only if the domain has a correct merge rule.

The same idea appears as compare-and-swap, conditional writes, version columns, or HTTP preconditions such as If-Match.

Optimistic concurrency is a good fit when:

conflicts are relatively rare;
holding locks across user think-time would be expensive;
recomputation or conflict handling is well defined;
the write target can expose a trustworthy version/precondition.

It is a poor fit when hot contention makes many writers repeatedly fail and retry. At that point, retries become contention amplification rather than useful concurrency.

6. Pessimistic concurrency: block conflicting work deliberately

Pessimistic concurrency acquires a lock or exclusive ownership before performing the protected decision. This is useful when conflicts are expected, the protected resource is concrete, and blocking is cheaper than repeated failed attempts.

Examples include:

a process mutex protecting one in-memory structure;
a database row lock (such as SELECT ... FOR UPDATE) while a transaction updates related state;
a single-threaded owner/actor processing operations for one key;
a lease when exclusive ownership must span processes.

The key questions are:

What exactly does the lock name or own?
Who can contend for it?
How long may it be held?
What happens if the holder crashes?
Can waiters time out or cancel?
Does the lock protect the real source of truth?

A lock is not correctness by itself. It works only when every conflicting path participates in the same coordination protocol.

7. Process-local locks stop at the process boundary

This is one of the most common backend mistakes:

request arrives
acquire process-local mutex for account 42
check durable account state
write durable account state
release mutex

That mutex can coordinate threads or async tasks inside one server process. It does not coordinate another replica with a different memory space.

Both MA and MB can be acquired at the same time because they are different locks.

Use a process-local lock only when the protected invariant is genuinely process-local, or as an optimization layered on top of a shared correctness mechanism.

For cross-replica correctness, prefer primitives owned by the shared state system when possible:

atomic conditional writes;
unique/check constraints;
transactions and appropriate isolation;
datastore-provided compare-and-swap/version checks;
a durable queue or single owner per partition.

Reach for a distributed lock only when exclusive ownership itself is the domain requirement and simpler state-native primitives do not express it well. Distributed locks introduce lease duration, fencing, clock/failure, and stale-owner concerns that deserve their own deeper treatment.

8. Transactions are one concurrency tool, not a magic shield

Database transactions are appropriate when the invariant is owned by database state and several reads/writes must form one logical decision.

But BEGIN/COMMIT alone does not prove safety. You still need to reason about:

which snapshots concurrent transactions can observe;
whether two transactions can both validate and then commit conflicting decisions;
whether row/predicate locks are required;
whether Serializable isolation may abort an unsafe interleaving;
whether the application retries the whole transaction when required.

PostgreSQL 18 documents that READ COMMITTED can permit stale read-check-write decisions, whereas stronger isolation can abort some conflicting transactions. SQLSTATE 40001 denotes serialization_failure; 40P01 denotes deadlock_detected. Retry the whole transaction and its decision logic with bounded attempts, not just the failed SQL statement. Do not automatically replay a remote side effect that might have completed before a retry. PostgreSQL also documents that explicit locks can deadlock and recommends consistent lock acquisition order as a primary defense.

That is why this lesson focuses on selecting the coordination scope, while Database Transactions & Isolation goes deeper on datastore-specific transaction semantics.

9. Deadlock: correct locks can still stop progress

A deadlock occurs when concurrent operations form a wait cycle.

Common defenses:

acquire multiple resources in one deterministic order;
keep critical sections short;
avoid remote calls while holding locks;
use lock/transaction timeouts;
detect and retry deadlock victims when the underlying system reports them;
reduce the number of resources one operation must hold simultaneously.

Do not “fix” deadlocks by removing locks without restoring the invariant another way. That trades an availability failure for silent data corruption.

10. Contention is a queue, whether you drew one or not

If 500 requests arrive together and require one exclusive critical section, only one can hold it at a time. Up to 499 others must wait, be rejected or be backpressured somewhere: in the application, runtime, connection pool, database, or lock manager.

That means concurrency design is also capacity design.

Useful signals include:

lock wait time;
transaction duration;
conflict/retry rate;
queue depth;
timeout/cancellation rate;
hot-key distribution;
work completed per critical-section acquisition.

When contention is high, possible improvements include:

shrink the critical section;
partition ownership by key;
batch compatible operations;
move pure computation before lock acquisition;
replace pessimistic locking with a correct atomic conditional operation;
serialize hot-key work deliberately rather than allowing unbounded retry storms.

The wrong optimization is to weaken the invariant because “the race is unlikely.”

Hypothetical production scenario: duplicate exports across replicas

Constructed teaching case, not a documented incident: A reporting API runs on six replicas and allows only one active export per (tenant_id, report_id). To prevent duplicate expensive jobs, each application process keeps an in-memory mutex keyed by that pair.

During a traffic spike, two identical requests arrive nearly together. The load balancer sends them to different replicas.

Each replica:

acquires its own process-local mutex;
queries and sees no active export yet;
creates an export row and schedules background work;
returns 202 Accepted.

Both requests succeed because the two mutexes never coordinate with each other.

Impact: duplicate exports consume database and worker capacity, users receive multiple completion notifications, and cleanup logic races over which artifact is canonical.

Root cause: the invariant “at most one active export per tenant/report” was durable and cross-replica, but the coordination primitive was process-local. The application protected the wrong scope. The read-then-create sequence also had no shared atomic uniqueness boundary.

Correct pattern: make the invariant enforceable where the shared state lives—for example, with a datastore uniqueness/conditional-create rule covering active exports, or another atomic ownership record. Treat a process-local mutex only as an optional contention optimization. If duplicate client delivery is also possible, give the request/job a durable idempotency identity. Schedule background work from committed durable state so a retry cannot create an untracked second job.

For PostgreSQL, a partial unique index can implement one active export per pair, assuming status transitions are made transactionally:

CREATE UNIQUE INDEX one_active_export_per_report
ON exports (tenant_id, report_id)
WHERE status IN ('queued', 'running');

A second insert or status transition that conflicts with the active row fails with a uniqueness violation; handle that as an existing-job/conflict response, not as a successful second export. A partial unique index is PostgreSQL-specific, not a portable SQL guarantee. Define which statuses count as active, and ensure job scheduling uses the committed row (for example via a transactional outbox) so a worker crash cannot silently drop the work.

The lesson is broader than exports: coordination must live at least as widely as the invariant it protects.

Common mistakes

“JavaScript/async code is single-threaded, so races cannot happen”

An await creates an interleaving point. Another request can mutate shared or durable state before the first continuation resumes. Concurrency does not require simultaneous CPU execution.

“We put a mutex around it”

Ask where the mutex lives. A per-process lock does not coordinate other replicas, worker processes, cron jobs, or administrative tools unless they share the same protocol.

“A transaction means the workflow is safe”

Atomic commit and isolation are different properties. Extract the invariant and prove that the selected transaction/isolation/constraint strategy rejects unsafe overlaps.

“Optimistic retries always scale better”

At high contention, repeated conflicts can create retry storms. Measure conflict rate and choose a serialization strategy appropriate to the hot path.

“We can hold the lock while calling another service”

Remote I/O makes lock duration depend on another failure domain. Prefer committing local ownership/state quickly, then coordinate remote effects with durable workflow patterns.

“Deadlock means locking was the wrong idea”

Deadlock means the wait graph can cycle. Fix ordering, scope, timeout, or ownership design while preserving correctness.

Self-check

A service has four replicas. Each replica keeps an in-memory Set of usernames currently being registered. A request checks the set, checks the database for availability, then inserts the new user. Two requests for the same username can land on different replicas.

What is the strongest correction?

Show the reasoning

The in-memory set protects only one process and therefore cannot enforce cross-replica uniqueness. The durable invariant belongs in the shared datastore.

Use a datastore uniqueness constraint or equivalent atomic conditional insert as the correctness boundary. The process-local set may remain as an optimization to reduce duplicate work within one replica, but correctness must not depend on it. If insertion conflicts, return or translate the conflict according to the API contract instead of trying to “check harder” before the insert.

Concurrency review checklist

Agent rule

When reviewing concurrent backend code, do not approve a mutex, transaction, retry loop, or distributed lock by name alone. First extract the invariant and the complete set of actors that can violate it. Then verify that the coordination primitive covers that scope, that unsafe interleavings are impossible or rejected, and that conflict, timeout, crash, retry, contention, and deadlock behavior are explicit.

Database Transactions & Isolation — protects database-owned invariants across multiple statements and defines serialization/locking behavior.
Idempotency — repeated delivery should not accidentally repeat a durable effect.
Distributed Locks — coordinates exclusive ownership across processes but adds lease and stale-owner failure modes.
Background Jobs — worker concurrency needs explicit ownership, retry, and deduplication rules.
Message Queues — can deliberately serialize or partition work instead of allowing uncontrolled overlap.

Continue through the Backend Systems path toward background jobs, caching, queues, partial failure, and resilience.

Sources

Primary references reviewed on 2026-11-05 (PostgreSQL 18/current):

Go Memory Model — a concrete language memory model illustrating why concurrent access to shared mutable state requires synchronization.
PostgreSQL documentation — Transaction Isolation
PostgreSQL documentation — Explicit Locking
PostgreSQL documentation — Serialization Failure Handling
PostgreSQL documentation — UPDATE / RETURNING
PostgreSQL documentation — Partial Indexes

This lesson is evolving with a 180-day review target. The invariant-first reasoning model is durable, while runtime primitives, datastore behavior, and operational guidance continue to evolve.

Backend Concurrency: Preserve Invariants When Requests Overlap

On this page