Making Login Thread-Safe: Four Concurrency Lessons Learned

Summary

• Scenario

  • A legacy Python backend running on k8s, with a single service started using multiple workers/threads. When calling other internal system APIs, it might work fine on day one but throw “insufficient permission” or session-expired errors on day two.

  • Request
      │
      ▼
    This system
      │
      ▼
    Login to other system → store credential
      │
      ▼
    Start work after successful login...

• Debugging train of thought

  • Since the API returned messages like “insufficient permission,” the first step was checking what that API actually did, which confirmed the flow got stuck reporting “not enough permission” right after login appeared to succeed.
  • The question became: why would login “appear” to succeed (the previous step) yet fail afterward? Logging was added specifically to debug this, but it didn’t reveal anything conclusive.
  • Eventually, with no other options, AI was asked to scan for where the problem might be and produce a report — which revealed the key issue: the Dockerfile’s service startup layer had multi-worker/threads enabled, so the session shared across requests (and unprotected) got hit by concurrency — the multi-worker setting itself wasn’t a bug, it was just the trigger that surfaced a latent thread-safety issue.
  • This wasn’t a syntax error — it was a runtime race that only clicks from a concurrency perspective. Reading the code statically makes it hard to spot at a glance (the session is shared AND running under multi-worker — the two clues only make sense together).

• Retrospective

  • Process-wise, next time this could be improved by reading the code in parallel to understand the full flow, while asking AI at the same time.
  • Takeaway: set up the logger for a concurrent service once with process/thread identifiers baked in, so every log line carries them automatically without manual additions. This time, logs were added but revealed nothing — what was missing was exactly this correlating key. Otherwise there’s no way to tell which worker’s which thread wrote a half-finished session.

    • logging.basicConfig(
          format="%(asctime)s [%(processName)s/%(threadName)s:%(thread)d] "
                 "%(levelname)s %(name)s - %(message)s"
      )

Details

A multithreaded worker/service shares the same SessionClient. After going live, sporadic “half-finished session” errors appeared — some workers read a connection that was only half logged in, some requests inexplicably carried empty cookies. It turned out the problem wasn’t the network — it was an incomplete understanding of “locking” from the start. Some research filled in the gaps, and the following blind spots were recorded to reinforce the lessons.

Blind Spot 1: mutex Only Provides ACID’s “I”, Not A, C, or D

Wrapping login() in a lock feels as safe as wrapping it in a database transaction. It isn’t. A mutex only provides one of ACID’s four letters: I (Isolation). It serializes concurrent work, but it does not roll back — if the critical section throws mid-way, it leaves a half-finished result behind.

A lock only gives you the “I” in ACID; the A, C, and D are on you.

Deeper dive: see the mutex vs. ACID isolation breakdown.

Blind Spot 2: A Lock Won’t Auto-Roll-Back a Half-Finished Result

Since a lock doesn’t roll back, atomicity has to be built manually. The rule fits in one sentence: defer assignment to the shared variable until last. Build the session in a local variable first, confirm it succeeded, and only then write it to the shared field in a single step.

session = requests.Session()          # build in a local variable first
resp = session.post(LOGIN_URL, ...)   # succeed first
resp.raise_for_status()
self.session = session                # publish only after success, single write

If post() throws, no shared state has been touched — the next thread won’t read torn state either.

Never publish an object that’s only half-initialized.

Deeper dive: see the finish-first, then-publish safe publication pattern.

Blind Spot 3: One Network Error and the Lock Never Releases

The critical section of login() calls session.post() and raise_for_status(), both of which can throw. The early version used manual acquire()/release() without wrapping it in try/finally — a single network error meant the lock never released, and every subsequent caller trying to acquire it got stuck, deadlocking the entire worker.

with self._login_lock:   # = acquire + try/finally release
    ...                  # even if this throws, release() is guaranteed to run

Use with whenever possible — it guarantees the lock releases even on exception.

Deeper dive: see how with lock: guarantees release on exception.

Blind Spot 4: Trying to Save on Locking Costs, But Getting Double-Checked Locking Wrong

Not every call should pay the cost of a lock. The standard pattern for lazy initialization is double-checked locking: check 1 takes the hot path without a lock, check 2 re-confirms while holding the lock. These two seemingly identical ifs actually serve different purposes — one was once removed, and two threads ended up rebuilding and overwriting each other simultaneously.

Two ifs that look the same are doing different jobs — remove either one and it breaks.

Deeper dive: see the double-checked locking breakdown of two checks, two jobs.

When to Use This, and When It’s Overkill

This entire manual-atomicity setup is only needed because a mutex isn’t a transaction. If a transactional store is already available, let it handle this — no need to build it by hand.

The same goes for double-checked locking: it only pays off when the hot read path has real contention and the critical section isn’t cheap. If every call is already dominated by a DB query (~ms), the lock’s cost (~ns) is just noise — a single check is simpler.

Closing

All four lessons are really one lesson: a mutex only serializes “execution,” it doesn’t guarantee “success.” Everything else — atomicity, release, deduplication — has to be handled manually.

Questions worth asking when writing code with a multithreading mindset going forward:

• Does the code anywhere assign a shared variable first, then run initialization that might fail?
• Can the critical section throw, while still relying on manual acquire()/release()?
• Is that lock protecting “data,” or “a code path”?