Two SSLs Behind One Migration Ticket

A migration task landed with one line: apply for new certificates, and move Kafka from port 9092 (plain) to port 9093 (encrypted). Short, sounds routine. I couldn’t see the point. The service already served HTTPS at its front door, so why apply for another certificate and encrypt another port?

A few terms first (read this before the rest)

• TLS / SSL. A protocol that does two jobs for a network connection: encryption (nobody in the middle can read it) and identity (you can tell who the other side is). “SSL” is the old name, “TLS” is what it’s actually called now.
• Certificate. A digital document that proves “who I am”, signed by an authority everyone agrees to trust.
• PKI (Public Key Infrastructure). The whole system that makes a certificate trustworthy: the authorities that sign certificates (CAs), the certificates themselves, and the private keys behind them. When people say “the TLS setup”, the PKI is the machinery underneath it.
• Port. A numbered channel on a server. One machine can run many services by giving each its own port number.
• Kafka. A system that lets one service hand events to another. A broker is one Kafka server.
• Ingress. In a Kubernetes cluster (a platform that runs lots of small services), the front-door layer that catches outside traffic and routes it to the right internal service.

The confusion had one root

There are two different “SSL”s here, pointing opposite ways. The front-door SSL protects traffic coming in, and it gets removed the moment traffic enters. The Kafka SSL protects traffic going out, and the service adds it. They never meet, which is exactly why the migration touched one and left the other alone.

When a system says “SSL” in two places, don’t assume they’re the same feature. Ask which direction each one protects. Half my confusion was one word doing two jobs.

What I actually had to understand

• What TLS really guarantees, and why it needs three files, not one.
• Why “mutual” TLS is a separate thing from the HTTPS I already knew.
• Why moving Kafka to “SSL” is a port change, not an upgrade of the existing connection.
• How the same client code runs encrypted in production but plain on my laptop.
• What that “other SSL” (the front-door one) had been doing all along.
• How the certificates even get onto the running service.

The six pieces

1. TLS guarantees two things, and needs three files. TLS gives you encryption (a shared secret key for the session) and identity (a certificate). The identity half is where three files show up: one verifies the other side, one is your public identity, and one is the private key that proves the identity is yours and never leaves your machine. Leak that private key and the whole thing is off. → Deeper dive: TLS basics and the three certificate files.

2. One-way vs mutual: who has to show a certificate. Normal HTTPS is one-way: only the server proves who it is. Mutual TLS makes the client prove itself too, so both ends check each other and refuse the connection if either certificate is bad. → Deeper dive: one-way TLS vs mutual TLS.

3. The Kafka change is a port change, not an upgrade. A Kafka server runs one protocol per port. Port 9092 is plain, 9093 is encrypted, and there’s no way to upgrade a live connection from one to the other. → Deeper dive: Kafka ports 9092 vs 9093.

4. The client turns encryption on by absence. The Kafka client library needs a handful of encryption settings, but only when you tell it to use them. Leave them out and it stays plain, which is what lets production run encrypted while my laptop config stays untouched. → Deeper dive: the Kafka client’s SSL config, and the absence-gating trick.

5. The “other SSL” was the front door. The SSL I kept pointing at was the front door: one-way TLS, handled automatically for inbound web traffic. It was never part of the Kafka work, because it faces the other way. → Deeper dive: ingress TLS termination.

6. The certificates arrive before the service starts. The certificate files aren’t fetched while the service runs. They’re pulled from a secret store and baked into the deployment ahead of time, then mounted as plain files. → Deeper dive: delivering certs to a pod.

Closing

The ticket was three words of jargon, and the fog was mostly one word (“SSL”) standing in for two unrelated directions of trust. Once I drew the arrow (inbound one-way at the front door, outbound mutual to Kafka), the migration stopped looking mysterious and turned back into what it was: point the client at a new port, with some certificate plumbing behind it.

Questions I’d ask myself faster next time:

• When something “already has SSL”, which direction does that SSL actually protect?
• Is this an upgrade of the existing connection, or just a different port speaking a different protocol?
• Where does the private key live, and who is allowed to read it?

When a system says “SSL” in two places, ask which direction each one protects before assuming they’re the same feature.

---

Related: start with the three certificate files and one-way vs mutual TLS; see why Kafka ports 9092 vs 9093 is a door, not an upgrade, and how the client gates SSL by absence; then the “other SSL”, ingress TLS termination, and how certs get delivered to a pod.