loke.dev
Header image for The Zombie Socket

The Zombie Socket

Physical cables don't just snap; in the cloud, connections die silently, leaving behind phantom resources that will eventually choke your production throughput.

· 7 min read

Have you ever wondered why a production service, showing 0% CPU load and plenty of available memory, suddenly stops accepting new traffic while its logs remain eerily silent?

It’s a haunting scenario. You check your metrics: the dashboard says the service is "Up." You check the process: it’s running. But the moment you try to send a request, it vanishes into a black hole. You aren’t looking at a crash; you’re looking at a graveyard of Zombie Sockets.

In the cloud, we’ve been conditioned to think of networking as an abstraction—a series of pipes that either work or break. But physical cables don't just "snap" in a virtualized environment. Instead, connections die silently. They leave behind phantom resources that occupy space in your file descriptor tables, hold onto memory, and eventually choke your throughput to death.

The Illusion of "Established"

In a perfect world, TCP is a polite protocol. When one side wants to close a connection, it sends a FIN packet. The other side acknowledges it, sends its own FIN, and they part ways. If a process crashes, the kernel is usually smart enough to clean up and send a RST (Reset) packet to the other side.

But the cloud is not a perfect world. Between your client and your server, there are Load Balancers, NAT Gateways, Firewalls, and Service Meshes. These "middleboxes" are stateful. They maintain a table of every connection passing through them.

The problem? These tables aren't infinite. To save space, middleboxes will unilaterally drop a connection from their state table if it has been idle for too long. Crucially, they often do this without telling the sender or the receiver.

From the perspective of your server, the socket is still in an ESTABLISHED state. It is waiting for data that will never come. From the perspective of the client, the connection is also ESTABLISHED. But the bridge between them has been demolished.

The Cost of Silence

Why does this matter? Can’t we just wait for the application to time out?

Not necessarily. By default, a Linux TCP connection with no data being sent can stay in the ESTABLISHED state for a very, very long time. If you don't implement heartbeats or keepalives, you end up with "leaked" sockets.

Every socket is a file descriptor. Every file descriptor has a cost. If your application handles a few thousand connections and 20% of them become zombies every hour, you will eventually hit the ulimit (the maximum number of open files). Once that happens, your accept() calls start failing with EMFILE. Your server is now a brick.

Finding the Dead: A Diagnostic Script

Before we fix it, we have to see it. If you suspect your service is being haunted, you can’t rely on application logs. You need to look at the networking stack.

I often use a combination of ss (socket statistics) and lsof to find these phantoms. Here is a quick bash snippet I use to find connections that have been idle for an abnormally long time:

# Look for TCP connections in ESTABLISHED state
# and show the "timer" column to see how long they've been idle.
ss -atpno state established | awk 'NR>1' | while read -r line; do
    # Example output: 
    # tcp ESTAB 0 0 10.0.1.4:443 192.168.1.100:54321 timer:(keepalive,1min5s,0)
    echo "$line"
done

If you see a massive list of connections with no data moving (Recv-Q and Send-Q at 0) and timers that seem stuck, you’re looking at zombies.

Fighting Back: TCP Keepalives

The first line of defense is the TCP Keepalive. This is a kernel-level feature where the OS sends a tiny probe packet with no data after a period of inactivity. If the other side (or the middlebox) doesn't acknowledge the probe, the kernel realizes the connection is dead and closes the socket, notifying your application.

In many languages, this is off by default. You have to opt-in.

Here is how you would enable and tune keepalives in a Go application. I prefer Go for this example because it gives you granular control over the net.Dialer and net.ListenConfig.

package main

import (
	"net"
	"time"
	"log"
)

func main() {
	// Create a listener with custom keepalive settings
	lc := net.ListenConfig{
		KeepAlive: 15 * time.Second, // Frequency of keepalive probes
	}

	listener, err := lc.Listen(nil, "tcp", ":8080")
	if err != nil {
		log.Fatal(err)
	}
	defer listener.Close()

	log.Println("Server started on :8080 with 15s KeepAlives")

	for {
		conn, err := listener.Accept()
		if err != nil {
			log.Println("Accept error:", err)
			continue
		}
		
		// At this point, the underlying TCP socket has keepalives enabled.
		// If the client vanishes, the OS will close this conn in ~1 minute.
		go handleRequest(conn)
	}
}

func handleRequest(conn net.Conn) {
	defer conn.Close()
	// Do work...
}

Warning: Setting the KeepAlive interval in the code is only half the battle. On Linux, the actual timing is often governed by system-wide sysctl settings:
- net.ipv4.tcp_keepalive_time: 7200 (seconds until first probe)
- net.ipv4.tcp_keepalive_intvl: 75 (seconds between probes)
- net.ipv4.tcp_keepalive_probes: 9 (how many probes to fail before killing)

The default tcp_keepalive_time is 2 hours. That is far too long for cloud environments where a NAT Gateway might time out after 5 minutes (300 seconds). You need to tune these values in your Dockerfile or your host's /etc/sysctl.conf.

When TCP Keepalive Fails: The User Timeout

There is a nasty edge case. What if you *are* trying to send data, but the connection is dead? TCP Keepalives only trigger when the connection is idle. If your application writes to a zombie socket, the data will sit in the kernel's retransmission queue. The kernel will keep trying to send that data, exponentially backing off, potentially for 15 minutes or more.

To fix this, Linux provides TCP_USER_TIMEOUT. This sets a hard limit on how long transmitted data can remain unacknowledged before the stack kills the connection.

Here is how you might set this in a C-based or Python environment (where you have access to setsockopt):

import socket

# Define the constant if not available in your python version
TCP_USER_TIMEOUT = 18  # This is the constant for Linux

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Set TCP_USER_TIMEOUT to 10,000 milliseconds (10 seconds)
sock.setsockopt(socket.IPPROTO_TCP, TCP_USER_TIMEOUT, 10000)

sock.connect(("your-db-instance.cloud.com", 5432))

Setting TCP_USER_TIMEOUT ensures that if the network path is severed while you're in the middle of a request, your worker thread isn't held hostage for 15 minutes. It will fail fast, allowing your application to retry or return an error.

Application-Level Heartbeats

Sometimes, the kernel isn't enough. If you are using a protocol like WebSockets, gRPC, or long-polling, the connection might be "alive" at the TCP level, but the application at the other end has locked up or entered an infinite loop.

In these cases, you need application-level heartbeats (often called PING/PONG).

If you’re building a gRPC service, don't rely on the default settings. Configure the "Keepalive" parameters on both the client and the server.

// gRPC Server-side Keepalive Enforcement
var kasp = keepalive.EnforcementPolicy{
    MinTime:             5 * time.Second, // Minimum time between client pings
    PermitWithoutStream: true,            // Allow pings even if there's no active stream
}

var kps = keepalive.ServerParameters{
    MaxConnectionIdle:     15 * time.Second, // Kill idle connections
    MaxConnectionAge:      30 * time.Minute, // Periodically rotate connections
    Time:                  5 * time.Second,  // Ping the client every 5s
    Timeout:               1 * time.Second,  // Wait 1s for response
}

s := grpc.NewServer(
    grpc.KeepaliveEnforcementPolicy(kasp),
    grpc.KeepaliveParams(kps),
)

By enforcing these at the application layer, you ensure that "zombie" clients that have crashed or lost power are purged from your server's memory quickly, freeing up resources for healthy traffic.

The Database Connection Pool Trap

The most common place I see zombie sockets is in database connection pools (Postgres, MySQL, Redis).

Imagine this:
1. Your App connects to Postgres via a Cloud NAT Gateway.
2. The NAT Gateway has an idle timeout of 5 minutes.
3. Your App has a connection pool that keeps connections open for 30 minutes.
4. Traffic drops at 2 AM. A connection sits idle for 6 minutes.
5. The NAT Gateway drops the mapping.
6. At 2:10 AM, a request comes in. The App tries to use the "Established" connection from its pool.
7. Hang. The App waits for a response that never comes because the packets are being dropped by the NAT Gateway.

To solve this, your connection pool's "Max Idle Lifetime" must be shorter than the idle timeout of any middlebox in your network path. If your AWS NLB times out at 350 seconds, set your pool's max idle time to 300 seconds.

Summary Checklist for Production

If you want to keep your production environment free of the undead, follow these rules:

1. Map your Network: Know the idle timeouts of your Load Balancers and Firewalls.
2. Turn on TCP Keepalives: Don't trust the defaults. Set them at the socket level.
3. Tuning sysctls: On Linux nodes, reduce net.ipv4.tcp_keepalive_time from 7200 to something like 300.
4. Use `TCP_USER_TIMEOUT`: If you are writing data, don't let the kernel hang for 15 minutes.
5. Pool Management: Ensure application-level connection pools expire idle connections *before* the network infrastructure does.
6. Monitor Socket States: Add alerts for high counts of ESTABLISHED sockets relative to your actual request volume.

The cloud gives us the illusion of infinite scale, but that scale is built on top of finite resources. Every zombie socket you allow to linger is a tiny leak in your ship. Eventually, those leaks will sink you. Don't let your connections die in silence—force them to speak or force them to leave.