loke.dev
Header image for How to Reclaim Ghost Sockets Without the Throughput Penalty of Application-Level Heartbeats

How to Reclaim Ghost Sockets Without the Throughput Penalty of Application-Level Heartbeats

Discover why standard TCP keep-alives fail to detect dead peers in modern cloud environments and how to implement precise connection pruning using the TCP_USER_TIMEOUT socket option.

· 7 min read

I remember staring at a netstat output for twenty minutes, trying to figure out how a server with zero active users was still reporting 14,000 established TCP connections. I had just finished building a high-throughput ingestion engine, and on paper, it was a beast. But in reality, it was bleeding file descriptors. Every time a mobile client moved from Wi-Fi to cellular or a spotty load balancer dropped a state table entry, my server held onto that socket like a sentimental keepsakes.

The "half-open" connection is a silent killer in distributed systems. Your server thinks the pipe is open; the client is long gone. If you've spent any time in the trenches of network programming, you’ve probably reached for application-level heartbeats—those "PING/PONG" messages sent every few seconds—to solve this. But for high-performance systems, application heartbeats are a tax you shouldn't have to pay. They wake up the CPU, bloat your protocol, and eat into your throughput.

There is a better way to reclaim these ghost sockets using the Linux kernel's internal timers, specifically a little-known option called TCP_USER_TIMEOUT.

The Lie of the "Established" State

In the TCP state machine, ESTABLISHED doesn't actually mean there is a live wire between two points. It simply means that, at some point in the past, a three-way handshake completed and neither side has sent a FIN or RST packet since.

If a client’s battery dies or they drive into a tunnel, no FIN is sent. The server has no way of knowing the peer is gone until it tries to send data and fails. But even then, TCP is designed to be "resilient," which is just a polite way of saying it’s incredibly stubborn. It will retry sending that data for minutes—sometimes hours—before giving up and closing the socket.

Most people assume SO_KEEPALIVE is the silver bullet here. They enable it, set it to a few seconds, and expect ghost sockets to vanish. They are usually disappointed.

Why Standard Keep-Alives Often Fail

Standard TCP keep-alives operate only when the connection is idle. If you have data sitting in the write buffer that hasn't been acknowledged (ACKed), the keep-alive timer doesn't even start. Instead, the retransmission timer takes over.

Here is the flow:
1. Your app writes data to the socket.
2. The kernel tries to send it.
3. The peer is dead, so no ACK comes back.
4. The kernel waits, then retransmits.
5. It keeps retrying, following an exponential backoff, governed by the system-wide tcp_retries2 setting.

On a standard Linux box, tcp_retries2 is usually set to 15. This can result in a socket hanging around for 13 to 30 minutes before the kernel finally kills it. During this time, your application-level "write" might have returned successfully (because it only wrote to the local kernel buffer), and you're just sitting there, leaking resources.

The Kernel-Level Solution: TCP_USER_TIMEOUT

Introduced in Linux 2.6.37 (and defined in RFC 5482), TCP_USER_TIMEOUT is the granular control we’ve been looking for. It allows you to specify exactly how long a transmitted packet can remain unacknowledged before the kernel forces the connection closed and returns an error to the application.

Unlike keep-alives, which only work when the line is quiet, TCP_USER_TIMEOUT works when you are actively trying to send data. If you use them together, you get a "pincer movement" that catches dead peers regardless of whether the connection is idle or busy.

Implementing in Python

If you're using Python's socket library, you won't find TCP_USER_TIMEOUT in the top-level constants. You have to grab the constant value (which is 18 on Linux) and apply it manually.

import socket
import struct

# The constant for TCP_USER_TIMEOUT is 18 on Linux
TCP_USER_TIMEOUT = 18

def create_hardened_socket():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    
    # 1. Enable standard Keep-Alives for idle periods
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    
    # Send a probe after 10 seconds of idleness
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
    # Send probes every 5 seconds after the initial idle period
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
    # Disconnect after 3 failed probes
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)

    # 2. Set the User Timeout for active data transmission
    # This value is in milliseconds. 
    # Here, we set it to 20 seconds (20000ms).
    timeout_ms = 20000
    sock.setsockopt(socket.IPPROTO_TCP, TCP_USER_TIMEOUT, timeout_ms)
    
    return sock

# Example usage
s = create_hardened_socket()
s.connect(("192.168.1.50", 8080))

In this setup, if the client disappears while the connection is idle, the keep-alives will clean it up in roughly 25 seconds (10 + 5*3). If the client disappears while the server is pushing a large buffer, the TCP_USER_TIMEOUT will kill the socket in exactly 20 seconds.

Precision Pruning in Go

Go’s net package is excellent, but it abstracts away many of these "knobs." To get down into the socket options, you need to use a Control function on a net.Dialer or net.ListenConfig.

Note that in Go, the TCP_USER_TIMEOUT constant is available in the syscall or unix packages.

package main

import (
	"context"
	"fmt"
	"net"
	"syscall"
	"time"

	"golang.org/x/sys/unix"
)

func main() {
	lc := net.ListenConfig{
		Control: func(network, address string, c syscall.RawConn) error {
			var err error
			c.Control(func(fd uintptr) {
				// Set TCP_USER_TIMEOUT to 15 seconds
				// The value is in milliseconds
				err = unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, unix.TCP_USER_TIMEOUT, 15000)
				if err != nil {
					return
				}

				// While we are here, let's also tighten the keep-alives
				err = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_KEEPALIVE, 1)
				if err != nil {
					return
				}
			})
			return err
		},
	}

	listener, err := lc.Listen(context.Background(), "tcp", ":9000")
	if err != nil {
		panic(err)
	}
	fmt.Println("Server listening on :9000 with custom timeouts")
    // ... accept connections
}

The beauty of this approach in Go is that it remains compatible with the standard net.Conn interface. Your application code doesn't need to know that the kernel is aggressively pruning dead wood in the background.

The Throughput Penalty of App-Level Pings

You might ask: "Why not just send a 1-byte heartbeat packet every 10 seconds?"

If you have 100 connections, it doesn't matter. If you have 100,000 connections, it matters a lot. Application-level heartbeats require:

1. Context Switches: The application has to wake up, move from kernel space to user space, process the timer, and move back to kernel space to send the packet.
2. Marshalling/Unmarshalling: Even a small heartbeat has to go through your serializer (JSON, Protobuf, etc.).
3. Buffer Bloat: You’re adding traffic to the wire that isn't productive data.
4. Power Consumption: On mobile clients, waking the radio every few seconds to say "I'm still here" is a battery death sentence.

By using TCP_USER_TIMEOUT, you are offloading the "dead peer detection" to the kernel's own retransmission logic. If data is flowing and being ACKed, the timer never fires. There is zero overhead. The kernel only acts when something is wrong.

Edge Cases: The "Intermediary" Problem

There is a specific scenario where TCP_USER_TIMEOUT is non-negotiable: Stateful Firewalls and NAT Gateways.

Many cloud environments (AWS, Azure) use stateful firewalls that drop connection tracking entries if they haven't seen traffic for a while (usually between 5 minutes and an hour). If your server tries to send data after a firewall has "forgotten" the connection, the firewall will silently drop the packets.

Your server's kernel will try to retransmit. Without TCP_USER_TIMEOUT, your server might wait 20 minutes before deciding the connection is dead, while your users are staring at a spinning loading icon. By setting the timeout to, say, 30 seconds, you fail fast, allowing your load balancer or client to reconnect and establish a fresh path.

A Note on System-Wide vs. Per-Socket

You *can* change these settings globally:
- sysctl -w net.ipv4.tcp_retries2=5

But I strongly advise against this. Global settings are blunt instruments. You might have a long-running database migration or a backup process that actually *needs* those retries to survive a brief network hiccup. By setting TCP_USER_TIMEOUT on a per-socket basis, you can be aggressive with your public-facing API connections while remaining lenient with your internal backbone traffic.

Final Recommendations

If you’re building a service that needs to handle high concurrency with high reliability, stop relying on defaults. The "Ghost Socket" problem is a primary cause of "Memory Leaks" that are actually just un-reclaimed file descriptors and their associated kernel buffers.

1. Always set `SO_KEEPALIVE` to detect dead peers during idle times.
2. Always set `TCP_USER_TIMEOUT` to detect dead peers when the network is congested or the peer has vanished during a data transfer.
3. Match your timeouts to your SLA. If your user expects a response in 30 seconds, your TCP_USER_TIMEOUT should probably be 20 seconds.
4. Trust the kernel. It's better at timing things than your application loop.

By moving this logic out of your app and into the transport layer, you simplify your code and reclaim your throughput. It’s one of those rare cases where you get more reliability by doing less work.