How to debug SYN backlog overflow on Linux

SYN backlog overflow means the system cannot keep up with incoming TCP connection attempts for a listening socket. Clients may see connect timeouts, intermittent connection failures, high connection latency, or resets depending on kernel settings and network behavior.

The useful question is:

Is the listener under a real connection burst, or is the application not accepting fast enough?

Do not start by changing every TCP sysctl. First prove which listener is pressured and whether the pressure comes from legitimate traffic, load balancer churn, SYN flood behavior, or application accept bottlenecks.

Look for kernel evidence

Check TCP counters:

netstat -s | egrep -i 'listen|overflow|SYN|reset|retrans'

On modern systems:

nstat -az | egrep -i 'Listen|Syncookies|Retrans|Reset'

Look for counters such as:

• listen queue overflow;
• SYNs to LISTEN sockets dropped;
• SYN cookies sent;
• retransmitted SYN/SYN-ACK;
• reset spikes.

Counter names vary by kernel and tool output, so preserve the exact command output and timestamp.

Inspect listeners and queues

ss -ltnp
ss -s

For a specific port:

ss -ltnp '( sport = :443 )'

Check:

• which process owns the listener;
• receive queue and send queue hints;
• configured backlog from the application;
• number of workers accepting connections;
• CPU, softirq, and scheduler pressure.

If one listener is affected and others are healthy, start with that process and its accept path.

Client-side symptom mapping

Backlog overflow is one possible cause of connect issues. Packet capture and counters are needed before naming it as root cause.

Capture handshake behavior

On the server:

tcpdump -tttt -nn -i any port 443 and 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'

Patterns:

If packet capture is too expensive on a busy host, sample carefully and use flow logs or load balancer metrics.

Case 1: Application not accepting fast enough

Strong signals:

• listener process CPU-bound or blocked;
• worker count too low;
• accept loop blocked by expensive work;
• deploy introduced synchronous startup or TLS work before accept;
• backlog counters rise while CPU or run queue is high.

Checks:

top -H -p <pid>
pidstat -t -p <pid> 1
journalctl -u <service> --since -30m

Fix path:

• separate accept loop from slow work;
• increase worker capacity where appropriate;
• reduce per-connection startup cost;
• use connection reuse upstream of the service;
• scale instances if the traffic pattern is legitimate.

Case 2: Load balancer connection churn

Strong signals:

• new connection rate spikes without matching request rate;
• keepalive disabled or too short;
• load balancer health checks are too frequent;
• clients do not reuse connections;
• TLS handshakes spike at the same time.

Fix path:

• enable or tune keepalive;
• reduce unnecessary connection churn;
• check load balancer health check interval and target count;
• spread load across more targets;
• verify idle timeout alignment.

Backlog tuning cannot fully compensate for avoidable connection churn.

Case 3: SYN flood or unwanted traffic

Strong signals:

• many source IPs with incomplete handshakes;
• SYN cookies increase sharply;
• request rate is low but SYN rate is high;
• traffic source does not match normal clients;
• firewall/load balancer security logs show attack patterns.

Fix path:

• use upstream DDoS or firewall controls;
• rate limit at the edge;
• preserve packet evidence;
• avoid blocking legitimate clients with broad rules;
• monitor SYN cookie and backlog counters after mitigation.

Tuning branch

Relevant settings may include:

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog
sysctl net.ipv4.tcp_syncookies

Application backlog also matters. For example, a server may call listen(fd, backlog) with a small backlog even if kernel limits are larger.

Fix path:

• confirm application backlog;
• align app backlog with kernel limits;
• tune per-service, not as a blind host-wide change;
• monitor memory and connection behavior after changes.

What not to do

• Do not change sysctls before identifying the listener and traffic pattern.
• Do not ignore application accept speed.
• Do not assume every connect timeout is backlog overflow.
• Do not disable SYN cookies without understanding attack exposure.
• Do not miss load balancer keepalive and health-check churn.

Decision tree

suspected SYN backlog overflow
|
+-- kernel counters increasing?
|   +-- preserve netstat/nstat output
|
+-- which listener owns the pressure?
|   +-- inspect ss -ltnp and process state
|
+-- app not accepting fast enough?
|   +-- inspect CPU, workers, accept loop, deploy changes
|
+-- new connection rate abnormal?
|   +-- inspect LB/client keepalive and health checks
|
+-- many incomplete handshakes from unusual sources?
    +-- inspect SYN flood/unwanted traffic branch

Minimal incident note

listener port:
process:
client symptom:
counter output:
new connection rate:
request rate:
CPU/softirq:
worker state:
packet sample:
load balancer behavior:
suspected branch:
fix:
verification:

The incident is solved when handshake failures stop under the same load pattern and the listener can accept connections without rising backlog/drop counters.

References

• Linux tcp(7) manual page
• Linux listen(2) manual page
• Linux ss(8) manual page
• tcpdump manual page