Ports, sockets, and telling failures apart

"It times out" and "connection refused" look equally broken and mean opposite things. One says the service is down; the other says the packet never arrived. Knowing which is most of the diagnosis.

1. Run a server and connect to it

Terminal one:

nc -l -p 9000

Terminal two:

sudo ss -tlnp 'sport = :9000'
nc 127.0.0.1 9000

Type in either terminal; it appears in the other. That is a TCP socket: a pair of address-and-port endpoints with a byte stream between them. Ctrl-C to close, and notice the other end sees the close immediately.

2. Bind address decides who can reach it

nc -l 127.0.0.1 9001 &
sleep 1
sudo ss -tlnp 'sport = :9001'
nc -z -w2 127.0.0.1 9001; echo "via loopback: exit=$?"
MYIP=$(ip route get 1.1.1.1 | grep -oP 'src \K\S+')
nc -z -w2 "$MYIP" 9001; echo "via real address: exit=$?"
kill %1 2>/dev/null

Bound to 127.0.0.1, it answers on loopback and refuses on the machine's real address. Now the opposite:

nc -l 0.0.0.0 9002 &
sleep 1
nc -z -w2 "$MYIP" 9002; echo "via real address: exit=$?"
kill %1 2>/dev/null

0.0.0.0 means every address. This is the single most common cause of "works locally, times out from anywhere else" — and it is a one-word config change, not a firewall problem.

3. The three failure modes

Refused — the packet arrived, nothing was listening:

time nc -z -w5 127.0.0.1 9099; echo "exit=$?"

Immediate. Routing and firewall are fine; the service is down or on a different port.

Filtered — something dropped the packet silently:

time nc -z -w5 10.255.255.1 9099; echo "exit=$?"

Hangs for the full timeout. That is a firewall, a security group, or a missing route — not the service.

Connected but silent — the handshake completes and nothing replies. That is a live socket with a wedged application behind it, and it is the one that fools health checks that only test whether the port opens.

4. Watch the handshake

nc -l 0.0.0.0 9003 &
sudo timeout 8 tcpdump -n -i lo "tcp port 9003" &
sleep 1
nc -z -w2 127.0.0.1 9003
wait
kill %1 2>/dev/null

Read the flags. [S] is the client's SYN, [S.] is the server's SYN-ACK, [.] is the client's ACK — the three-way handshake. Then [F.] or [R] closing it.

What you see tells you where it stopped:

• SYN with no reply → dropped outbound, or the reply is dropped
• SYN then [R] (reset) → actively refused
• Full handshake then silence → the network is fine, the application is not answering

5. Which process owns a port

sudo ss -tlnp | head -n 8
sudo lsof -i :22 2>/dev/null | head -n 3
sudo fuser -n tcp 22 2>/dev/null

Three routes to the same answer, and you will meet all three because different boxes have different tools installed. This is what you run against "address already in use" — something is holding the port, and these name it.

6. Well-known ports worth knowing by heart

Ports below 1024 require root to bind, which is why a web server starts as root and drops privileges, and why a container running as non-root cannot listen on 80 without a capability or a port mapping.

Clean up

pkill -f "nc -l" 2>/dev/null; echo "listeners stopped"

Where this goes next

You can reach a port and classify a failure. Next: building a network between two hosts on one machine, so routing stops being abstract.