ACK packets being rejected in SYNPROXY, unless there was a valid TCP connection before it

[Yadunandan Pillai]

Hi,

I have written an XDP program that intercepts ACK handshake packets, and rewrites the acknowledgement number with a randomly generated syncookie (bpf_tcp_raw_gen_syncookie_ipv4). The idea is to eventually create a custom SYNPROXY. 

I've got a TCP socket server that logs whenever "accept" finishes (connection was established). Initially, my ACK packets kept getting rejected with RST packets. However, I found that once I unlink the XDP program, complete a normal TCP handshake, and then link the XDP program again, the ACK packets are accepted.

I figured there's some issue with hanging socket states so I tried killing all TIME_WAIT, and other sockets in various close states with `ss`, but the same behavior continued.

"xdp_synproxy_kern.c" is the only relevant example I can find in the kernel source, and it does the same thing I do with ACK packets. Only difference is, the sample code generates a SYNACK directly in the XDP program, while I let the kernel do it for me and just modify the ACK. I looked through the kernel source, and I didn't see any context being added to the listening port with SYNCOOKIES enabled until the ACK packet is received (as should be the behavior, otherwise you defeat the purpose of SYNCOOKIES).
What am I missing here?