<meta http-equiv="Content-Type" content="text/html; charset=GB18030"><div><p cid="n0" mdtype="paragraph" class="md-end-block md-p" style=" box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space-collapse: preserve; position: relative; color: rgb(51, 51, 51) ; ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain md-expand" style="box-sizing: border-box;"> I’m encountering an issue during a high-concurrency short-lived connection stress test in a distributed database system. The system under test is running on a single machine. The server sometimes </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">sends a TCP RST after completing the three-way handshake</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">, and while the </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">client receives the RST and closes the connection</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">, the </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">server-side socket remains in ESTABLISHED state</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">.</span></p><h3 cid="n3" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">System Information</span></h3><pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n4" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; break-inside: avoid; white-space-collapse: collapse; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); position: relative !important;"><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">Linux systest104 5.15.0-305.176.4.el9uek.x86_64 #2 SMP Tue Jan 28 20:15:04 PST 2025 x86_64 x86_64 x86_64 GNU/Linux</span></pre><h3 cid="n5" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Problem Description</span></h3><p cid="n6" mdtype="paragraph" class="md-end-block md-p" style=" box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space-collapse: preserve; position: relative; color: rgb(51, 51, 51) ; ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">During the short-link stress test (many TCP connections being rapidly established and closed), the server (PostgreSQL) sends an unexpected </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">RST</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> after the handshake. Despite the RST and client-side closure, the </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">server socket stays in ESTABLISHED</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">. This behavior repeats under high load.</span></p><h3 cid="n7" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="code" spellcheck="false" style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: var(--monospace); vertical-align: initial; border: 1px solid rgb(231, 234, 237); background-color: rgb(243, 244, 244); border-radius: 3px; padding: 0px 2px; font-size: inherit;">netstat</code></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> Output</span></h3><pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n8" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; break-inside: avoid; white-space-collapse: collapse; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); position: relative !important;"><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">[root@systest104 tools]# netstat -atpn | grep 45129 | grep LI</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">tcp 0 0 10.13.8.104:45129 0.0.0.0:* LISTEN 3961360/postgres:</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">tcp 0 0 10.13.8.104:45129 10.13.8.104:45052 ESTABLISHED 3961360/postgres:</span></pre><h3 cid="n9" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">TCP Traffic Capture (tcpdump)</span></h3><h4 cid="n10" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.25em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.4; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">From server (port 45129) to client (port 45052):</span></h4><pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n11" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; break-inside: avoid; white-space-collapse: collapse; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); position: relative !important;"><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:03.972859 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45129 > 10.13.8.104.45052: Flags [S.], cksum 0x2518 (incorrect -> 0xe388), seq 175553476, ack 2894832976, win 65535, options [mss 65495,sackOK,TS val 1670382316 ecr 1670382316,nop,wscale 11], length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:04.218997 IP (tos 0x0, ttl 64, id 9377, offset 0, flags [DF], proto TCP (6), length 52)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45129 > 10.13.8.104.45052: Flags [.], cksum 0x2510 (incorrect -> 0x0a6e), seq 1, ack 42, win 32, options [nop,nop,TS val 1670382564 ecr 1670382522], length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:04.979788 IP (tos 0x0, ttl 64, id 9378, offset 0, flags [DF], proto TCP (6), length 52)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45129 > 10.13.8.104.45052: Flags [F.], cksum 0x2510 (incorrect -> 0x0776), seq 1, ack 42, win 32, options [nop,nop,TS val 1670383323 ecr 1670382522], length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:06.494295 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45129 > 10.13.8.104.45052: Flags [S.], cksum 0x2518 (incorrect -> 0x7ac8), seq 214961739, ack 2894898555, win 65535, options [mss 65495, sackOK, TS val 1670384838 ecr 1670384838,nop,wscale 11], length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:06.497830 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45129 > 10.13.8.104.45052: Flags [R], cksum 0x0f95 (correct), seq 214961740, win 0, length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;"><span cm-text="" style="box-sizing: border-box;">​</span></span></pre><h4 cid="n12" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.25em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.4; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">From client (port 45052) to server (port 45129):</span></h4><pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n13" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; break-inside: avoid; white-space-collapse: collapse; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); position: relative !important;"><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45052 > 10.13.8.104.45129: Flags [S], cksum 0x2518 (incorrect -> 0x9bdf), seq 2894898554, win 65535, options [mss 65495,sackOK,TS val 1670384838 ecr 1670383323,nop,wscale 11], length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:06.496318 IP (tos 0x0, ttl 64, id 15240, offset 0, flags [DF], proto TCP (6), length 52)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45052 > 10.13.8.104.45129: Flags [.], cksum 0x2510 (incorrect -> 0xa39b), seq 65579, ack 39408264, win 32, options [nop,nop,TS val 1670384839 ecr 1670384838], length 0</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">02:58:06.496321 IP (tos 0x0, ttl 64, id 15241, offset 0, flags [DF], proto TCP (6), length 84)</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">10.13.8.104.45052 > 10.13.8.104.45129: Flags [P.], cksum 0x2530 (incorrect -> 0x2b50), seq 65579:65611, ack 39408264, win 32, options [nop,nop,TS val 1670384839ecr 1670384838], length 32</span></pre><h3 cid="n14" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Kernel Socket State Transitions</span></h3><pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n15" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; break-inside: avoid; white-space-collapse: collapse; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); position: relative !important;"><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">ffff9de27cb78000 3499722 postgres   10.13.8.104 45052 → 45129   SYN_RECV → ESTABLISHED 0.003s</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">ffff9dd18eb25580 3961360 postgres   10.13.8.104 45129 → 45052   SYN_SENT → ESTABLISHED 1.165s</span><br><span role="presentation" style="box-sizing: border-box; padding-right: 0.1px;">ffff9dd18eb25580 353      ksoftirqd 10.13.8.104 45129 → 45052   ESTABLISHED → CLOSE    4.154s</span></pre><h3 cid="n16" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Summary</span></h3><ul class="ul-list" cid="n17" mdtype="list" data-mark="*" style=" box-sizing: border-box; margin: 0.8em 0px; padding-left: 30px; position: relative; color: rgb(51, 51, 51) ; ; ; ; ; ; ; ; "><li class="md-list-item" cid="n18" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n19" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space-collapse: preserve; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">The client (45052) initiates a connection to the server (45129).</span></p></li><li class="md-list-item" cid="n20" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n21" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space-collapse: preserve; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">The connection completes (ESTABLISHED).</span></p></li><li class="md-list-item" cid="n22" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n23" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space-collapse: preserve; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Then a </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">new SYN is seen</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">, followed shortly by an </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">RST from the server</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">.</span></p></li><li class="md-list-item" cid="n24" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n25" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space-collapse: preserve; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">However, the server socket does </span><span md-inline="strong" class="" style="box-sizing: border-box;"><strong style="box-sizing: border-box;">not transition out of ESTABLISHED</strong></span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"> despite sending the RST.</span></p></li><li class="md-list-item" cid="n26" mdtype="list_item" style="box-sizing: border-box; margin: 0px; position: relative;"><p cid="n27" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0px 0px 0.5rem; white-space-collapse: preserve; position: relative;"><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">The connection appears "stuck" in the ESTABLISHED state from the server’s perspective.</span></p></li></ul><h3 cid="n28" mdtype="heading" class="md-end-block md-heading" style=" box-sizing: border-box; break-after: avoid-page; break-inside: avoid; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; line-height: 1.43; cursor: text; white-space-collapse: preserve; color: rgb(51, 51, 51) ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Question</span></h3><p cid="n29" mdtype="paragraph" class="md-end-block md-p" style=" box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space-collapse: preserve; position: relative; color: rgb(51, 51, 51) ; ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">What could cause the kernel to send an RST but still leave the server-side socket in ESTABLISHED state? Shouldn’t the kernel remove the connection after sending the RST?</span></p><p cid="n30" mdtype="paragraph" class="md-end-block md-p" style=" box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space-collapse: preserve; position: relative; color: rgb(51, 51, 51) ; ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Any insight into what may be going wrong here (socket leak, improper closure, application issue, kernel bug) would be appreciated.</span></p><p cid="n31" mdtype="paragraph" class="md-end-block md-p" style=" box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space-collapse: preserve; position: relative; color: rgb(51, 51, 51) ; ; ; ; ; ; ; ; "><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">Thanks,</span><span md-inline="softbreak" class="md-softbreak" style="box-sizing: border-box;">
</span><span md-inline="plain" class="md-plain" style="box-sizing: border-box;">  ZHAO</span></p></div><div><span md-inline="plain" class="md-plain" style="box-sizing: border-box;"><br></span></div><div></div>