lots of connections in SYN_RECV state
Hello, I have a couple of servers, for the past few days I am seeing lots of connections in SYN_RECV state in netstat commands output. Is there a solution to this problem? Thanks and Regards Puneet
Hi Am 2014-11-06 14:27, schrieb Puneet Agarwal:
I have a couple of servers, for the past few days I am seeing lots of connections in SYN_RECV state in netstat commands output.
This *could* be the result of a SYN-flood attack on your server.
Is there a solution to this problem?
The Wikipedia page mentions some countermeasures (if it indeed is an attack and not a networking stack issue). http://en.wikipedia.org/wiki/SYN_flood Maybe you could check from which IP addresses these SYNs come from and investigate why they do not answer to the SYN-ACK request of your server. Cheers, Silvan
nslookup does not give any result on those IP addresses. So not able to figure out who is causing this. Is there a way to check the reason, why they do not answer to the SYN-ACK's? Thanks and Regards Puneet ----------------------------------------
Date: Thu, 6 Nov 2014 15:30:46 +0100 From: me@sillymon.ch To: puneet.agr@outlook.com CC: kernelnewbies@kernelnewbies.org Subject: Re: lots of connections in SYN_RECV state
Hi
Am 2014-11-06 14:27, schrieb Puneet Agarwal:
I have a couple of servers, for the past few days I am seeing lots of connections in SYN_RECV state in netstat commands output.
This *could* be the result of a SYN-flood attack on your server.
Is there a solution to this problem?
The Wikipedia page mentions some countermeasures (if it indeed is an attack and not a networking stack issue).
http://en.wikipedia.org/wiki/SYN_flood
Maybe you could check from which IP addresses these SYNs come from and investigate why they do not answer to the SYN-ACK request of your server.
Cheers,
Silvan
2014-11-06 16:15,Puneet Agarwal:
Is there a way to check the reason, why they do not answer to the SYN-ACK's?
I don't think so. After all, they just don't answer and they won't tell you why (AFAIK there is no way to ask them why either)... You could try to check for patterns in the incoming IP addresses to see from how many different places these connections are being made. I think that way it should be possible to figure out from which geographic location these problematic connections are coming from as well. What you would do with these findings I am not sure though. If these connection negatively impact the performance of your servers you should definitely look into to countermeasures mentioned in the RFC here. http://tools.ietf.org/html/rfc4987
Latest kernel provides a TCP SYN Cookie feature to defense from SYN flooding. -daveti
On Nov 6, 2014, at 11:58 PM, Silvan Jegen <me@sillymon.ch> wrote:
2014-11-06 16:15,Puneet Agarwal:
Is there a way to check the reason, why they do not answer to the SYN-ACK's?
I don't think so. After all, they just don't answer and they won't tell you why (AFAIK there is no way to ask them why either)...
You could try to check for patterns in the incoming IP addresses to see from how many different places these connections are being made. I think that way it should be possible to figure out from which geographic location these problematic connections are coming from as well. What you would do with these findings I am not sure though.
If these connection negatively impact the performance of your servers you should definitely look into to countermeasures mentioned in the RFC here.
http://tools.ietf.org/html/rfc4987
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Oops, my bad. I remember seeing sth on LWN for the 3.x kernel talking about a new feature related with TCP SYN. Thought this pretty old stuff was the one… -daveti
On Nov 8, 2014, at 12:58 AM, Valdis.Kletnieks@vt.edu wrote:
On Fri, 07 Nov 2014 23:49:35 +0800, Dave Tian said:
Latest kernel provides a TCP SYN Cookie feature to defense from SYN flooding.
If by "latest" you mean "since Andi Kleen submitted a patch for 2.1.44", back in July 1997....
I use linux kernel 2.6. I have enabled SYN cookies already. But that does not seem to solve the problem. Overall request latency is very high with these many half open connections. Thanks and Regards Puneet ----------------------------------------
Subject: Re: lots of connections in SYN_RECV state From: dave.jing.tian@gmail.com Date: Fri, 7 Nov 2014 23:49:35 +0800 CC: puneet.agr@outlook.com; kernelnewbies@kernelnewbies.org To: me@sillymon.ch
Latest kernel provides a TCP SYN Cookie feature to defense from SYN flooding.
-daveti
On Nov 6, 2014, at 11:58 PM, Silvan Jegen <me@sillymon.ch> wrote:
2014-11-06 16:15,Puneet Agarwal:
Is there a way to check the reason, why they do not answer to the SYN-ACK's?
I don't think so. After all, they just don't answer and they won't tell you why (AFAIK there is no way to ask them why either)...
You could try to check for patterns in the incoming IP addresses to see from how many different places these connections are being made. I think that way it should be possible to figure out from which geographic location these problematic connections are coming from as well. What you would do with these findings I am not sure though.
If these connection negatively impact the performance of your servers you should definitely look into to countermeasures mentioned in the RFC here.
http://tools.ietf.org/html/rfc4987
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Fri, 07 Nov 2014 23:11:26 +0530, Puneet Agarwal said:
I use linux kernel 2.6. I have enabled SYN cookies already. But that does not seem to solve the problem. Overall request latency is very high with these many half open connections.
So, out of curiosity, where are all these half open connections coming from? Are they from addresses in your local network? Outside sites that *should* be connecting? Places you've never heard and and probably *shouldn't* be connecting? (Also, if you have properly implemented syncookies, you shouldn't *have* any half-open connections. That's the whole point of syncookies....)
These connections are from outside the network, and the IP's are legitimate ones which should be connecting. I don't know whether the IP's which I could see are the real ones or spoofed ones. sysctl -a says net.ipv4.tcp_syncookies = 1 cat /proc/sys/net/ipv4/tcp_syncookies also gives 1 Isn't this sufficient to enable syncookies? Thanks and Regards Puneet ----------------------------------------
To: puneet.agr@outlook.com CC: dave.jing.tian@gmail.com; me@sillymon.ch; kernelnewbies@kernelnewbies.org Subject: Re: lots of connections in SYN_RECV state From: Valdis.Kletnieks@vt.edu Date: Fri, 7 Nov 2014 13:10:05 -0500
On Fri, 07 Nov 2014 23:11:26 +0530, Puneet Agarwal said:
I use linux kernel 2.6. I have enabled SYN cookies already. But that does not seem to solve the problem. Overall request latency is very high with these many half open connections.
So, out of curiosity, where are all these half open connections coming from? Are they from addresses in your local network? Outside sites that *should* be connecting? Places you've never heard and and probably *shouldn't* be connecting?
(Also, if you have properly implemented syncookies, you shouldn't *have* any half-open connections. That's the whole point of syncookies....)
participants (4)
-
Dave Tian -
Puneet Agarwal -
Silvan Jegen -
Valdis.Kletnieks＠vt.edu