thread polling for timestamp on socket's error queue doesn't wake up
I've notice that polling for timestamp in socket's error queue from a thread that hasn't sended the packet results in not waking the polling thread when an event arrives. A simple proof of that is the following code. When compiled with SEND_IN_SAME_THREAD defined, the program will poll on the same thread it has sent the packet. Otherwise the waiter thread polls and the main thread send packets. Please first look in the two SEND_IN_SAME_THREAD preprocessor conditionals before check the rest of the code. Is this the expected behavior? Should I block waiting for packet to be sent while I could send more packets? kernel documentation about network timestamping (see section 1.3): https://www.kernel.org/doc/Documentation/networking/timestamping.txt kernel version: 4.12.14 glibc version: 2.24 /* * 10/11/2017 * polling for timestamp on socket's error queue */ #include <arpa/inet.h> /* hton*() */ #include <poll.h> /* poll() */ #include <pthread.h> /* pthread_*() */ #include <string.h> /* memset() */ #include <sys/types.h> /* socket() recv() */ #include <sys/socket.h> /* socket() recv() */ #include <unistd.h> /* close() */ #include <linux/net_tstamp.h> /* timestamp stuff */ #define PORT 8080 static int do_poll(int sfd) { struct pollfd pfd; /* prepare */ memset(&pfd, 0, sizeof(pfd)); pfd.fd = sfd; pfd.events = POLLERR; /* poll */ return poll(&pfd, 1, -1); } static void do_recv(int sfd) { char tmp; while (recv(sfd, &tmp, sizeof(tmp), MSG_ERRQUEUE) == sizeof(tmp)); } static void do_send(int sfd) { struct sockaddr_in saddr; char tmp = 'a'; /* prepare */ saddr.sin_family = AF_INET; saddr.sin_port = htons(PORT); saddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); sendto(sfd, &tmp, sizeof(tmp), 0, (struct sockaddr*) &saddr, sizeof(saddr)); sleep(1); } static void* waiter(void *data) { int sfd = *((int*) data); for (;;) { #ifdef SEND_IN_SAME_THREAD do_send(sfd); #endif do_poll(sfd); do_recv(sfd); } return NULL; } static int set_ts_opt(int sfd) { unsigned int opt; /* set timestamp option */ opt = SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_TX_SCHED | SOF_TIMESTAMPING_OPT_CMSG; if (setsockopt(sfd, SOL_SOCKET, SO_TIMESTAMPING, (char*) &opt, sizeof(opt)) == -1) return -1; return 0; } static int open_socket(void) { int sfd; int tmp; /* open socket in non-blocking mode */ sfd = socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, 0); if (sfd == -1) return -1; tmp = set_ts_opt(sfd); if (tmp == -1) goto _go_close_socket; return sfd; _go_close_socket: close(sfd); return -1; } int main(int argc, char **argv) { pthread_t waiter_thread; int sfd; if ((sfd = open_socket()) == -1) return 1; /* create thread and wait for it to terminate */ pthread_create(&waiter_thread, NULL, waiter, &sfd); #ifndef SEND_IN_SAME_THREAD for (;;) do_send(sfd); #endif pthread_join(waiter_thread, NULL); return 0; } Cheers! -- Ricardo Biehl Pasquali
On Fri, 10 Nov 2017 14:39:00 -0200, Ricardo Biehl said:
I've notice that polling for timestamp in socket's error queue from a thread that hasn't sended the packet results in not waking the polling thread when an event arrives.
That seems reasonable. Under what conditions would you want to receive an event for a packet that you haven't actually sent? And how would an event related to a non-sent packet even be generated? Or are you referring to sending a packet from one thread, generating an event that is then being received by a different thread? And you're surprised that having received it in the other thread, it's no longer available in the first thread? How would you solve that, other than keeping a near-infinite queue of events "just in case another thread wanted it"? (Although this sounds like a "well, don't do that, then" type of problem...)
2017-11-10 15:29 GMT-02:00, valdis.kletnieks@vt.edu <valdis.kletnieks@vt.edu>:
That seems reasonable. Under what conditions would you want to receive an event for a packet that you haven't actually sent? And how would an event related to a non-sent packet even be generated?
Yes, I've sent the packet :-) and I want the event (the timestamp) to be received in another thread because the thread which has sent the packet wants to send more packets while the kernel flushes the send queue.
Or are you referring to sending a packet from one thread, generating an event that is then being received by a different thread? And you're surprised that having received it in the other thread, it's no longer available in the first thread? How would you solve that, other than keeping a near-infinite queue of events "just in case another thread wanted it"? (Although this sounds like a "well, don't do that, then" type of problem...)
Only one thread is polling. Look at the code I've sent. Compile without SEND_IN_SAME_THREAD, use strace, and check if the poll() call returns in thread B after the packet is sent in thread A. Here it doesn't. However, when I compile with SEND_IN_SAME_THREAD the poll() returns on every packet sent. -- Ricardo Biehl Pasquali
Hello David! Could you take a look at the issue I've commented on this thread? Cheers! -- Ricardo Biehl Pasquali
From: Ricardo Biehl <rbpoficial@gmail.com> Date: Tue, 14 Nov 2017 13:26:55 -0200
Hello David!
Could you take a look at the issue I've commented on this thread?
No, sorry, I do not have the time as I am overloaded with merge window and stable submission work. You can ask anyone on the netdev mailing list for help on any networking topic.
2017-11-10 14:39 GMT-02:00, Ricardo Biehl <rbpoficial@gmail.com>:
I've notice that polling for timestamp in socket's error queue from a thread that hasn't sended the packet results in not waking the polling thread when an event arrives.
Not sure my statement was completely true because there is at least one case where poll() returns. It returns POLLERR when the event requested is POLLIN. However it doesn't return when event requested is POLLERR. P.S.: I know POLLERR is ignored. I use it just to make the code clearer :-) Some strace ... Let's say thread A send packets and thread B poll() on socket's error queue for timestamps of packets thread A has sent. When thread B poll with pfd.events = POLLERR poll() returns in first call, but in following calls it simply doesn't return: [thread A] sendto(3, "a", 1, 0, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...> [thread B] <... set_robust_list resumed> ) = 0 [thread A] <... sendto resumed> ) = 1 [thread B] poll([{fd=3, events=POLLERR}], 1, -1 <unfinished ...> [thread A] nanosleep({tv_sec=1, tv_nsec=0}, <unfinished ...> [thread B] <... poll resumed> ) = 1 ([{fd=3, revents=POLLERR}]) [thread B] recvfrom(3, "\0", 1, MSG_ERRQUEUE, NULL, NULL) = 1 [thread B] recvfrom(3, 0x7fa3de98ef1f, 1, MSG_ERRQUEUE, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) [thread B] poll([{fd=3, events=POLLERR}], 1, -1 <unfinished ...> [thread A] <... nanosleep resumed> 0x7ffec7cc2970) = 0 [thread A] sendto(3, "a", 1, 0, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 1 [thread A] nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffec7cc2970) = 0 [thread A] sendto(3, "a", 1, 0, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 1 [thread A] nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffec7cc2970) = 0 ---------------------------- When pfd.events = POLLIN we wake up after every packet thread A has sent: [thread A] sendto(3, "a", 1, 0, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...> [thread B] <... set_robust_list resumed> ) = 0 [thread A] <... sendto resumed> ) = 1 [thread B] poll([{fd=3, events=POLLIN}], 1, -1 <unfinished ...> [thread A] nanosleep({tv_sec=1, tv_nsec=0}, <unfinished ...> [thread B] <... poll resumed> ) = 1 ([{fd=3, revents=POLLERR}]) [thread B] recvfrom(3, "\0", 1, MSG_ERRQUEUE, NULL, NULL) = 1 [thread B] recvfrom(3, 0x7f7ac2370f1f, 1, MSG_ERRQUEUE, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable) [thread B] poll([{fd=3, events=POLLIN}], 1, -1 <unfinished ...> [thread A] <... nanosleep resumed> 0x7ffe56cfaa60) = 0 [thread A] sendto(3, "a", 1, 0, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...> [thread B] <... poll resumed> ) = 1 ([{fd=3, revents=POLLERR}]) [thread A] <... sendto resumed> ) = 1 ---------------------------- -- Ricardo Biehl Pasquali
participants (3)
-
David Miller -
Ricardo Biehl -
valdis.kletnieks＠vt.edu