<p dir="ltr">So, I ran perf on my host and it came back far more true. The top consumers of time were all atomics and some function called sse3, which I believe is a super fast memcpy implementation provided the the arch. In addition, all the highest time consumers are within my image- it stayed out of the kernel as designed and it used additional extensions and features.</p>
<p dir="ltr">I just thought of something-what if there is some kind of page size difference between my host and my Linux kernel causing the performance problems?</p>
<p dir="ltr">On Nov 24, 2016 11:33 AM, "Kenneth Adam Miller" <<a href="mailto:kennethadammiller@gmail.com">kennethadammiller@gmail.com</a>> wrote:<br>
><br>
> On Thu, Nov 24, 2016 at 11:13 AM, Greg KH <<a href="mailto:greg@kroah.com">greg@kroah.com</a>> wrote:<br>
> > On Thu, Nov 24, 2016 at 10:31:18AM -0500, Kenneth Adam Miller wrote:<br>
> >> On Nov 24, 2016 2:18 AM, "Greg KH" <<a href="mailto:greg@kroah.com">greg@kroah.com</a>> wrote:<br>
> >> ><br>
> >> > On Thu, Nov 24, 2016 at 02:01:41AM -0500, Kenneth Adam Miller wrote:<br>
> >> > > Hello,<br>
> >> > ><br>
> >> > ><br>
> >> > > I have a scheduler issue in two different respects:<br>
> >> > ><br>
> >> > > 1) I have a process that is supposed to tight loop, and it is being<br>
> >> > > given very very little time on the system. I don't want that - I want<br>
> >> > > those who would use the processor to be given the resources to run as<br>
> >> > > fast as they each can.<br>
> >> ><br>
> >> > What is causing it to give up its timeslice? Is it waiting for I/O?<br>
> >> > Doing something else to sleep?<br>
> >><br>
> >> It's multithreaded, so it reads in a loop in one thread and writes in<br>
> >> another thread. What I saw when I ran strace on it is each process<br>
> >> would run for too long- the program is designed to try and stay out of<br>
> >> the kernel on each side, so it checks some shared variables before it<br>
> >> ever goes.<br>
> ><br>
> > So locking/cpu contention for those "shared variables" perhaps?<br>
><br>
> I don't think that could possibly be it, because the shared variables<br>
> are controlled by atomics. It's just some memory operation to check to<br>
> see if it needs to go to the kernel, as in is there more data in the<br>
> shm region for me to read? If not, I'll go wait on this OS semaphore.<br>
> It's lightening fast on my host machine.<br>
><br>
> ><br>
> >> > > 2) I am seeing with perf that the maximum overhead at each section<br>
> >> > > does not sum up to be more than 15 percent. Total, probably something<br>
> >> > > like 18% of cpu time is used, and my binary has rocketed in slowness<br>
> >> > > from about 2 seconds or less total to several minutes.<br>
> >> ><br>
> >> > What changed to make things slower? Did you change kernel versions or<br>
> >> > did you change something in your userspace program?<br>
> >> ><br>
> >><br>
> >> The kernel versions specifically couldnt have anything to do with it<br>
> >> but it was different kernels. The test runs in less that 2 seconds on<br>
> >> my host. When I copy it to our custom linux, it takes minutes for it<br>
> >> to run. I think it's some extra setting that we're missing while<br>
> >> building the kernel, and I don't know what that is. I got a huge<br>
> >> improvement when I changed the multicore scheduling to allow<br>
> >> preemption "(desktop)" but there's still a problem as I've described<br>
> >> with one of the processes not using the core as it should.<br>
> ><br>
> > What do you mean by "custom linux"? Is this the exact same hardware as<br>
> > your machine? Or different? If so, what is different? What is<br>
> > different between the different kernel versions you are using? Does the<br>
> > perf output look different from running on the two different machines?<br>
> > If so, where?<br>
><br>
> I am building with buildroot a linux that is meant to be really<br>
> stripped down and only have the things we want. In my case, the what<br>
> the bzImage sees is either what QEMU gives it or what it sees in our<br>
> dedicated hardware, with is just off the shelf i7 and other stuff you<br>
> get a market - nothing custom in the sense you are thinking. Custom as<br>
> in, roll your own linux.<br>
><br>
> The kernel versions between my host and the target are 3.13.x and<br>
> 3.14.5x; they don't change so much, and certainly don't affect<br>
> performance on their own. I'm missing some setting or something with<br>
> how I'm configuring or building linux.<br>
><br>
> I haven't had a chance to run perf on my host. I can't find what<br>
> ubuntu package it is just yet, but I will search for it in a minute. I<br>
> have to go somewhere and will be right back immediately.<br>
><br>
> ><br>
> > Have you changed the priority levels of your application at all? Have<br>
> > you thought about just forcing your app to a specific CPU and getting<br>
> > the kernel off of that CPU in order so that the kernel isn't even an<br>
> > option here at all (Linux allows you to do this, details are somewhere<br>
> > in the documentation, sorry, can't remember off the top of my head...)<br>
> ><br>
><br>
> No, that may be it or help though. I thought that binding an<br>
> application to a particular cpu had something to do with affinity and<br>
> that there was some C api for it or something. That would work for our<br>
> particular scenario, and we've even talked about it, I just don't know<br>
> how to do it yet.<br>
><br>
> > But really, you should track down what the differences are between your<br>
> > two machines/environments, as something is different that is causing the<br>
> > slow down.<br>
><br>
> True - the kernel configuration is most suspect based on everything I<br>
> know. The hardware differences between my host to the target we're<br>
> building for is each modern, and well supported by linux. I'm thinking<br>
> it absolutely must have something to do with the way I've built linux.<br>
><br>
> ><br>
> > You haven't even said what kernel version you are using, and if you have<br>
> > any of your own kernel patches in those kernels.<br>
> ><br>
><br>
> For the target hardware is 3.14.5x, and there aren't any kernel<br>
> patches at this time; I've disabled grsec while in the process of<br>
> narrowing down what the problem is.<br>
><br>
> >> > > I think that<br>
> >> > > the linux scheduler isn't scheduling it, because this process is just<br>
> >> > > some unit tests that double as benchmarks in that they shm_open a file<br>
> >> > > and write into it with memcpy's.<br>
> >> ><br>
> >> > Are you sure that I/O isn't happening here like through swap or<br>
> >> > something else?<br>
> >> ><br>
> >><br>
> >> Well, we're using tmpfs and don't have a disk in the machine, but I<br>
> >> will say this process is using all lot of the address space. One<br>
> >> problem here is that the kernel has more ram than it thinks it does,<br>
> ><br>
> > What do you mean, is this a hardware issue?<br>
><br>
> I don't think it's hardware; we're using this proprietary software<br>
> beneath the linux kernel, but it's still ram of course. I can't say<br>
> too too much, but what I can say is that while how much linux thinks<br>
> it has could be affecting how it behaves, on our end we have the<br>
> resources and can just change the configuration to make sure that<br>
> linux sees and has enough ram. So that we can test on our end, and<br>
> indeed we will.<br>
><br>
> ><br>
> >> but what I want to emphasize is that I haven't changed the program to<br>
> >> allocate any more than it was previously. I'm not sure if that's a<br>
> >> kernel change or some setting, but it went from 85% to 98%.<br>
> ><br>
> > What exactly went up by 17%?<br>
><br>
> Consider the process that I was talking about that is meant to tight<br>
> loop and burn on a core to be the "end product process". This is<br>
> different from the test benchmarks that I was explaining run so<br>
> poorly.<br>
><br>
> ><br>
> >> The reason<br>
> >> why is that there is a large latency even without that big program in<br>
> >> there; I can't run my standalone tests in qemu without it also taking<br>
> >> minutes. I understand qemu has to emulate, and that's its not just a<br>
> >> VM, but I'm going from host CPU to guest, and the settings are the<br>
> >> same.<br>
> ><br>
> > That doesn't really make much sense, why is qemu even in the picture<br>
> > here? And no, qemu doesn't always emulate things, that depends on the<br>
> > hardware you are running it on, and what type of image you are running<br>
> > on it.<br>
><br>
> Well, when I'm not at work, I have to be able to run the bzImage on<br>
> something, and I don't have a dedicated machine. So I run it in QEMU.<br>
><br>
> ><br>
> >> > What does perf say is taking all of your time?<br>
> >><br>
> >> When I ran perf what it appeared to indicate is that the largest<br>
> >> consumer of time was my library, which should be right in either<br>
> >> scenario because it should use stay out of the kernel as I've designed<br>
> >> it. In addition, the work takes place there anyway, so that's right.<br>
> >> What's not right is the fact that the largest percent of time used is<br>
> >> around 15%, and all the others combined don't add up to anything near<br>
> >> 100.<br>
> ><br>
> > So perhaps you have other processes running on the machine that you are<br>
> > not noticing that is taking up the time slices? Are you _sure_ nothing<br>
> > else is running?<br>
><br>
> I'm certain that there are other processes alive, but they are not<br>
> using the CPU. This process is the only one running. I even gave qemu<br>
> "-smp 4" because I want it to behave as close as possible to what it<br>
> would if it were just on the host.<br>
><br>
> ><br>
> > Basically, you have a bunch of variables, and haven't been very specific<br>
> > with what really is changing, or even being used here, so there's not<br>
> > much specific that I can think of at the moment.<br>
> ><br>
> > thanks,<br>
> ><br>
> > greg k-h<br></p>