In fact, it's being set to 0xff exactly, not just the VM_EXEC flag being set. vma->vm_flags & VM_EXEC resolves true, because vma->vm_flags is 0xff On Thu, Jan 14, 2016 at 11:04 AM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote:
I have a custom drive and userland program pair that I'm using for a very special use case at my workplace where we are mapping specific physical address ranges into userland memory with a mmap callback. Everything works together well with a C userland program that calls into our driver's ioctl and mmap definitions, but for our case we are using an alternative systems language just for the userland program. That mmap call is failing (properly as we want) out from the driver's mmap implementation due to the fact that the vm_flags have the VM_EXEC flag set. We do not want users to be able to map the memory range as executable, so the driver should check for this as it does. The issue is in the fact that somewhere between where mmap is called and when the parameters are given to the driver, the vma->vm_flags are being set to 255. I've manually checked the values being given to the mmap call in our non-C binary, and they are *equivalent* in value to that of the C program.
My question is, is there anything that can cause the vma->vm_flags to be changed in the trip between when the user land program calls mmap and when control is delivered to the mmap callback?