Fwd: How to change page permission from inside the kernel?
---------- Forwarded message ---------- From: Ahmed Soliman <ahmedsoliman0x666@gmail.com> Date: 6 July 2018 at 23:56 Subject: Re: How to change page permission from inside the kernel? To: Valdis Kletnieks <valdis.kletnieks@vt.edu>
Implementing some kernel protection against subset of rootkits that manipulates kernel static data (memory pages as well as their mappings) by having them enforced by hypervisor which is KVM in our
Can you give an actual example of a case where *all* of the following are true?
I think there is a little bit confusion between the current R/O protection and what are we doing. so I will need to make some distinctions than I will answer all the questions. First here is a bunch of accronyms that I will use for convenience (pulled out KVM mmu.txt) pfn host page frame number hpa host physical address hva host virtual address gfn guest frame number gpa guest physical address gva guest virtual address To aviod confusion I will refer to the thing we are doing as ROE (read only enforcement) and it is alittle bit different from typical R/O protection in 2 aspects: - R/O is enforced by hardware but ROE is enforced by Hypervisor(still using Hardware protection but with hypervisor on top of it) - R/O can be enabled and disabled by the kernel at its own will but ROE can be enabled by the guest kernel and once enabled the hypervisor will make sure it never gets disabled again, so if even if the kernel decided to modify a paged that has ROE, it can't without a reboot. Here is a very simple description of how ROE is supposed to works (I am missing out some tiny details like remapping pages): The kernel passes a gva of a page to its hypervisor. The hypervisor converts gva to hva then places R/O on that page note that is 2 levels of paging we have the host page table with the appropriate R/O protection (including guest's own ROE requests) and the guest page table its own R/O protections that the guest can modify. There is another part that isn't designed yet to handle DMAs and registers with static content
1) It's a page that's safe to make R/O out from under the code that uses that page. Yes because the guest kernel and hypervisor will coordinate together which pages of the _guest kernel_ memory should have this protections. of course the userspace process running the hypervisor (let it be qemu or any other software) do own the page and can do whatever it wishes with it. 2) It's a kernel static data that's R/W (Hint: stuff known to be R/O is already set to R/O at boot or module load time, so if it's R/W it probably *needs* to be that...) I think this question is from the confusion between R/W and ROE what we are simply going to do is that have every R/O page that will never be R/W again to also be ROE if the kernel is compiled with the right options (we will add that too) to be a kvm guest. 3) the rootkit *is* able to screw with kernel pages, but somehow *is not* able to disable your protection (remember, all it takes is one NOP or BRANCH opcode in the right place).
I can't think of a way to do it. having a rootkit being able to manipulate kernel pages can only be done though one of these ways none of which is considered a real threat (except for the DMA part thats why we are stil learning about current DMA memory protections and limitations): - VMM escape where it can then modify the paging protections in the host allocated memory that was assigned to the guest. - Screwing the page table itself which is not static data. but parts of the details I am missing is that hypervisor will keep track of all guest page number, guest frame number of all ROE protected pages and make sure that it is never the wrong mapping when updating the TLB. - DMA and we are planning to do similar protection for DMA memory but still design in progress. other than these described above, I can't think of a way to manipulate a ROE protected page without a reboot.
On Fri, 06 Jul 2018 23:59:30 +0200, Ahmed Soliman said:
ROE can be enabled by the guest kernel and once enabled the hypervisor will make sure it never gets disabled again, so if even if the kernel decided to modify a paged that has ROE, it can't without a reboot.
So in essence, you're willing to *permanently* remove pages from the usable set in the guest? (Usable as in "able to write to it"). What happens after you've been up for 3 weeks and you're running out of usable pages? How does this interact with ballooning? What ways can malware in the guest use this to DoS or otherwise break the kernel? (Setting the page that contains the 'struct proc' for PID 1 would be amusing, and I'm sure there's plenty of amusement with race conditions to make other kernel threads fail when they encounter a page they were expecting to be R/W) Bonus points for explaining the hypervisor/guest interface needed to make this work.
Can you give an actual example of a case where *all* of the following are true? 1) It's a page that's safe to make R/O out from under the code that uses that page. 2) It's a kernel static data that's R/W (Hint: stuff known to be R/O is already set to R/O at boot or module load time, so if it's R/W it probably *needs* to be that...) 3) the rootkit *is* able to screw with kernel pages, but somehow *is not* able to disable your protection (remember, all it takes is one NOP or BRANCH opcode in the right place).
Well actually, what I was looking for was a description of the rootkit that you're defending against - where it lives, what kernel data it attacks, and so on....
I can't think of a way to do it. having a rootkit being able to manipulate kernel pages can only be done though one of these ways none of which is considered a real threat
If you're unable to explain the threat you're trying to guard against, you're going to be unable to guard against it properly. Hint: Your protection is trivially bypassed by simply *replacing* the protected page. The attacker snarfs up a R/W 4K page, copies the protected page to the new page, injects whatever malware bits they find amusing, and change the virtual memory pointer tables so this new page frame is referenced by the virtual address of the old page. If the hypervisor even notices the swap, all it will probably do is make the new page R/O - but that matters not because it's already compromised.
other than these described above, I can't think of a way to manipulate a ROE protected page without a reboot.
You missed the point - your protection can be bypassed without manipulating a ROE page.
What happens after you've been up for 3 weeks and you're running out of usable pages? That can't happen, it is my mistake missing some details, this is for only protecting Kernel Pages, Pages that are hold code or static data that is created once and assumed to be there for ever, like kernel code section as well as any static data. However this isn't to be used on any user space memory (from the guest's perspective) process because processes start and end.
How does this interact with ballooning? I haven't thought about this _yet_. Ballooning is complex, I can start simple and expand later to support all of the current KVM features, Also I am not even sure if ballooning is considered related or not because If Kernel pages can be part of the Balloon then I would that when they are copied they are copied with the right permissions and this is done completely independent of gva and gpa.
What ways can malware in the guest use this to DoS or otherwise break the kernel? (Setting the page that contains the 'struct proc' for PID 1 would be amusing, and I'm sure there's plenty of amusement with race conditions to make other kernel threads fail when they encounter a page they were expecting to be R/W)
I should state that hypercalls are done only from kernel mode (I am not sure about KVM but t is the case in Xen and if KVM allows user mode hypercall we can check for this specific one to be created from kernel mode it is easy) In this case, worrying about a if kernel code can Dos the kernel via ROE is relatively meaningless because there are already many other ways to break the system once one have kernel mode access. However If you mean that this way guest kernel can break host kernel, then this is a different story, it will not be possible because of kvm MMU is designed to make guest not being able to access memory that host never allocated to it. So we are replacing some way to hide a rootkit in a kernel by a way to crash the kernel (given that there are many ways to do both). I think in most cases it would be rather better if the guest OS crashes (thus indicating that something is wrong) rather than have a rootkit working silently.
Well actually, what I was looking for was a description of the rootkit that you're defending against - where it lives, what kernel data it attacks, and so on.... What about some real example? I just found this one from google https://github.com/nurupo/rootkit/blob/master/rootkit.c While I aggree that the rootkit will make its way into the kernel but it _will not_ be able to achieve all of its functionalities. anything that uses "_asm_hook_patch" function should fail because the system call tables is protected. I am not sure if ROE will prevent other parts of this rootkit from working or not but here is one :). But I am pretty sure that there are other static data that can be abused by rootkits inside the kernel that should rather be protected.
If you're unable to explain the threat you're trying to guard against, you're going to be unable to guard against it properly. Well I kinda had one, but you never asked for it. We assume that the attacker is running in kernel mode and his goal is to alter static data/ code section in the kernel for whatever reason with out a reboot. if reboot is needed well it can be trivially done by replacing the whole kernel with a new one including any malicious code and then faulting the running kernel thus forcing a reboot or even. Also attacks that involve dynamic data (except for the page table) is out of scope including attacks that involve any block device. We plan to handle memory either that is accessed either via CPU or DMA as well as privileged registers that are set once and never modify again. We haven't agreed on how an unauthorized write to ROE protected memory would be handled but worst case it can lead to crashing the guest OS, Crashing the Host OS is considered a big breach and it shouldn't be possible. Although this thing isn't really part of threat model but it is kinda of a motto that crash is better than rootkit. because once it happens we will know it is there and the indecent can be thoroughly inspected. Hint: Your protection is trivially bypassed by simply *replacing* the protected page. The attacker snarfs up a R/W 4K page, copies the protected page to the new page, injects whatever malware bits they find amusing, and change the virtual memory pointer tables so this new page frame is referenced by the virtual address of the old page. If the hypervisor even notices the swap, all it will probably do is make the new page R/O - but that matters not because it's already compromised.
I think I did explain how we handled this:
- Screwing the page table itself which is not static data. but parts of the details I am missing is that hypervisor will keep track of all guest page number, guest frame number of all ROE protected pages and make sure that it is never the wrong mapping when updating the TLB.
You missed the point - your protection can be bypassed without manipulating a ROE page. Changing the virtual memory pointer table is ok but again these memory mappings will never make it to the TLB and will be caught during by KVM MMU because it will explicitly check both gva and gpa when resolving a gva so perhaps this will put the guest kernel in some inconsistent state (again crashing the guest is considered much much better than a rootkit running in silence).
On Sat, 07 Jul 2018 01:31:45 +0200, Ahmed Soliman said:
You missed the point - your protection can be bypassed without manipulating a ROE page. Changing the virtual memory pointer table is ok but again these memory mappings will never make it to the TLB and will be caught during by KVM MMU because it will explicitly check both gva and gpa when resolving a gva so perhaps
Did you actually *check* that this will happen, or will KVM just say "Oh, the VM changed their mapping" and go with the new mapping? In other words, *prove* that they won't make it to the TLB - and while you're at it, deal with the case where the TLB isn't the source of the mapping (which is the case on every single TLB miss, where it has to go consult the page tables for the mapping. And given Intel chipset's over-exuberant approach to speculative execution and pipelining, there's a good chance that the instant the proper mapping is found from the page tables, it's used to make the actual memory reference, while the TLB gets updated after the fact. Plus, a lot of that happens in hardware, without KVM getting a chance to inspect things. (I don't actually know how the TLB works - but the point is you'll have to go reading the Intel and AMD processor manuals to get that part right...) While you're reading it, look for the implications of the "load/store bypassing cache" instructions - can you force a surprise by storing directly to main memory for a page table entry that is in L3 cache on the other socket? (Note that the kernel has memory barriers all over the place to avoid such surprises - but examine the converse, where malware intentionally doesn't use a barrier so there's inconsistency between cache, memory, and registers) I take it you assuming more special-case checking which will require patches to the KVM side (at a minimum, the KVM needs to keep a list of pages that should not be changed, and check that at every required point? Bonus points for properly handling the case where KVM pages an ROE page out of real memory, and then pages it back into a different page frame (unless you want to claim it works like mlock() - at which point you need to consider the implications of a guest permanently mlock()ing up to 16M or 32M or even more). You might want to stop and ask yourself why the people who did all the W^X code for the kernel didn't extend it another step and add your ROE scheme.. ;)
participants (2)
-
Ahmed Soliman -
valdis.kletnieks＠vt.edu