Dear all, For research purposes I need some userspace memory pages to be in uncacheable mode. I am using two different Intel architectures (Sandy Bridge and Haswell) and two different kernels (2.6.32-358 and 3.19.0-28). The non-temporal stores from Intel assembly are not a valid solution so I am programming a kernel module that gets a set of pages from user space reserved with posix_memalign (get_user_pages) and then sets them as uncacheable (I have tried set_pages_uc and set_pages_array_uc). When I use one page, the access times are not very coherent and with more than one page the module crashes (in both architectures and both kernels). I wonder if I am using the correct approach or if I have to use kernel space pages in order to work with uncacheable memory. Or if I have to remap the memory. Just in case it makes it clearer, I am attaching the relevant lines of a kernel module function that should set the pages as uncacheable. (This function is the .write of a misc device; count is treated as the number of pages). Best and Thanks, Sabela. struct page *pages; //defined outside in order to be able to set them to WB in the release function. int numpages; static ssize_t setup_memory(struct file *filp, const char __user *buf, size_t count, loff_t * ppos) { int res; struct vm_area_struct *vmas; numpages = count/4096; down_read(¤t->mm->mmap_sem); res = get_user_pages(current, current->mm, (unsigned long) buf, numpages, /* Number of pages */ 0, /* Do want to write into it */ 1, /* do force */ &pages, &vmas); up_read(¤t->mm->mmap_sem); numpages=res; if (res > 0) { set_pages_uc(pages, numpages); /* Uncached */ printk("Write: %d pages set as uncacheable\n",numpages); } else{ pr_err("Couldn't get pages to set them as UC :(\n"); return -EAGAIN; } }
Sorry, little mistake copypasting and cleaning. The pages and vma structs should look like that: struct page *pages --> struct page *pages[MAX_PAGES]; struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES]; Where MAX_PAGES is defined to 5. Sabela. 2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
Dear all,
For research purposes I need some userspace memory pages to be in uncacheable mode. I am using two different Intel architectures (Sandy Bridge and Haswell) and two different kernels (2.6.32-358 and 3.19.0-28).
The non-temporal stores from Intel assembly are not a valid solution so I am programming a kernel module that gets a set of pages from user space reserved with posix_memalign (get_user_pages) and then sets them as uncacheable (I have tried set_pages_uc and set_pages_array_uc). When I use one page, the access times are not very coherent and with more than one page the module crashes (in both architectures and both kernels).
I wonder if I am using the correct approach or if I have to use kernel space pages in order to work with uncacheable memory. Or if I have to remap the memory. Just in case it makes it clearer, I am attaching the relevant lines of a kernel module function that should set the pages as uncacheable. (This function is the .write of a misc device; count is treated as the number of pages).
Best and Thanks,
Sabela.
struct page *pages; //defined outside in order to be able to set them to WB in the release function. int numpages;
static ssize_t setup_memory(struct file *filp, const char __user *buf, size_t count, loff_t * ppos) { int res; struct vm_area_struct *vmas;
numpages = count/4096;
down_read(¤t->mm->mmap_sem); res = get_user_pages(current, current->mm, (unsigned long) buf, numpages, /* Number of pages */ 0, /* Do want to write into it */ 1, /* do force */ &pages, &vmas); up_read(¤t->mm->mmap_sem);
numpages=res;
if (res > 0) { set_pages_uc(pages, numpages); /* Uncached */ printk("Write: %d pages set as uncacheable\n",numpages); } else{ pr_err("Couldn't get pages to set them as UC :(\n"); return -EAGAIN; } }
On Fri, Sep 11, 2015 at 7:59 AM, Sabela Ramos Garea <sabelaraga@gmail.com> wrote:
Sorry, little mistake copypasting and cleaning. The pages and vma structs should look like that:
struct page *pages --> struct page *pages[MAX_PAGES]; struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
Where MAX_PAGES is defined to 5.
Sabela.
2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
Dear all,
For research purposes I need some userspace memory pages to be in uncacheable mode. I am using two different Intel architectures (Sandy Bridge and Haswell) and two different kernels (2.6.32-358 and 3.19.0-28).
The non-temporal stores from Intel assembly are not a valid solution so I am programming a kernel module that gets a set of pages from user space reserved with posix_memalign (get_user_pages) and then sets them as uncacheable (I have tried set_pages_uc and set_pages_array_uc). When I use one page, the access times are not very coherent and with more than one page the module crashes (in both architectures and both kernels).
set_pages_uc might just set the kernel virtual address mappings for those pages as uncacheable via the Page Attribute Table, but you are accessing from userspace virtual addresses. If you have the userspace virtual address of the pages you can manually twiddle the PAT bits of the page tables in your module to make the pages UC.
I wonder if I am using the correct approach or if I have to use kernel space pages in order to work with uncacheable memory. Or if I have to remap the memory. Just in case it makes it clearer, I am attaching the relevant lines of a kernel module function that should set the pages as uncacheable. (This function is the .write of a misc device; count is treated as the number of pages).
Best and Thanks,
Sabela.
struct page *pages; //defined outside in order to be able to set them to WB in the release function. int numpages;
static ssize_t setup_memory(struct file *filp, const char __user *buf, size_t count, loff_t * ppos) { int res; struct vm_area_struct *vmas;
numpages = count/4096;
down_read(¤t->mm->mmap_sem); res = get_user_pages(current, current->mm, (unsigned long) buf, numpages, /* Number of pages */ 0, /* Do want to write into it */ 1, /* do force */ &pages, &vmas); up_read(¤t->mm->mmap_sem);
numpages=res;
if (res > 0) { set_pages_uc(pages, numpages); /* Uncached */ printk("Write: %d pages set as uncacheable\n",numpages); } else{ pr_err("Couldn't get pages to set them as UC :(\n"); return -EAGAIN; } }
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Sabela, On Fri, Sep 11, 2015 at 8:29 PM, Sabela Ramos Garea <sabelaraga@gmail.com> wrote:
Sorry, little mistake copypasting and cleaning. The pages and vma structs should look like that:
struct page *pages --> struct page *pages[MAX_PAGES]; struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
Where MAX_PAGES is defined to 5.
Sabela.
2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
Dear all,
For research purposes I need some userspace memory pages to be in uncacheable mode. I am using two different Intel architectures (Sandy Bridge and Haswell) and two different kernels (2.6.32-358 and 3.19.0-28).
The non-temporal stores from Intel assembly are not a valid solution so I am programming a kernel module that gets a set of pages from user space reserved with posix_memalign (get_user_pages) and then sets them as uncacheable (I have tried set_pages_uc and set_pages_array_uc). When I use one page, the access times are not very coherent and with more than one page the module crashes (in both architectures and both kernels).
I wonder if I am using the correct approach or if I have to use kernel space pages in order to work with uncacheable memory. Or if I have to remap the memory. Just in case it makes it clearer, I am attaching the relevant lines of a kernel module function that should set the pages as uncacheable. (This function is the .write of a misc device; count is treated as the number of pages).
Best and Thanks,
Sabela.
struct page *pages; //defined outside in order to be able to set them to WB in the release function. int numpages;
static ssize_t setup_memory(struct file *filp, const char __user *buf, size_t count, loff_t * ppos) { int res; struct vm_area_struct *vmas;
shouldn't this be rounded this up?
numpages = count/4096;
down_read(¤t->mm->mmap_sem); res = get_user_pages(current, current->mm, (unsigned long) buf, numpages, /* Number of pages */ 0, /* Do want to write into it */ 1, /* do force */ &pages, &vmas); up_read(¤t->mm->mmap_sem);
numpages=res;
if (res > 0) { set_pages_uc(pages, numpages); /* Uncached */
what about high-mem pages. set_memory_uc does __pa, so perhaps that's the reason for your kernel oops?
printk("Write: %d pages set as uncacheable\n",numpages); } else{ pr_err("Couldn't get pages to set them as UC :(\n"); return -EAGAIN; } }
_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
-- ---P.K.S
Hi Pranay, 2015-09-12 3:12 GMT+02:00 Pranay Srivastava <pranjas@gmail.com>:
Hi Sabela,
On Fri, Sep 11, 2015 at 8:29 PM, Sabela Ramos Garea <sabelaraga@gmail.com> wrote:
Sorry, little mistake copypasting and cleaning. The pages and vma structs should look like that:
struct page *pages --> struct page *pages[MAX_PAGES]; struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
Where MAX_PAGES is defined to 5.
Sabela.
2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
Dear all,
For research purposes I need some userspace memory pages to be in uncacheable mode. I am using two different Intel architectures (Sandy Bridge and Haswell) and two different kernels (2.6.32-358 and 3.19.0-28).
The non-temporal stores from Intel assembly are not a valid solution so I am programming a kernel module that gets a set of pages from user space reserved with posix_memalign (get_user_pages) and then sets them as uncacheable (I have tried set_pages_uc and set_pages_array_uc). When I use one page, the access times are not very coherent and with more than one page the module crashes (in both architectures and both kernels).
I wonder if I am using the correct approach or if I have to use kernel space pages in order to work with uncacheable memory. Or if I have to remap the memory. Just in case it makes it clearer, I am attaching the relevant lines of a kernel module function that should set the pages as uncacheable. (This function is the .write of a misc device; count is treated as the number of pages).
Best and Thanks,
Sabela.
struct page *pages; //defined outside in order to be able to set them to WB in the release function. int numpages;
static ssize_t setup_memory(struct file *filp, const char __user *buf, size_t count, loff_t * ppos) { int res; struct vm_area_struct *vmas;
shouldn't this be rounded this up?
numpages = count/4096;
For the current tests I am assuming that count is multiple of 4096 and the user *buf is aligned. Anyway, isn't it safer if I just round down so I don't mess with addresses outside the range of pages that have to be set as uncached?
down_read(¤t->mm->mmap_sem); res = get_user_pages(current, current->mm, (unsigned long) buf, numpages, /* Number of pages */ 0, /* Do want to write into it */ 1, /* do force */ &pages, &vmas); up_read(¤t->mm->mmap_sem);
numpages=res;
if (res > 0) { set_pages_uc(pages, numpages); /* Uncached */
what about high-mem pages. set_memory_uc does __pa, so perhaps that's the reason for your kernel oops?
I have used kmap to map the user addresses in kernel space as follows: if (res > 0) { for(i=0; i<res; i++){ kaddress = kmap(pages[i]); set_memory_uc(kaddress,1);//userspace addresses doesn't have to be contiguous... } //set_pages_array_uc(pages, count); /* Uncached */ printk("Write: %d pages set as uncacheable\n",numpages); } But the effect in the test code (user space) that tries to measure cached vs. uncached accesses obtains lower latency for uncached pages. Accesses are performed and measure like that: CL_1 = (int *) buffer; CL_2 = (int *) (buffer+CACHELINE); //flush caches //get timestamp for(j=0;j<10;j++){ CL_2 = (int *) (buffer+CACHELINE); for (i=1; i<naccesses; i++){ *CL_1 = *CL_2+i; *CL_2 = *CL_1+i; CL_2 = (int *)((char *)CL_2+CACHELINE); } } //get timestamp I've tried to do it within the kernel space but the results are similar. Thanks, Sabela.
participants (3)
-
David Matlack -
Pranay Srivastava -
Sabela Ramos Garea