I was using memset on a reserved area of memory (64bit x86 kernel and system), and noticed that as soon as I exceed a size of 2GB, the function becomes extremely slow, e.g just below 2GB it takes typically about 0.3s, and just above 2GB is takes about 39s to complete... I tried tracing the eventual function that is called in the kernel, and I think it resolves to the below (even on x86_64 if I'm not mistaken): static inline void *__memset_generic(void *s, char c, size_t count) { int d0, d1; asm volatile("rep\n\t" "stosb" : "=&c" (d0), "=&D" (d1) : "a" (c), "1" (s), "0" (count) : "memory"); return s; } size_t is defined as (unsigned long) on my platform, but I suspect the d0 and d1 variables above cause problems because they are int... Is this a kernel bug, or known limitation, or what?
I was wrong about the 2GB size and memset, neither this size nor the use of memset is the important factor here. The slowness really comes in with any access to memory returned from an ioremap_cache() call which crosses a physical 4GB memory boundary... I can map 4GB at offset 4GB, and memset that, and it is relatively quick, but if I map 2GB at offset 3GB, then memset even just the first 1GB of that is extremely slow. Mapping 2GB at 2GB is fast again. Any ideas? Obviously the magical 4GB is the limit of a 32bit value, but why should that matter on a 64bit CPU and 64bit kernel? This shouldn't trigger PAE should it? This is on stock 64bit Ubuntu 10.04.1 LTS kernel : # # uname -srvmpio Linux 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:09:38 UTC 2010 x86_64 unknown unknown GNU/Linux from /proc/cpuinfo: model name : Intel(R) Xeon(R) CPU E5645 @ 2.40GHz On 01 Feb 2011, at 12:23 PM, Jason Nymble wrote:
I was using memset on a reserved area of memory (64bit x86 kernel and system), and noticed that as soon as I exceed a size of 2GB, the function becomes extremely slow, e.g just below 2GB it takes typically about 0.3s, and just above 2GB is takes about 39s to complete...
I tried tracing the eventual function that is called in the kernel, and I think it resolves to the below (even on x86_64 if I'm not mistaken): static inline void *__memset_generic(void *s, char c, size_t count) { int d0, d1; asm volatile("rep\n\t" "stosb" : "=&c" (d0), "=&D" (d1) : "a" (c), "1" (s), "0" (count) : "memory"); return s; }
size_t is defined as (unsigned long) on my platform, but I suspect the d0 and d1 variables above cause problems because they are int... Is this a kernel bug, or known limitation, or what?
On Tue, Feb 1, 2011 at 3:53 PM, Jason Nymble <jason.nymble@gmail.com> wrote:
I was using memset on a reserved area of memory (64bit x86 kernel and system), and noticed that as soon as I exceed a size of 2GB, the function becomes extremely slow, e.g just below 2GB it takes typically about 0.3s, and just above 2GB is takes about 39s to complete...
I tried tracing the eventual function that is called in the kernel, and I think it resolves to the below (even on x86_64 if I'm not mistaken): static inline void *__memset_generic(void *s, char c, size_t count) { int d0, d1; asm volatile("rep\n\t" "stosb" : "=&c" (d0), "=&D" (d1) : "a" (c), "1" (s), "0" (count) : "memory"); return s; }
size_t is defined as (unsigned long) on my platform, but I suspect the d0 and d1 variables above cause problems because they are int... Is this a kernel bug, or known limitation, or what? _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Jason, How did you allocate the memory? What is the size of physical memory that is attached to your machine ? -Vinit
On 01 Feb 2011, at 2:48 PM, vinit dhatrak wrote:
On Tue, Feb 1, 2011 at 3:53 PM, Jason Nymble <jason.nymble@gmail.com> wrote: I was using memset on a reserved area of memory (64bit x86 kernel and system), and noticed that as soon as I exceed a size of 2GB, the function becomes extremely slow, e.g just below 2GB it takes typically about 0.3s, and just above 2GB is takes about 39s to complete...
I tried tracing the eventual function that is called in the kernel, and I think it resolves to the below (even on x86_64 if I'm not mistaken): static inline void *__memset_generic(void *s, char c, size_t count) { int d0, d1; asm volatile("rep\n\t" "stosb" : "=&c" (d0), "=&D" (d1) : "a" (c), "1" (s), "0" (count) : "memory"); return s; }
size_t is defined as (unsigned long) on my platform, but I suspect the d0 and d1 variables above cause problems because they are int... Is this a kernel bug, or known limitation, or what? _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi Jason,
How did you allocate the memory? What is the size of physical memory that is attached to your machine ?
The machine has 48GB of physical memory, and I reserve a chunk as a kernel param with the memmap=nn[KMG]$ss[KMG] syntax (see Documentation/kernel-parameters.txt), e.g. memmap=8G$4G to reserve an 8GB chunk of mem starting at offset 4GB. Then in the driver I simply use ioremap_cache() with the physical base offset and size I require to get a kernel virtual address. If I do that across any 4GB physical boundary, then access to that memory via that kernel virtual address pointer is extremely slow.
Hi Jason... On Tue, Feb 1, 2011 at 20:23, Jason Nymble <jason.nymble@gmail.com> wrote:
The machine has 48GB of physical memory, and I reserve a chunk as a kernel param with the memmap=nn[KMG]$ss[KMG] syntax (see Documentation/kernel-parameters.txt), e.g. memmap=8G$4G to reserve an 8GB chunk of mem starting at offset 4GB. Then in the driver I simply use ioremap_cache() with the physical base offset and size I require to get a kernel virtual address. If I do that across any 4GB physical boundary, then access to that memory via that kernel virtual address pointer is extremely slow.
I think somehow somewhere the kernel repeatedly do kmap() / kunmap() when you do memory access...and since it's x86_64 (not true 64 bit like Itanium), it's not really 64 bit... -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
participants (3)
-
Jason Nymble -
Mulyadi Santosa -
vinit dhatrak