Hi All, I am trying to understand how vmalloc memory is reserved in kernel. In 3.4+ latest kernel, the default vmalloc size is 240MB. Is this a carveout from the 1GiB memory that kernel has? In other words can I do a __pa(VMALLOC_START) or __pa(VMALLOC_END) or __pa(highmemory) irrespective of the vmalloc size that I request, say doing vmalloc=1G? Thank you and Regards Subbu
Hi.... :) On Sun, Jun 24, 2012 at 1:32 AM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote:
Hi All, I am trying to understand how vmalloc memory is reserved in kernel. In 3.4+ latest kernel, the default vmalloc size is 240MB.
vmalloc reserved address range, you meant?
Is this a carveout from the 1GiB memory that kernel has?
Yup....
In other words can I do a __pa(VMALLOC_START) or __pa(VMALLOC_END) or __pa(highmemory) irrespective of the vmalloc size that I request, say doing vmalloc=1G?
uhm....not sure....vmalloc-ed is not physically contigous. Also, it is set up far after identity mapping setup. So, the value you get from __pa() IMHO would be likely invalid or has no meaning. -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
On Mon, Jun 25, 2012 at 1:00 PM, Mulyadi Santosa <mulyadi.santosa@gmail.com>wrote:
Hi.... :)
On Sun, Jun 24, 2012 at 1:32 AM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote:
Hi All, I am trying to understand how vmalloc memory is reserved in kernel. In 3.4+ latest kernel, the default vmalloc size is 240MB.
vmalloc reserved address range, you meant?
Is this a carveout from the 1GiB memory that kernel has?
Yup....
In other words can I do a __pa(VMALLOC_START) or __pa(VMALLOC_END) or __pa(highmemory) irrespective of the vmalloc size that I request, say doing vmalloc=1G?
uhm....not sure....vmalloc-ed is not physically contigous. Also, it is set up far after identity mapping setup. So, the value you get from __pa() IMHO would be likely invalid or has no meaning.
I thought that, as log as we know that the address is within the 1GiB, we could get the pa of the virtual address using __pa(). Is this not the case?
-- regards,
Mulyadi Santosa Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Hi Subbu, On Mon, Jun 25, 2012 at 11:18 AM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote:
On Mon, Jun 25, 2012 at 1:00 PM, Mulyadi Santosa <mulyadi.santosa@gmail.com> wrote:
Hi.... :)
On Sun, Jun 24, 2012 at 1:32 AM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote:
Hi All, I am trying to understand how vmalloc memory is reserved in kernel. In 3.4+ latest kernel, the default vmalloc size is 240MB.
vmalloc reserved address range, you meant?
Is this a carveout from the 1GiB memory that kernel has?
Yup....
In other words can I do a __pa(VMALLOC_START) or __pa(VMALLOC_END) or __pa(highmemory) irrespective of the vmalloc size that I request, say doing vmalloc=1G?
uhm....not sure....vmalloc-ed is not physically contigous. Also, it is set up far after identity mapping setup. So, the value you get from __pa() IMHO would be likely invalid or has no meaning.
I thought that, as log as we know that the address is within the 1GiB, we could get the pa of the virtual address using __pa(). Is this not the case?
__pa only works on kernel direct addresses. __pa doesn't work on the addresses from vmalloc Using __pa on VMALLOC_START or VMALLOC_END doesn't really make sense. If there were any physical memory there, it would be highmem. __pa should only be used on memory from PAGE_OFFSET through to (high_memory - 1) -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
On Mon, Jun 25, 2012 at 3:00 PM, Dave Hylands <dhylands@gmail.com> wrote:
Hi Subbu,
On Mon, Jun 25, 2012 at 11:18 AM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote:
On Mon, Jun 25, 2012 at 1:00 PM, Mulyadi Santosa <
mulyadi.santosa@gmail.com>
wrote:
Hi.... :)
On Sun, Jun 24, 2012 at 1:32 AM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote:
Hi All, I am trying to understand how vmalloc memory is reserved in kernel. In 3.4+ latest kernel, the default vmalloc size is 240MB.
vmalloc reserved address range, you meant?
Is this a carveout from the 1GiB memory that kernel has?
Yup....
In other words can I do a __pa(VMALLOC_START) or __pa(VMALLOC_END) or __pa(highmemory) irrespective of the vmalloc size that I request,
say
doing vmalloc=1G?
uhm....not sure....vmalloc-ed is not physically contigous. Also, it is set up far after identity mapping setup. So, the value you get from __pa() IMHO would be likely invalid or has no meaning.
I thought that, as log as we know that the address is within the 1GiB, we could get the pa of the virtual address using __pa(). Is this not the case?
__pa only works on kernel direct addresses.
__pa doesn't work on the addresses from vmalloc
Using __pa on VMALLOC_START or VMALLOC_END doesn't really make sense. If there were any physical memory there, it would be highmem.
Okay, but my intention was to just print out the pa for the VMALLOC_START and END, just to know where they are. And I thought they will work. no?
__pa should only be used on memory from PAGE_OFFSET through to (high_memory
- 1)
-- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
Hi Subbu, ...snip,,,
__pa only works on kernel direct addresses.
__pa doesn't work on the addresses from vmalloc
Using __pa on VMALLOC_START or VMALLOC_END doesn't really make sense. If there were any physical memory there, it would be highmem.
Okay, but my intention was to just print out the pa for the VMALLOC_START and END, just to know where they are. And I thought they will work. no?
You'll get some numbers, but they're meaningless. Let's take a concrete example (using ARM mappings) Suppose your machine has 1 GB of RAM with a physical addresses of 0x40000000 thru 0x7FFFFFFF Let's suppose that PHYS_OFFSET = 0xC0000000 Let's suppose that VMALLOC_START is set to 0xE0000000 and VMALLOC_END is at 0xEFFFFFFF (256 Mb) So what we have is that memory from 0xC0000000 thru 0xDFFFFFFF (512 Mb) is kernel direct mapped memory and maps to physical addresses 0x40000000 thru 0x5FFFFFFF Physical memory from 0x60000000 thru 0x7FFFFFFF (512Mb) is considered high-mem. For the most part this is only usable by user-space. From kernel space, you need to use the kmap/kunmap functions to even access the memory. If you use __pa(VMALLOC_START) you'll ALWAYS (with the above example) get 0xE0000000. Calling __pa(VMALLOC_END) would return 0xEFFFFFFF which is some address from the middle of high-memory, essentially a meaningless number. However, if you call vmalloc and lets suppose that vmalloc just happens to return 0xE0000000. The physical address of the first page might be 0xD2345000. What's important is that the physical pages which back up the vmalloc area all come from the kernel direct mapped area. They won't ever be backed by pages from high-memory. So the physical addresses will all be in the range 0x40000000 thru 0x5FFFFFFF. You can use vmalloc_to_pfn to return the PFN of the page which is backing a vmalloc'd virtual address. (if the physical address of 0xE0000000 was 0xD2345000, then the pfn would be 0xD2345, assuming 4K pages). -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
Hi Dave, Thank you for taking your time to provide this example. This is indeed very insightful. On Mon, Jun 25, 2012 at 7:41 PM, Dave Hylands <dhylands@gmail.com> wrote:
Hi Subbu,
...snip,,,
__pa only works on kernel direct addresses.
__pa doesn't work on the addresses from vmalloc
Using __pa on VMALLOC_START or VMALLOC_END doesn't really make sense. If there were any physical memory there, it would be highmem.
Okay, but my intention was to just print out the pa for the VMALLOC_START and END, just to know where they are. And I thought they will work. no?
You'll get some numbers, but they're meaningless.
Let's take a concrete example (using ARM mappings)
Suppose your machine has 1 GB of RAM with a physical addresses of 0x40000000 thru 0x7FFFFFFF
Let's suppose that PHYS_OFFSET = 0xC0000000 Let's suppose that VMALLOC_START is set to 0xE0000000 and VMALLOC_END is at 0xEFFFFFFF (256 Mb)
Agreed!
So what we have is that memory from 0xC0000000 thru 0xDFFFFFFF (512 Mb) is kernel direct mapped memory and maps to physical addresses 0x40000000 thru 0x5FFFFFFF
Agreed
Physical memory from 0x60000000 thru 0x7FFFFFFF (512Mb) is considered high-mem. For the most part this is only usable by user-space. From kernel space, you need to use the kmap/kunmap functions to even access the memory.
Agreed
If you use __pa(VMALLOC_START) you'll ALWAYS (with the above example) get 0xE0000000. Calling __pa(VMALLOC_END) would return 0xEFFFFFFF which is some address from the middle of high-memory, essentially a meaningless number.
Agreed this one too!!!
However, if you call vmalloc and lets suppose that vmalloc just happens to return 0xE0000000. The physical address of the first page might be 0xD2345000.
What's important is that the physical pages which back up the vmalloc area all come from the kernel direct mapped area. They won't ever be backed by pages from high-memory. So the physical addresses will all be in the range 0x40000000 thru 0x5FFFFFFF.
This is the one that I had a completely wrong understanding of!!!!!!! I understand from the above statement that the vmalloc'ed virtual address will _ALWAYS_ correspond to a physical address from the lowmem region! I was under the impression that the carved out region for the vmalloc, is the one that would back any vmalloc'ed virtual address, which is absolutely wrong by what you are saying.
Now this also means that increasing vmalloc inadvertently reduces lowmem. Why is this designed such a way? Essentially, the idea that we increase vmalloc is because we expect more memory to be consumed via vmalloc calls, and hence we might need more physical address backing. But increasing vmalloc decreases low mem, which would also mean that we have less backing. Am I missing something here too :)? You can use vmalloc_to_pfn to return the PFN of the page which is
backing a vmalloc'd virtual address. (if the physical address of 0xE0000000 was 0xD2345000, then the pfn would be 0xD2345, assuming 4K pages).
Agreed on this too! Thanks again for your time and patience.
-- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
Hi Subbu, On Mon, Jun 25, 2012 at 7:45 PM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote: ...snip...
However, if you call vmalloc and lets suppose that vmalloc just happens to return 0xE0000000. The physical address of the first page might be 0xD2345000.
What's important is that the physical pages which back up the vmalloc area all come from the kernel direct mapped area. They won't ever be backed by pages from high-memory. So the physical addresses will all be in the range 0x40000000 thru 0x5FFFFFFF.
This is the one that I had a completely wrong understanding of!!!!!!! I understand from the above statement that the vmalloc'ed virtual address will _ALWAYS_ correspond to a physical address from the lowmem region! I was under the impression that the carved out region for the vmalloc, is the one that would back any vmalloc'ed virtual address, which is absolutely wrong by what you are saying.
Actually, I was the one that was wrong. vmalloc pages can come from low or highmem.
Now this also means that increasing vmalloc inadvertently reduces lowmem. Why is this designed such a way?
It may or may not depending on the amount of physical memory and the size of the vmalloc space. vmalloc space will normally increase vmalloc_end, which won't reduce lowmem. If the end can't advance any further, then I believe that the start can be reduced. This will reduce lowmem, if the lowmem overlaps with vmalloc memory.
Essentially, the idea that we increase vmalloc is because we expect more memory to be consumed via vmalloc calls, and hence we might need more physical address backing. But increasing vmalloc decreases low mem, which would also mean that we have less backing. Am I missing something here too :)?
No - that was my mistake. -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
On Tue, Jun 26, 2012 at 12:41 AM, Dave Hylands <dhylands@gmail.com> wrote:
Hi Subbu,
On Mon, Jun 25, 2012 at 7:45 PM, Subramaniam Appadodharana <c.a.subramaniam@gmail.com> wrote: ...snip...
However, if you call vmalloc and lets suppose that vmalloc just happens to return 0xE0000000. The physical address of the first page might be 0xD2345000.
What's important is that the physical pages which back up the vmalloc area all come from the kernel direct mapped area. They won't ever be backed by pages from high-memory. So the physical addresses will all be in the range 0x40000000 thru 0x5FFFFFFF.
This is the one that I had a completely wrong understanding of!!!!!!! I understand from the above statement that the vmalloc'ed virtual address will _ALWAYS_ correspond to a physical address from the lowmem region! I was under the impression that the carved out region for the vmalloc, is the one that would back any vmalloc'ed virtual address, which is absolutely wrong by what you are saying.
Actually, I was the one that was wrong. vmalloc pages can come from low or highmem.
Thanks for clarifying! Not a problem.
Now this also means that increasing vmalloc inadvertently reduces lowmem.
Why is this designed such a way?
It may or may not depending on the amount of physical memory and the size of the vmalloc space.
vmalloc space will normally increase vmalloc_end, which won't reduce lowmem. If the end can't advance any further, then I believe that the start can be reduced. This will reduce lowmem, if the lowmem overlaps with vmalloc memory.
Okay! got that!
Essentially, the idea that we increase vmalloc is because we expect more memory to be consumed via vmalloc calls, and hence we might need more physical address backing. But increasing vmalloc decreases low mem, which would also mean that we have less backing. Am I missing something here too :)?
No - that was my mistake.
No problem again. Thanks for correcting it!
-- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
Now this also means that increasing vmalloc inadvertently reduces lowmem. Why is this designed such a way?
It may or may not depending on the amount of physical memory and the size of the vmalloc space. vmalloc space will normally increase vmalloc_end, which won't reduce lowmem. If the end can't advance any further, then I believe that the start can be reduced. This will reduce lowmem, if the lowmem overlaps with vmalloc memory. Okay! got that! VMALLOC_END is a define, it is decided staticly when compile. but VMALLOC_START is a dynamic value, which is calculated when running. VMALLOC_START depends on the "vmalloc=XXX" parameter which is passed by bootloader(u-boot). so if you specify a large vmalloc size in bootloader, for example "vmalloc=512M", the VMALLOC_START will be reduced accordingly. the rough formula is as below: 1. define VMALLOC_END = 0xF0000000 2. pass in parameter "vmalloc=256M" in u-boot. 3. in kernel SW, a global variable VMALLOC_MIN = VMALLOC_END - vmalloc_size = 0xF0000000 - 256M = 0xE0000000 4. calculate total size of "mem=" which is passed by u-boot, for example total memory size is 1GB. 5. global variable high_memory = VMALLOC_MIN = 0xE0000000, so lowmem is from 0xC0000000 ~ 0xE0000000, totoal 512MB, and total high memory is 512MB. 6. VMALLOC_START = high_memory + 8MB = 0xE0800000. at last , the vmalloc region will be 0xE0800000 ~ 0xF0000000
Suppose your machine has 1 GB of RAM with a physical addresses of 0x40000000 thru 0x7FFFFFFF
Let's suppose that PHYS_OFFSET = 0xC0000000 Let's suppose that VMALLOC_START is set to 0xE0000000 and VMALLOC_END is at 0xEFFFFFFF (256 Mb)
is this PAGE_OFFSET or PHYS_OFFSET?
So what we have is that memory from 0xC0000000 thru 0xDFFFFFFF (512 Mb) is kernel direct mapped memory and maps to physical addresses 0x40000000 thru 0x5FFFFFFF
Physical memory from 0x60000000 thru 0x7FFFFFFF (512Mb) is considered high-mem. For the most part this is only usable by user-space. From kernel space, you need to use the kmap/kunmap functions to even access the memory.
i don't think high-memory is only used by user-space. vmalloc() funciton will try to allocate physical memory from high_memory first, if there is no high_memory, then allocate from normal region.
However, if you call vmalloc and lets suppose that vmalloc just happens to return 0xE0000000. The physical address of the first page might be 0xD2345000.
suppose vmalloc() return 0xE0000000, which is the VMALLOC_START, then the physical memory address should be 0x60000000 thru 0x7FFFFFFF in your example.
What's important is that the physical pages which back up the vmalloc area all come from the kernel direct mapped area. They won't ever be backed by pages from high-memory. So the physical addresses will all be in the range 0x40000000 thru 0x5FFFFFFF.
what do you mean "back up", maybe i have some misunderstanding. in my opnion, vmalloc area should map to physical address 0x60000000 thru 0x7FFFFFFF first, if this resion is exhausted, then normal area will be allocated.
Hi, On Mon, Jun 25, 2012 at 9:01 PM, 卜弋天 <buyit@live.cn> wrote:
Suppose your machine has 1 GB of RAM with a physical addresses of 0x40000000 thru 0x7FFFFFFF
Let's suppose that PHYS_OFFSET = 0xC0000000 Let's suppose that VMALLOC_START is set to 0xE0000000 and VMALLOC_END is at 0xEFFFFFFF (256 Mb)
is this PAGE_OFFSET or PHYS_OFFSET?
PAGE_OFFSET is a virtual address, and PHYS_OFFSET is a physical address, so you are correct. That should have been PAGE_OFFSET.
So what we have is that memory from 0xC0000000 thru 0xDFFFFFFF (512 Mb) is kernel direct mapped memory and maps to physical addresses 0x40000000 thru 0x5FFFFFFF
Physical memory from 0x60000000 thru 0x7FFFFFFF (512Mb) is considered high-mem. For the most part this is only usable by user-space. From kernel space, you need to use the kmap/kunmap functions to even access the memory.
i don't think high-memory is only used by user-space. vmalloc() funciton will try to allocate physical memory from high_memory first, if there is no high_memory, then allocate from normal region.
That also seems to be correct. The code in vmalloc that calls the underlying page allocator passes in __GFP_HIGHMEM, so it will try to use highmem for vmalloc'd pages.
However, if you call vmalloc and lets suppose that vmalloc just happens to return 0xE0000000. The physical address of the first page might be 0xD2345000.
suppose vmalloc() return 0xE0000000, which is the VMALLOC_START, then the physical memory address should be 0x60000000 thru 0x7FFFFFFF in your example.
What's important is that the physical pages which back up the vmalloc area all come from the kernel direct mapped area. They won't ever be backed by pages from high-memory. So the physical addresses will all be in the range 0x40000000 thru 0x5FFFFFFF.
what do you mean "back up", maybe i have some misunderstanding. in my opnion, vmalloc area should map to physical address 0x60000000 thru 0x7FFFFFFF first, if this resion is exhausted, then normal area will be allocated.
By backed-by, I meant the associated physical page. I was incorrect in saying that only lowmem pages would be used. -- Dave Hylands Shuswap, BC, Canada http://www.davehylands.com
participants (4)
-
Dave Hylands -
Mulyadi Santosa -
Subramaniam Appadodharana -
卜弋天