Hi, Thanks for the link.I really appreciate but I need something more basic and something that explains these concepts from a broader perspective and not in the context of a particular cpu architecture.Please send me more such links if you come across any. Thanks Vaibhav Jain On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta <daniel.baluta@gmail.com>wrote:

Hi,

...

I am eager to understand the basics of Memory models (flat, segmented etc) , CPU modes (real,protected) and address translation (physical to logical etc.) and how all of them work together. I am very confused about this and would really appreciate if someone could provide good references to these topics.

You may find useful information inside i366 Programmers Manual. Anyhow, reading materials is the first step in understanding these concepts. You will have to actually read/write/debug pieces of code related to them.

thanks, Daniel.

[1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf

The document and osdev are very good and useful. Thanks a lot. - Paraneetharan C On 13 July 2011 14:23, amit mehta <gmate.amit@gmail.com> wrote:

Tons of resource http://wiki.osdev.org/

On Wed, Jul 13, 2011 at 2:07 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:

...

Hi,

Thanks for the link.I really appreciate but I need something more basic and something that explains these concepts from a broader perspective and not in the context of a particular cpu architecture.Please send me more such links if you come across any.

Thanks Vaibhav Jain

On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta <daniel.baluta@gmail.com> wrote:

...

Hi,

...

I am eager to understand the basics of Memory models (flat, segmented etc) , CPU modes (real,protected) and address translation (physical to logical etc.) and how all of them work together. I am very confused about this and would really appreciate if someone could provide good references to these topics.

You may find useful information inside i366 Programmers Manual. Anyhow, reading materials is the first step in understanding these concepts. You will have to actually read/write/debug pieces of code related to them.

thanks, Daniel.

[1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf

_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

-- Life is elsewhere. Cross Frontiers. Fly away.

- Salman Rushdie

_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

-- Regards, Paraneetharan C

On Wed, Jul 13, 2011 at 6:29 PM, Jeff Haran <jharan@bytemobile.com> wrote:

** **

** **

*From:* kernelnewbies-bounces@kernelnewbies.org [mailto: kernelnewbies-bounces@kernelnewbies.org] *On Behalf Of *Vaibhav Jain *Sent:* Wednesday, July 13, 2011 1:38 AM *To:* Daniel Baluta *Cc:* kernelnewbies@kernelnewbies.org *Subject:* Re: Need help understanding memory models,cpu modes and address translation****

** **

Hi,

Thanks for the link.I really appreciate but I need something more basic and something that explains these concepts from a broader perspective and not in the context of a particular cpu architecture.Please send me more such links if you come across any.

Thanks Vaibhav Jain****

On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta <daniel.baluta@gmail.com> wrote:****

Hi,

...

I am eager to understand the basics of Memory models (flat, segmented etc) , CPU modes (real,protected) and address translation (physical to logical etc.) and how all of them work together. I am very confused about this and would really appreciate if someone could provide good references to these topics.****

You may find useful information inside i366 Programmers Manual. Anyhow, reading materials is the first step in understanding these concepts. You will have to actually read/write/debug pieces of code related to them.

thanks, Daniel.

[1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf****

** **

You might want to try “Understanding the Linux Virtual Memory Manager” by Mel Gorman. It’s freely available on the web.****

** **

It’s Linux specific, but since Linux needs to run on a multitude of different processors with different memory management hardware, the approach it takes to doing so in inherently broad.****

** **

A lot of basic concepts can be learnt from JIm Turley's book on 8086 Other book is modern operating systems by Tanenbaum which explains everything in much detail with code snippets.

I think you should first read the book Operating System Concepts by Silberschatz Galvin. It is a general book on OS concepts and will help you gain understanding of these concepts. -Kaustubh On Thu, Jul 14, 2011 at 1:17 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:

Hi,

Thanks a lot for the resources! I just had a cursory look at them and could see that they explain virtual memory and address translation in depth. Just wanted to ask if they explain the concepts Memory models (flat, segmented ) and cpu modes (real,protected) equally well. I am more confused about these concepts.

Thanks Vaibhav Jain

On Wed, Jul 13, 2011 at 10:40 AM, Anuz Pratap Singh Tomar < chambilkethakur@gmail.com> wrote:

...

On Wed, Jul 13, 2011 at 6:29 PM, Jeff Haran <jharan@bytemobile.com>wrote:

...

** **

** **

*From:* kernelnewbies-bounces@kernelnewbies.org [mailto: kernelnewbies-bounces@kernelnewbies.org] *On Behalf Of *Vaibhav Jain *Sent:* Wednesday, July 13, 2011 1:38 AM *To:* Daniel Baluta *Cc:* kernelnewbies@kernelnewbies.org *Subject:* Re: Need help understanding memory models,cpu modes and address translation****

** **

Hi,

Thanks for the link.I really appreciate but I need something more basic and something that explains these concepts from a broader perspective and not in the context of a particular cpu architecture.Please send me more such links if you come across any.

Thanks Vaibhav Jain****

On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta <daniel.baluta@gmail.com> wrote:****

Hi,

...

I am eager to understand the basics of Memory models (flat, segmented etc) , CPU modes (real,protected) and address translation (physical to logical etc.) and how all of them work together. I am very confused about this and would really appreciate if someone could provide good references to these topics.****

You may find useful information inside i366 Programmers Manual. Anyhow, reading materials is the first step in understanding these concepts. You will have to actually read/write/debug pieces of code related to them.

thanks, Daniel.

[1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf****

** **

You might want to try “Understanding the Linux Virtual Memory Manager” by Mel Gorman. It’s freely available on the web.****

** **

It’s Linux specific, but since Linux needs to run on a multitude of different processors with different memory management hardware, the approach it takes to doing so in inherently broad.****

** **

A lot of basic concepts can be learnt from JIm Turley's book on 8086 Other book is modern operating systems by Tanenbaum which explains everything in much detail with code snippets.

_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

On Fri, Jul 15, 2011 at 10:04 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:

Hi,

Actually I have read that book. But when i started reading other books such as those on assembly they had these concpets of Real Mode , Protected Mode , Flat Memory model, Segmented Memory model which are specific to intel 32-bit architecture and which has got me highly confused. So I am looking for references that explain these concepts in depth.

Then you should read the book "Understanding The Linux Kernel", and Intel's reference manuals which document these features. Also this is the book we read in Undergrad for x86 internals :) It was considered to be the bible for the topic (atleast at the time) http://www.amazon.com/Intel-Microprocessors-80186-80286-80386/dp/0132606704/... Regards, Joel

On Sat, Jul 16, 2011 at 8:34 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:

Hi,

Actually I have read that book. But when i started reading other books such as those on assembly they had these concpets of Real Mode , Protected Mode , Flat Memory model, Segmented Memory model which are specific to intel 32-bit architecture and which has got me highly confused. So I am looking for references that explain these concepts in depth.

What I suggest is. You should download a datasheet for Intel 8086, designed by Intel in late 1970's. Read espacially the memory organization part. This will give you some basic understanding of read mode and segmentation stuff and how it all started. The current processor manuals are too complex to start with. Link: http://datasheets.chipdb.org/Intel/x86/808x/datashts/8086/

Thanks Vaibhav Jain

On Thu, Jul 14, 2011 at 9:26 PM, Kaustubh Ashtekar <ksashtekar@gmail.com> wrote:

...

I think you should first read the book Operating System Concepts by  Silberschatz Galvin. It is a general book on OS concepts and will help you gain understanding of these concepts. -Kaustubh On Thu, Jul 14, 2011 at 1:17 AM, Vaibhav Jain <vjoss197@gmail.com> wrote:

...

Hi,

Thanks a lot for the resources! I just had a cursory look at them and could see that they explain virtual memory and address translation in depth. Just wanted to ask if they explain the concepts Memory models (flat, segmented ) and cpu modes (real,protected) equally well. I am more confused about these concepts.

Thanks Vaibhav Jain

On Wed, Jul 13, 2011 at 10:40 AM, Anuz Pratap Singh Tomar <chambilkethakur@gmail.com> wrote:

...

On Wed, Jul 13, 2011 at 6:29 PM, Jeff Haran <jharan@bytemobile.com> wrote:

...

From: kernelnewbies-bounces@kernelnewbies.org [mailto:kernelnewbies-bounces@kernelnewbies.org] On Behalf Of Vaibhav Jain Sent: Wednesday, July 13, 2011 1:38 AM To: Daniel Baluta Cc: kernelnewbies@kernelnewbies.org Subject: Re: Need help understanding memory models,cpu modes and address translation

Hi,

Thanks for the link.I really appreciate but I need something more basic and something that explains these concepts from a broader perspective and not in the context of a particular cpu architecture.Please send me more such links if you come across any.

Thanks Vaibhav Jain

On Wed, Jul 13, 2011 at 1:14 AM, Daniel Baluta <daniel.baluta@gmail.com> wrote:

Hi,

...

I am eager to understand the basics of Memory models  (flat, segmented etc) , CPU modes (real,protected) and address translation (physical to logical etc.) and how all of them work together. I am very confused about this and would really appreciate if someone could provide good references to these topics.

You may find useful information inside i366 Programmers Manual. Anyhow, reading materials is the first step in understanding these concepts. You will have to actually  read/write/debug pieces of code related to them.

thanks, Daniel.

[1] pdos.csail.mit.edu/6.828/2010/readings/i386.pdf

You might want to try “Understanding the Linux Virtual Memory Manager” by Mel Gorman. It’s freely available on the web.

It’s Linux specific, but since Linux needs to run on a multitude of different processors with different memory management hardware, the approach it takes to doing so in inherently broad.

A lot of basic concepts can be learnt from JIm Turley's book on 8086 Other book is modern operating systems by Tanenbaum which explains everything in much detail with  code snippets.

_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

On Wed, Jul 13, 2011 at 9:41 PM, Mulyadi Santosa <mulyadi.santosa@gmail.com> wrote:

On Wed, Jul 13, 2011 at 14:33, Vaibhav Jain <vjoss197@gmail.com> wrote:

...

Hi,

I am eager to understand the basics of Memory models  (flat, segmented etc) , CPU modes (real,protected) and address translation (physical to logical etc.) and how all of them work together.

maybe something like this?

http://tldp.org/HOWTO/KernelAnalysis-HOWTO.html

This should nail it :) http://duartes.org/gustavo/blog/post/how-the-kernel-manages-your-memory http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory Thanks, Joel

Hello Vaibhav... What you asked, is something really difficult to summarize in short...but let me try... On Sat, Jul 16, 2011 at 10:14, Vaibhav Jain <vjoss197@gmail.com> wrote:

Hi, Thanks a lot for all the links! I read the two articles below by Gustavo but I am still very confused about segmentation. One of the articles below says that the segments in virtual address space are different from 'intel-style segments' whereas I used to think of them as being the same.

I am not really sure what segments in virtual address space really means. What we have in virtual address space is AFAIK kernel space and user space. User space, is a space where OS put user and data from your casual programs (e.g your games, your word processors etc). Assuming it's in x86 32 bit, the range in virtual address is between address 0 and slightly under 3 GiB. Whereas kernel space, is a space where OS put its own data and code (i.e interrupt handler, system call handler, page tables). Again in x86 32 bit, it lies between 3 GiB up to 4 GiB.

Could somebody please state the difference clearly for me and explain how these two work together. I would really appreciate if someone could explain the whole chain from generation of addresses by compiler and then translation of those addresses in case of  Segmentation working along with Paging.

when you generate object code from your source code (let's say in C) using gcc, first your code and variables (data) are turn into Position Independent Code. It means, it is just an offset. If there is an offset, surely we need base address, right? But not at this object (resulting in .o) stage. Then in reach producing final ELF binary (executable). Using known standart ELF rule, those offset are turn into final final address. So let's, code are placed starting at 0x080499f0 and so on. When that binary is loaded into memory, loader (ld.so) take that information and use it as a clue on where to put the code and data. Using standart mmap() syscall, memory area is reserved and data/code is loaded there. The exception is stack, where it is allocated dynamically (and grows down, for Intel arch) starting at the upper limit of user space (near 3 GiB). The new feature, called PIE (position independent code) allow loader to do further address reorganization. So instead of let's say taking 0x080499f0 for start of code segment for granted, it can relocate it to another address (but still in user space). However, in order to do that, every addresing done by the code must be compiled differently, so instead using constant address, now it uses combination of base register and offset AFAIK. Code runs, data addressed, they are all under virtual address. But memory is pointed using real address. Here comes the work of MMU (Memory Management Unit). With the help of tables of address translation called page tables, virtual address is translated into real address. The detail is left for you for further research. But enough to say it works like lookup table, e.g "12" in virtual address could means real address 10012 and so on. Please notice that continous address in virtual address doesn't neccessarily means continous address in real address e.g 12-20 in virtual address could means 10012 up to 10016 then 40017 up to 40020. Getting clearer now? :D -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

On Sat, Jul 16, 2011 at 9:45 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com> wrote:

Hello Vaibhav...

What you asked, is something really difficult to summarize in short...but let me try...

On Sat, Jul 16, 2011 at 10:14, Vaibhav Jain <vjoss197@gmail.com> wrote:

...

Hi, Thanks a lot for all the links! I read the two articles below by Gustavo but I am still very confused about segmentation. One of the articles below says that the segments in virtual address space are different from 'intel-style segments' whereas I used to think of them as being the same.

I am not really sure what segments in virtual address space really means. What we have in virtual address space is AFAIK kernel space and user space.

I think in linux all segments(__USER_CS,__USER_DS,__KERNEL_CS,__KERNEL_DS) starts at 0x00000000.So offset fileld of logical address always coincide with linear address.So in case of Linux, segmentation is some dumb kind of thing.can any one correct me if i am wrong.

User space, is a space where OS put user and data from your casual programs (e.g your games, your word processors etc). Assuming it's in x86 32 bit, the range in virtual address is between address 0 and slightly under 3 GiB.

Whereas kernel space, is a space where OS put its own data and code (i.e interrupt handler, system call handler, page tables). Again in x86 32 bit, it lies between 3 GiB up to 4 GiB.

...

Could somebody please state the difference clearly for me and explain how these two work together. I would really appreciate if someone could explain the whole chain from generation of addresses by compiler and then translation of those addresses in case of  Segmentation working along with Paging.

when you generate object code from your source code (let's say in C) using gcc, first your code and variables (data) are turn into Position Independent Code. It means, it is just an offset. If there is an offset, surely we need base address, right? But not at this object (resulting in .o) stage.

Then in reach producing final ELF binary (executable). Using known standart ELF rule, those offset are turn into final final address. So let's, code are placed starting at 0x080499f0 and so on.

When that binary is loaded into memory, loader (ld.so) take that information and use it as a clue on where to put the code and data. Using standart mmap() syscall, memory area is reserved and data/code is loaded there. The exception is stack, where it is allocated dynamically (and grows down, for Intel arch) starting at the upper limit of user space (near 3 GiB).

The new feature, called PIE (position independent code) allow loader to do further address reorganization. So instead of let's say taking 0x080499f0 for start of code segment for granted, it can relocate it to another address (but still in user space). However, in order to do that, every addresing done by the code must be compiled differently, so instead using constant address, now it uses combination of base register and offset AFAIK.

Code runs, data addressed, they are all under virtual address. But memory is pointed using real address. Here comes the work of MMU (Memory Management Unit). With the help of tables of address translation called page tables, virtual address is translated into real address. The detail is left for you for further research. But enough to say it works like lookup table, e.g "12" in virtual address could means real address 10012 and so on.

Please notice that continous address in virtual address doesn't neccessarily means continous address in real address e.g 12-20 in virtual address could means 10012 up to 10016 then 40017 up to 40020.

Getting clearer now? :D

-- regards,

Mulyadi Santosa Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

_______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Hi Mulyadi, Thanks for the explanation.Its really nice! But what I was referring to the was this article on virtual address layout of program : http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory which mentions : "The distinct bands in the address space correspond to *memory segments* like the heap, stack, and so on. Keep in mind these segments are simply a range of memory addresses and *have nothing to do* with Intel-style segments<http://duartes.org/gustavo/blog/post/memory-translation-and-segmentation> ." This gave rise to all the confusion.I used to think that code, stack and heap segments in the virtual address layout of a program are the same as the segments which we talk about when referring to hardware provided Segmentation. But seems like this is not the case. I also read a little about real and protected mode and came to know that in 32-bit protected mode all the Segment registers point to the same address This confused me even more. So I need an explanation of how all these work together. I am Sorry if the question is not clear or if it sounds confusing. Thanks Vaibhav Jain On Fri, Jul 15, 2011 at 9:15 PM, Mulyadi Santosa <mulyadi.santosa@gmail.com>wrote:

Hello Vaibhav...

What you asked, is something really difficult to summarize in short...but let me try...

On Sat, Jul 16, 2011 at 10:14, Vaibhav Jain <vjoss197@gmail.com> wrote:

...

Hi, Thanks a lot for all the links! I read the two articles below by Gustavo but I am still very confused about segmentation. One of the articles below says that the segments in virtual address space are different from 'intel-style segments' whereas I used to think of them as being the same.

I am not really sure what segments in virtual address space really means. What we have in virtual address space is AFAIK kernel space and user space.

User space, is a space where OS put user and data from your casual programs (e.g your games, your word processors etc). Assuming it's in x86 32 bit, the range in virtual address is between address 0 and slightly under 3 GiB.

Whereas kernel space, is a space where OS put its own data and code (i.e interrupt handler, system call handler, page tables). Again in x86 32 bit, it lies between 3 GiB up to 4 GiB.

...

Could somebody please state the difference clearly for me and explain how these two work together. I would really appreciate if someone could explain the whole chain from generation of addresses by compiler and then translation of those addresses in case of Segmentation working along with Paging.

when you generate object code from your source code (let's say in C) using gcc, first your code and variables (data) are turn into Position Independent Code. It means, it is just an offset. If there is an offset, surely we need base address, right? But not at this object (resulting in .o) stage.

Then in reach producing final ELF binary (executable). Using known standart ELF rule, those offset are turn into final final address. So let's, code are placed starting at 0x080499f0 and so on.

When that binary is loaded into memory, loader (ld.so) take that information and use it as a clue on where to put the code and data. Using standart mmap() syscall, memory area is reserved and data/code is loaded there. The exception is stack, where it is allocated dynamically (and grows down, for Intel arch) starting at the upper limit of user space (near 3 GiB).

The new feature, called PIE (position independent code) allow loader to do further address reorganization. So instead of let's say taking 0x080499f0 for start of code segment for granted, it can relocate it to another address (but still in user space). However, in order to do that, every addresing done by the code must be compiled differently, so instead using constant address, now it uses combination of base register and offset AFAIK.

Code runs, data addressed, they are all under virtual address. But memory is pointed using real address. Here comes the work of MMU (Memory Management Unit). With the help of tables of address translation called page tables, virtual address is translated into real address. The detail is left for you for further research. But enough to say it works like lookup table, e.g "12" in virtual address could means real address 10012 and so on.

Please notice that continous address in virtual address doesn't neccessarily means continous address in real address e.g 12-20 in virtual address could means 10012 up to 10016 then 40017 up to 40020.

Getting clearer now? :D

-- regards,

Mulyadi Santosa Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

Thanks a lot Joel! This is a great explanation. Just one more question. I used to think that the compiler always assigns/generates starting from 0 as Mulyadi has also mentioned. In the case when Segmentation(intel-style) is being used how does the compiler assigns addresses? Thanks Vaibhav Jain On Sat, Jul 16, 2011 at 11:13 PM, Joel A Fernandes <agnel.joel@gmail.com>wrote:

On Sat, Jul 16, 2011 at 3:47 PM, Vaibhav Jain <vjoss197@gmail.com> wrote:

...

Hi Mulyadi,

Thanks for the explanation.Its really nice! But what I was referring to the was this article on virtual address layout of program :

http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory

which mentions : "The distinct bands in the address space correspond to memory segments like the heap, stack, and so on. Keep in mind these segments are simply a range of memory addresses and have nothing to do with Intel-style segments."

This gave rise to all the confusion.I used to think that code, stack and heap segments in the virtual address layout of a program are the same as the segments which we talk about when referring to hardware provided Segmentation. But seems like this is not the case.

Segmentation and the virtual address layout are independent of each other so you shouldn't confuse the two. You can choose to have segmentation and then virtual addressing.

The way it works is:

logical address (segmented) -> virtual address -> physical address

The logical to virtual conversion is called segmentation, and virtual to physical is called paging. The Software always uses logical addresses.

The article explains "heap", "stack" etc are segments created by the operating system and have nothing to do with traditional Intel-style "segments". Infact the hardware would not be even aware of the presence of the heap. The HW only knows virtual addresses.

...

I also read a little about real and protected mode and came to know that in 32-bit protected mode all the Segment registers point to the same address

It happens so that Linux sets up the processor tables in such a way that the logical addresses map to the same as virtual addresses. IIRC, For x86 there's a "Global Descriptor table" that Linux manipulates to produce this one-to-one mapping.

...

This confused me even more. So I need an explanation of how all these work together. I am Sorry if the question is not clear or if it sounds confusing.

Hope this clears it, do read the introduction chapters of Understanding the Linux Kernel which touches on a lot of these topics. Feel free to ask more questions.

Thanks, Joel