Overwriting copy functionality in filesystem
Hey everyone, I was wondering how we can overwrite the copy functionality while writing our own filesystem in linux. VFS does not offer any sort of API for copy. I think it calls create and write when we execute the copy the file/dir. I am interested in overwriting the way copy happens in my filesystem(which I am writing for fun :p). Any ideas? Thanks
On Sat, 23 Mar 2019 22:29:45 +0530, Bharath Vedartham said:
I was wondering how we can overwrite the copy functionality while writing our own filesystem in linux. VFS does not offer any sort of API for copy. I think it calls create and write when we execute the copy the file/dir.
Which you can verify using strace. Which you should already be familiar with if you have the experience needed to write a usable filesystem.
I am interested in overwriting the way copy happens in my filesystem(which I am writing for fun :p).
And what, exactly, do you want copy to do differently on the API level, and on the file system level?
On Sat, Mar 23, 2019 at 03:01:56PM -0400, Valdis Klētnieks wrote:
On Sat, 23 Mar 2019 22:29:45 +0530, Bharath Vedartham said:
I was wondering how we can overwrite the copy functionality while writing our own filesystem in linux. VFS does not offer any sort of API for copy. I think it calls create and write when we execute the copy the file/dir.
Which you can verify using strace. Which you should already be familiar with if you have the experience needed to write a usable filesystem.
Yes I did that. I have observed that it is a mixture of create,read,write system calls.
I am interested in overwriting the way copy happens in my filesystem(which I am writing for fun :p).
And what, exactly, do you want copy to do differently on the API level, and on the file system level?
I was interested in implementing copy-on-write for my filesystem(for fun :P). When I do a "cp" operation, I do not want to create a seperate inode for the new file. I only want to create a inode when I make a change to the file. There is no vfs api for cp. I would need to make creat syscall aware of the fact that it is being executed by "cp". My immediate idea was to check if a file with the same data exists in the filesystem but that would be way too much overhead.
On Sun, 24 Mar 2019 18:48:08 +0530, Bharath Vedartham said:
I was interested in implementing copy-on-write for my filesystem(for fun :P). When I do a "cp" operation, I do not want to create a seperate inode for the new file. I only want to create a inode when I make a change to the file.
Actually, /bin/cp isn't where copy-on-write gets you benefits. Where it really shines is when you have a versioning filesystem that keeps track of the last N versions of a file with minimum overhead. So if you have a 100 megabyte file, open it, write 5 blocks of data, and close it, you now can read back either the original or new versions of the file, and you're only using 100M plus 5 blocks plus a tiny bit of metadata.
There is no vfs api for cp. I would need to make creat syscall aware of the fact that it is being executed by "cp". My immediate idea was to check if a file with the same data exists in the filesystem but that would be way too much overhead.
Have you looked at other filesystems that already support copy-on-write? Hint: How do file systems that support point-in-time snapshots work?
On Sun, Mar 24, 2019 at 10:06:36AM -0400, Valdis Klētnieks wrote:
On Sun, 24 Mar 2019 18:48:08 +0530, Bharath Vedartham said:
I was interested in implementing copy-on-write for my filesystem(for fun :P). When I do a "cp" operation, I do not want to create a seperate inode for the new file. I only want to create a inode when I make a change to the file.
Actually, /bin/cp isn't where copy-on-write gets you benefits. Where it really shines is when you have a versioning filesystem that keeps track of the last N versions of a file with minimum overhead. So if you have a 100 megabyte file, open it, write 5 blocks of data, and close it, you now can read back either the original or new versions of the file, and you're only using 100M plus 5 blocks plus a tiny bit of metadata.
That is a very interesting use case. Any examples of filesystems like that in the kernel? I was thinking of a use case where we are copying a huge file (say 100 GB), if we do copy-on-write we can speed up /bin/cp for such files i feel. Any comments on this?
There is no vfs api for cp. I would need to make creat syscall aware of the fact that it is being executed by "cp". My immediate idea was to check if a file with the same data exists in the filesystem but that would be way too much overhead.
Have you looked at other filesystems that already support copy-on-write?
I have been looking at overlayfs. But overlayfs uses a lower read-only layer and an upper layer where changes are reflected. I am more interested in filesystems without any layering which support copy-on-write. Are there other filesystems in the kernel which support copy-on-write?
Hint: How do file systems that support point-in-time snapshots work?
On Fri, 29 Mar 2019 00:00:17 +0530, Bharath Vedartham said:
I was thinking of a use case where we are copying a huge file (say 100 GB), if we do copy-on-write we can speed up /bin/cp for such files i feel. Any comments on this?
Hmm.. wait a minute. What definition of "copy on write" are you using? Hint - if you're copying an *entire* 100GB file, the *fastest* way is to simply make a second hard link to the file. If you're determined to make an entire second copy, you're going to be reading 100GB and writing 100GB, and the exact details aren't going to matter all that much. Now, where you can get clever is if you create your 100GB file, and then somebody only changes 8K of the file. There's no need to copy all 100GB into a new file if you are able to record "oh, and this 8K got changed". You only need to write the 8K of changes, and some metadata. (Similar tricks are used for shared libraries and pre-zero'ed storage. Everybody gets a reference to the same copy of the page(s) in memory - until somebody scribbles on a page. So say you have a 30MB shared object in memory, with 5 users. That's 5 references to the same data. Now one user writes to it. The system catches that write (usually via a page fault), copies just the one page to a new page, and then lets the write to the new page complete. Now we have 5 users that all have references to the same (30M-4K) of data, 4 users that have a reference to the old copy of that 4K, and one user that has a reference to the modified copy of that 4K. https://en.wikipedia.org/wiki/Copy-on-write
Hi all! On 23/03/2019 17:59, Bharath Vedartham wrote: [..]
I was wondering how we can overwrite the copy functionality while writing our own filesystem in linux. VFS does not offer any sort of API for copy. I think it calls create and
Yes.
write when we execute the copy the file/dir. I am interested in overwriting the way copy happens in my filesystem(which I am writing for fun :p). Any ideas?
strace a "cp" and see which system calls are used. Look into the source code and see how it's implemented. There is actually more than one method but the usual (while simplest, most versatile and probably fastest) method used is: open source and destination files, read from the source, write into the destination until EOF. MfG, Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at
On Sat, Mar 23, 2019 at 08:03:20PM +0100, Bernd Petrovitsch wrote:
Hi all!
On 23/03/2019 17:59, Bharath Vedartham wrote: [..]
I was wondering how we can overwrite the copy functionality while writing our own filesystem in linux. VFS does not offer any sort of API for copy. I think it calls create and
Yes.
write when we execute the copy the file/dir. I am interested in overwriting the way copy happens in my filesystem(which I am writing for fun :p). Any ideas?
strace a "cp" and see which system calls are used. Look into the source code and see how it's implemented.
There is actually more than one method but the usual (while simplest, most versatile and probably fastest) method used is: open source and destination files, read from the source, write into the destination until EOF.
I have observed that it is a mixture of system calls. I am interested in implementing copy-on-write in my filesystem(for fun). I feel that, it would require my creat vfs api to be aware of the fact that "cp" is calling it. I want to keep things as generic as possible. I ll proceed by reading the source of union filesystems (overlayfs) to see how they implement copy-on-write.
MfG, Bernd -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at
On Sat, Mar 23, 2019 at 10:29:45PM +0530, Bharath Vedartham wrote:
Hey everyone,
I was wondering how we can overwrite the copy functionality while writing our own filesystem in linux. VFS does not offer any sort of API for copy. I think it calls create and write when we execute the copy the file/dir. I am interested in overwriting the way copy happens in my filesystem(which I am writing for fun :p).
There is no "copy" syscall, sorry.
participants (4)
-
Bernd Petrovitsch -
Bharath Vedartham -
Greg KH -
Valdis Klētnieks