Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue
On Mon, 30 Aug 2021 at 22:25, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:
Hello,
On Mon, 30 Aug 2021 21:55:19 +0530 Pintu Agarwal <pintu.ping@gmail.com> wrote:
Sorry for coming back to this again.. Unfortunately, none of the options is working for us with squashfs (bootloader, initramfs). initramfs have different kinds of challenges because of the partition size issue. So, our preferred option is still the bootloader command line approach..
Is there a proven and working solution of dm-verity with squashfs ? If yes, please share some references.
The current problem with squashfs is that we could not append the verity-metadata to squashfs, so we store it on a separate volume and access it.
Here, it definitely worked to append the hash tree to the squashfs image and store them in the same partition.
By specifying it like : /dev/mtdblock53
Then we get the error like this: { [ 4.950276] device-mapper: init: attempting early device configuration. [ 4.957577] device-mapper: init: adding target '0 95384 verity 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10 restart_on_corruption ignore_zero_blocks use_fec_from_device /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026' [ 4.975283] device-mapper: verity: sha256 using implementation "sha256-generic" [ 4.998728] device-mapper: init: dm-0 is ready
Could you show the full kernel command line ?
Shared below
Do you see any other problem here with dm-verity cmdline or with squashfs ?
Is squashfs ever proved to be working with dm-verity on higher kernel version ? Currently our kernel version is 4.14.
I confirm we used squashfs on dm-verity successfully. For sure on 4.19, perhaps on older kernels as well.
ohh that means we already have a working reference. If possible can you share the details, even 4.19 or higher will be also a good reference.
Or, another option is to use the new concept from 5.1 kernel that is: dm-mod.create = ? How are you doing it today without dm-mod.create ? I think in 4.14 we don't have dm-mod.create right ?
Again, please give your complete kernel command line.
Here is our kernel command line: [ 0.000000] Kernel command line: ro rootwait console=ttyMSM0,115200,n8 .... verity="95384 11923 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026 " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0 .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10 restart_on_corruption ignore_zero_blocks use_fec_from_device /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ... Do you see any issue here ? Can you share your command line for squashfs to compare ? Thank you, Pintu
Hello, On Mon, 30 Aug 2021 23:48:40 +0530 Pintu Agarwal <pintu.ping@gmail.com> wrote:
ohh that means we already have a working reference. If possible can you share the details, even 4.19 or higher will be also a good reference.
Or, another option is to use the new concept from 5.1 kernel that is: dm-mod.create = ? How are you doing it today without dm-mod.create ? I think in 4.14 we don't have dm-mod.create right ?
No, but you can backport it easily. Back at http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html I provided backports of this feature to OpenWrt, for the 4.14 and 4.19 kernels.
Here is our kernel command line:
[ 0.000000] Kernel command line: ro rootwait console=ttyMSM0,115200,n8 .... verity="95384 11923 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026 " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0 .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10 restart_on_corruption ignore_zero_blocks use_fec_from_device /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...
I don't see how this can work without the dm-mod.create feature. Are you sure the verity= and dm= kernel arguments exist? Best regards, Thomas -- Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hi,
On Tue, 31 Aug 2021 at 00:42, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Mon, 30 Aug 2021 23:48:40 +0530
> Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> > ohh that means we already have a working reference.
> > If possible can you share the details, even 4.19 or higher will be
> > also a good reference.
> >
> > > > Or, another option is to use the new concept from 5.1 kernel that is:
> > > > dm-mod.create = ?
> > > How are you doing it today without dm-mod.create ?
> > I think in 4.14 we don't have dm-mod.create right ?
>
> No, but you can backport it easily. Back at
> http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
> I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
> kernels.
>
Yes, I can backport it to our 4.14 Kernel.
Can you share the list of patches to be backported to make it work on 4.14 ?
If it's backported also I need to report to our internal kernel, but
it might be slightly easier.
Please share the details.
> > Here is our kernel command line:
> >
> > [ 0.000000] Kernel command line: ro rootwait
> > console=ttyMSM0,115200,n8 .... verity="95384 11923
> > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026
> > " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0
> > .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity
> > 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
> > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
> > aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> > restart_on_corruption ignore_zero_blocks use_fec_from_device
> > /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...
>
> I don't see how this can work without the dm-mod.create feature. Are
> you sure the verity= and dm= kernel arguments exist?
Sorry, I am not a security guy and this was done by someone from the
security team.
But, I know that this is already working with ext4.
The moment we change to squashfs, it does not work.
The only difference with squashfs are:
=> verity metadata are kept on separate volume
=> The rootfstype and related stuff are different
=> verity command line related stuff are almost the same.
Also, you mentioned:
>>> Here, it definitely worked to append the hash tree to the squashfs
>>> image and store them in the same partition.
Can you share some details about it ?
How it can be done since squashfs is readonly.
Do, we need to change some parameters during squashfs image generation ?
{
$(STAGING_DIR_HOST)/bin/mksquashfs4 $(call mkfs_target_dir,$(1)) $@ \
- -nopad -noappend -root-owned \
+ -noappend -root-owned \
}
Also, for the above cmdline, is there any problem with the block size ?
As @Mikulas said before that the block size could be the issue
Also, for squashfs we are passing like this for root=. Is it fine ?
rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0
I see that dm-0 is already passed elsewhere so do we really need it ?
I suspect it is not required as a block device.
Thanks,
Pintu
Dear Thomas, Mikulas, Need your help in root causing my dm-verity issue with squashfs. Please see my comments inline. On Tue, 31 Aug 2021 at 18:49, Pintu Agarwal <pintu.ping@gmail.com> wrote:
No, but you can backport it easily. Back at http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html I provided backports of this feature to OpenWrt, for the 4.14 and 4.19 kernels.
Yes, I can backport it to our 4.14 Kernel. Can you share the list of patches to be backported to make it work on 4.14 ? If it's backported also I need to report to our internal kernel, but it might be slightly easier. Please share the details.
I am interested to backport dm-mod.create related patches to our 4.14 kernel. Please let me know where can I find all the patches ? Is it already part of mainline 4.14 ? Please share the list of commits (from mainline) that we need to pull and backport.
Here is our kernel command line:
[ 0.000000] Kernel command line: ro rootwait console=ttyMSM0,115200,n8 .... verity="95384 11923 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026 " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0 .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10 restart_on_corruption ignore_zero_blocks use_fec_from_device /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...
I don't see how this can work without the dm-mod.create feature. Are you sure the verity= and dm= kernel arguments exist?
I checked a little further and yes there is "dm=" command line in kernel available. This is already working with ext4 glue, but was never tried with squashfs. I think it is mainline derived from Android. https://patchwork.kernel.org/project/dm-devel/patch/2c01b2a43a46fab760208d7a... https://github.com/projectceladon/device-androidia-kernel/blob/master/init/d... Mostly, this is the main repo where our source might be derived: https://github.com/android-linux-stable/msm-4.14 Can we backport the patches here ? If I get the list I can try it.
Also, you mentioned:
Here, it definitely worked to append the hash tree to the squashfs image and store them in the same partition. Can you share some details about it ? How it can be done since squashfs is readonly.
Can you share your reference, how are you appending the hash tree ? Let me try the same. But it seems like the underlying concept is the same for both "dm-mod.create" and "dm=". However, I am not sure if there are any changes required for squashfs as block device.. Errors: Currently, we are getting this in boot logs: [ 4.962188] device-mapper: init: attempting early device configuration. [ 4.969699] device-mapper: init: created device '253:0' [ 4.975503] device-mapper: init: adding target '0 95384 verity 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256 8fc2e4bb751f4b3145a486a0f4f1b58149ba3eedc2a67312f31fbee131380dab aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10 restart_on_corruption ignore_zero_blocks use_fec_from_device /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026' [ 4.992323] device-mapper: verity: sha256 using implementation "sha256-generic" [ 5.015568] device-mapper: init: dm-0 is ready [ 10.080065] prepare_namespace: dm_run_setup - done [ 10.080093] prepare_namespace: saved_root_name: /dev/dm-0 [ 10.083903] prepare_namespace: Inside: name_to_dev_t [ 10.089605] prepare_namespace: Calling - mount_root() ... [ 10.094519] [PINTU]: mount_block_root: called with input name: /dev/root, fs_names: squashfs [ 10.263510] [PINTU]: do_mount_root: sys_mount failed: err: -22 [ 10.263544] [PINTU]: mount_block_root: do_mount_root: err: -22, p: squashfs, flags: 32769, root_mount_data: (null) [..] [ 10.745672] No filesystem could mount root, tried: [ 10.745676] squashfs [ 10.748015] [ 10.755232] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(253,0) It seems the rootfs could not mount due to invalid arguments. Not sure which arguments are invalid here... Thanks, Pintu
Hi, On Mon, 6 Sept 2021 at 21:58, Pintu Agarwal <pintu.ping@gmail.com> wrote:
On Tue, 31 Aug 2021 at 18:49, Pintu Agarwal <pintu.ping@gmail.com> wrote:
No, but you can backport it easily. Back at http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html I provided backports of this feature to OpenWrt, for the 4.14 and 4.19 kernels.
Can you please let me know where to get the below patches for backporting to our kernel: create mode 100644 target/linux/generic/backport-4.14/390-dm-add-support-to-directly-boot-to-a-mapped-device.patch create mode 100644 target/linux/generic/backport-4.14/391-dm-init-fix-max-devices-targets-checks.patch create mode 100644 target/linux/generic/backport-4.14/392-dm-ioctl-fix-hang-in-early-create-error-condition.patch create mode 100644 target/linux/generic/backport-4.14/393-Documentation-dm-init-fix-multi-device-example.patch
On Wed, Sep 08, 2021 at 04:57:45PM +0530, Pintu Agarwal wrote:
Hi,
On Mon, 6 Sept 2021 at 21:58, Pintu Agarwal <pintu.ping@gmail.com> wrote:
On Tue, 31 Aug 2021 at 18:49, Pintu Agarwal <pintu.ping@gmail.com> wrote:
No, but you can backport it easily. Back at http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html I provided backports of this feature to OpenWrt, for the 4.14 and 4.19 kernels.
Can you please let me know where to get the below patches for backporting to our kernel: create mode 100644 target/linux/generic/backport-4.14/390-dm-add-support-to-directly-boot-to-a-mapped-device.patch create mode 100644 target/linux/generic/backport-4.14/391-dm-init-fix-max-devices-targets-checks.patch create mode 100644 target/linux/generic/backport-4.14/392-dm-ioctl-fix-hang-in-early-create-error-condition.patch create mode 100644 target/linux/generic/backport-4.14/393-Documentation-dm-init-fix-multi-device-example.patch
If you are stuck on an older kernel version, then you need to get support from the vendor that is forcing you to be on that kernel version, as you are already paying them for support. Please take advantage of that, as no one knows what is really in "your kernel". thanks, greg k-h
Hi All, On Wed, 8 Sept 2021 at 17:38, Greg KH <gregkh@linuxfoundation.org> wrote:
No, but you can backport it easily. Back at http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html I provided backports of this feature to OpenWrt, for the 4.14 and 4.19 kernels.
Can you please let me know where to get the below patches for backporting to our kernel: create mode 100644 target/linux/generic/backport-4.14/390-dm-add-support-to-directly-boot-to-a-mapped-device.patch create mode 100644 target/linux/generic/backport-4.14/391-dm-init-fix-max-devices-targets-checks.patch create mode 100644 target/linux/generic/backport-4.14/392-dm-ioctl-fix-hang-in-early-create-error-condition.patch create mode 100644 target/linux/generic/backport-4.14/393-Documentation-dm-init-fix-multi-device-example.patch
If you are stuck on an older kernel version, then you need to get support from the vendor that is forcing you to be on that kernel version, as you are already paying them for support. Please take advantage of that, as no one knows what is really in "your kernel".
This is to update this thread that now I am able to successfully bring-up dm-verity with NAND+ubiblock+squashfs on our 4.14 kernel itself without backporting the patches. Now, I am able to boot dm-verity using both initramfs and bootloader approach. The initramfs booting issue was our internal issue which was related to Kernel size configuration in UEFI. The bootloader approach issue was related to system image size issue, where we need to pass the exact image size to find the verity-metadata offset at the end of system image. However, I felt that dm-verity documentation still needs to be enhanced further with a better example. With the 5.4 Kernel, I may further explore the option of using dm-mod.create option, then I might get more clarity on how best to use it. Thank you all for your help and support! Regards, Pintu
participants (3)
-
Greg KH -
Pintu Agarwal -
Thomas Petazzoni