why does an in-tree loadable module taint the kernel
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel [ 7.449072] CPU: 0 PID: 203 Comm: systemd-udevd Tainted: G E 5.13.0-rc6-lm1-00004-g28dc6f490a7f #106
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine. thanks, greg k-h
On Mon, Jun 14, 2021 at 1:20 AM Greg KH <greg@kroah.com> wrote:
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine.
OK, I understand better now - its nothing wrong with serio_raw, its just the 1st module to load, and warning comes just once. kernel/module.c 3962: pr_notice_once("%s: module verification failed: signature " Whats odd is that the same module has a signature when modinfo'd in the kernel running the laptop, but not from the same kernel running inside a VM. Does this constitute a bug of some sort ? A pretty small one, likely fixable by changing CONFIG_SECURITY etc. For anyone else needing to decode a taint: https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
thanks,
greg k-h
thank you
On Tue, Jun 15, 2021 at 10:06:08AM -0600, jim.cromie@gmail.com wrote:
On Mon, Jun 14, 2021 at 1:20 AM Greg KH <greg@kroah.com> wrote:
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine.
OK, I understand better now -
its nothing wrong with serio_raw, its just the 1st module to load, and warning comes just once. kernel/module.c 3962: pr_notice_once("%s: module verification failed: signature "
Whats odd is that the same module has a signature when modinfo'd in the kernel running the laptop, but not from the same kernel running inside a VM. Does this constitute a bug of some sort ?
I do not understand, what is different here and what is not working properly? If you rebuild modules for a kernel without having the key, yes, you will get this warning. You have to have the same key here. thanks, greg k-h
On Tue, Jun 15, 2021 at 10:24 AM Greg KH <greg@kroah.com> wrote:
On Tue, Jun 15, 2021 at 10:06:08AM -0600, jim.cromie@gmail.com wrote:
On Mon, Jun 14, 2021 at 1:20 AM Greg KH <greg@kroah.com> wrote:
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine.
OK, I understand better now -
its nothing wrong with serio_raw, its just the 1st module to load, and warning comes just once. kernel/module.c 3962: pr_notice_once("%s: module verification failed: signature "
Whats odd is that the same module has a signature when modinfo'd in the kernel running the laptop, but not from the same kernel running inside a VM. Does this constitute a bug of some sort ?
I do not understand, what is different here and what is not working properly?
I have built and installed 5.13-rc6 onto my laptop, Im running it now. When I modinfo something, it shows a signature [jimc@frodo ~]$ modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload sig_id: PKCS#7 signer: Build time autogenerated kernel key sig_key: 73:9F:4D:24:D7:05:0A:55:AE:5C:B1:F6:52:B1:BA:E0:5C:68:32:36 sig_hashalgo: sha512 signature: 47:10:D7:A0:79:BE:B5:24:B1:BE:7F:53:8D:EF:4E:73:BD:39:5C:B4: CB:7A:CD:3F:C8:96:E4:7A:72:17:A0:2B:42:63:5A:0F:F6:8B:70:7E: ... when I run precisely the same kernel inside a virtme/kvm/qemu VM, the same modinfo lacks that sig stuff Note that vermagic matches exactly bash-5.1# modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload bash-5.1#
If you rebuild modules for a kernel without having the key, yes, you will get this warning. You have to have the same key here.
heres how Ive configured: - copy distro .config from /boot (Fedora) - make localmodconfig (to drop building parts I wont need) - virtme-configkernel --update (to get support for 9P, virtio etc to mount host disks) all the SECURITY stuff came from the distro config, I havent yet tried to unconfigure it. I havent done anything specific with keys, I dont know why whatever key is involved is not available for both scenarios. here's the relevant (I hope) config items: [jimc@frodo local-i915m]$ grep SALT .config CONFIG_BUILD_SALT="5.8.12-200.fc32.x86_64" [jimc@frodo local-i915m]$ grep _KEY .config | grep -v '#' CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y CONFIG_INPUT_KEYBOARD=y CONFIG_KEYBOARD_ATKBD=y CONFIG_KEYS=y CONFIG_KEYS_REQUEST_CACHE=y CONFIG_PERSISTENT_KEYRINGS=y CONFIG_ENCRYPTED_KEYS=y CONFIG_KEY_DH_OPERATIONS=y CONFIG_KEY_NOTIFICATIONS=y CONFIG_INTEGRITY_ASYMMETRIC_KEYS=y CONFIG_INTEGRITY_TRUSTED_KEYRING=y CONFIG_INTEGRITY_PLATFORM_KEYRING=y CONFIG_LOAD_UEFI_KEYS=y CONFIG_IMA_KEYRINGS_PERMIT_SIGNED_BY_BUILTIN_OR_SECONDARY=y CONFIG_IMA_MEASURE_ASYMMETRIC_KEYS=y CONFIG_IMA_QUEUE_EARLY_BOOT_KEYS=y CONFIG_ASYMMETRIC_KEY_TYPE=y CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y CONFIG_MODULE_SIG_KEY="certs/signing_key.pem" CONFIG_SYSTEM_TRUSTED_KEYRING=y CONFIG_SYSTEM_TRUSTED_KEYS="" CONFIG_SECONDARY_TRUSTED_KEYRING=y CONFIG_SYSTEM_BLACKLIST_KEYRING=y [jimc@frodo local-i915m]$ [jimc@frodo local-i915m]$ grep SECURITY .config | grep -v '#' CONFIG_IP_NF_SECURITY=m CONFIG_IP6_NF_SECURITY=m CONFIG_EXT4_FS_SECURITY=y CONFIG_SECURITY=y CONFIG_SECURITY_WRITABLE_HOOKS=y CONFIG_SECURITYFS=y CONFIG_SECURITY_NETWORK=y CONFIG_SECURITY_NETWORK_XFRM=y CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_SECURITY_SELINUX_DISABLE=y CONFIG_SECURITY_SELINUX_DEVELOP=y CONFIG_SECURITY_SELINUX_AVC_STATS=y CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=0 CONFIG_SECURITY_SELINUX_SIDTAB_HASH_BITS=9 CONFIG_SECURITY_SELINUX_SID2STR_CACHE_SIZE=256 CONFIG_SECURITY_YAMA=y CONFIG_SECURITY_LOCKDOWN_LSM=y CONFIG_SECURITY_LOCKDOWN_LSM_EARLY=y CONFIG_DEFAULT_SECURITY_SELINUX=y
On Tue, Jun 15, 2021 at 12:26:19PM -0600, jim.cromie@gmail.com wrote:
On Tue, Jun 15, 2021 at 10:24 AM Greg KH <greg@kroah.com> wrote:
On Tue, Jun 15, 2021 at 10:06:08AM -0600, jim.cromie@gmail.com wrote:
On Mon, Jun 14, 2021 at 1:20 AM Greg KH <greg@kroah.com> wrote:
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine.
OK, I understand better now -
its nothing wrong with serio_raw, its just the 1st module to load, and warning comes just once. kernel/module.c 3962: pr_notice_once("%s: module verification failed: signature "
Whats odd is that the same module has a signature when modinfo'd in the kernel running the laptop, but not from the same kernel running inside a VM. Does this constitute a bug of some sort ?
I do not understand, what is different here and what is not working properly?
I have built and installed 5.13-rc6 onto my laptop, Im running it now. When I modinfo something, it shows a signature
[jimc@frodo ~]$ modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload sig_id: PKCS#7 signer: Build time autogenerated kernel key sig_key: 73:9F:4D:24:D7:05:0A:55:AE:5C:B1:F6:52:B1:BA:E0:5C:68:32:36 sig_hashalgo: sha512 signature: 47:10:D7:A0:79:BE:B5:24:B1:BE:7F:53:8D:EF:4E:73:BD:39:5C:B4: CB:7A:CD:3F:C8:96:E4:7A:72:17:A0:2B:42:63:5A:0F:F6:8B:70:7E: ...
when I run precisely the same kernel inside a virtme/kvm/qemu VM, the same modinfo lacks that sig stuff Note that vermagic matches exactly
bash-5.1# modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload bash-5.1#
Are you sure the modinfo you are running inside the vm knows to read the signature information? Odds are a busybox/toybox modinfo does not know this type of thing. Let me check: $ ./busybox modinfo visor | grep signature $ modinfo visor | grep signature signature: 51:F5:13:E1:F9:49:FA:20:68:45:F8:32:67:E2:4D:9C:DD:B6:55:EA: Yup, busybox does not know about these things. Is the md5sum the same for these modules?
If you rebuild modules for a kernel without having the key, yes, you will get this warning. You have to have the same key here.
heres how Ive configured: - copy distro .config from /boot (Fedora) - make localmodconfig (to drop building parts I wont need) - virtme-configkernel --update (to get support for 9P, virtio etc to mount host disks)
all the SECURITY stuff came from the distro config, I havent yet tried to unconfigure it.
I havent done anything specific with keys, I dont know why whatever key is involved is not available for both scenarios. here's the relevant (I hope) config items:
Look at the CONFIG_MODULE_SIG* items, that's the relevant ones. Here's what I use in my custom kernels for my systems: $ zcat /proc/config.gz | grep MODULE_SIG CONFIG_MODULE_SIG_FORMAT=y CONFIG_MODULE_SIG=y CONFIG_MODULE_SIG_FORCE=y CONFIG_MODULE_SIG_ALL=y # CONFIG_MODULE_SIG_SHA1 is not set # CONFIG_MODULE_SIG_SHA224 is not set # CONFIG_MODULE_SIG_SHA256 is not set # CONFIG_MODULE_SIG_SHA384 is not set CONFIG_MODULE_SIG_SHA512=y CONFIG_MODULE_SIG_HASH="sha512" CONFIG_MODULE_SIG_KEY="certs/signing_key.pem" Watch out when using MODULE_SIG_FORCE, that requires you to sign all modules. If you want to keep a keyset around when building custom kernels, that's fine, but I delete mine instantly after signing the kernel so that it's impossible (well harder) to load modules that were not signed as part of my local build process. I do that by doing: rm certs/signing_key.* after installing the kernel. Hope this helps, greg k-h
On Wed, Jun 16, 2021 at 2:05 AM Greg KH <greg@kroah.com> wrote:
On Tue, Jun 15, 2021 at 12:26:19PM -0600, jim.cromie@gmail.com wrote:
On Tue, Jun 15, 2021 at 10:24 AM Greg KH <greg@kroah.com> wrote:
On Tue, Jun 15, 2021 at 10:06:08AM -0600, jim.cromie@gmail.com wrote:
On Mon, Jun 14, 2021 at 1:20 AM Greg KH <greg@kroah.com> wrote:
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine.
OK, I understand better now -
its nothing wrong with serio_raw, its just the 1st module to load, and warning comes just once. kernel/module.c 3962: pr_notice_once("%s: module verification failed: signature "
Whats odd is that the same module has a signature when modinfo'd in the kernel running the laptop, but not from the same kernel running inside a VM. Does this constitute a bug of some sort ?
I do not understand, what is different here and what is not working properly?
I have built and installed 5.13-rc6 onto my laptop, Im running it now. When I modinfo something, it shows a signature
[jimc@frodo ~]$ modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload sig_id: PKCS#7 signer: Build time autogenerated kernel key sig_key: 73:9F:4D:24:D7:05:0A:55:AE:5C:B1:F6:52:B1:BA:E0:5C:68:32:36 sig_hashalgo: sha512 signature: 47:10:D7:A0:79:BE:B5:24:B1:BE:7F:53:8D:EF:4E:73:BD:39:5C:B4: CB:7A:CD:3F:C8:96:E4:7A:72:17:A0:2B:42:63:5A:0F:F6:8B:70:7E: ...
when I run precisely the same kernel inside a virtme/kvm/qemu VM, the same modinfo lacks that sig stuff Note that vermagic matches exactly
bash-5.1# modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload bash-5.1#
Are you sure the modinfo you are running inside the vm knows to read the signature information? Odds are a busybox/toybox modinfo does not know this type of thing. Let me check:
$ ./busybox modinfo visor | grep signature $ modinfo visor | grep signature signature: 51:F5:13:E1:F9:49:FA:20:68:45:F8:32:67:E2:4D:9C:DD:B6:55:EA:
Yup, busybox does not know about these things.
that is interesting. Its not the reason here, IIUC. Im using virtme for my VM, its big advantage is that it remounts the host system disks, so you get your whole home/host environment unchanged. Theres also no need to install a kernel to run it in the VM, I just happen to have installed the same kernel to run the whole laptop, (more to test my patches than to chase down this particular weirdness)
Is the md5sum the same for these modules?
Yes. module is same, so is cksum /sbin/modinfo
If you rebuild modules for a kernel without having the key, yes, you will get this warning. You have to have the same key here.
heres how Ive configured: - copy distro .config from /boot (Fedora) - make localmodconfig (to drop building parts I wont need) - virtme-configkernel --update (to get support for 9P, virtio etc to mount host disks)
all the SECURITY stuff came from the distro config, I havent yet tried to unconfigure it.
I havent done anything specific with keys, I dont know why whatever key is involved is not available for both scenarios. here's the relevant (I hope) config items:
Look at the CONFIG_MODULE_SIG* items, that's the relevant ones.
Here's what I use in my custom kernels for my systems:
$ zcat /proc/config.gz | grep MODULE_SIG CONFIG_MODULE_SIG_FORMAT=y CONFIG_MODULE_SIG=y CONFIG_MODULE_SIG_FORCE=y CONFIG_MODULE_SIG_ALL=y # CONFIG_MODULE_SIG_SHA1 is not set # CONFIG_MODULE_SIG_SHA224 is not set # CONFIG_MODULE_SIG_SHA256 is not set # CONFIG_MODULE_SIG_SHA384 is not set CONFIG_MODULE_SIG_SHA512=y CONFIG_MODULE_SIG_HASH="sha512" CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
Watch out when using MODULE_SIG_FORCE, that requires you to sign all modules.
If you want to keep a keyset around when building custom kernels, that's fine, but I delete mine instantly after signing the kernel so that it's impossible (well harder) to load modules that were not signed as part of my local build process. I do that by doing: rm certs/signing_key.* after installing the kernel.
Hope this helps,
It does. My original concern was that the tainting then disabled lockdep, and might disable other things. (I disable kalsr on cmdline in vm) Im just gonna drop all the signing ;-) thanks
greg k-h
On Wed, Jun 16, 2021 at 11:50 AM <jim.cromie@gmail.com> wrote:
On Wed, Jun 16, 2021 at 2:05 AM Greg KH <greg@kroah.com> wrote:
On Tue, Jun 15, 2021 at 12:26:19PM -0600, jim.cromie@gmail.com wrote:
On Tue, Jun 15, 2021 at 10:24 AM Greg KH <greg@kroah.com> wrote:
On Tue, Jun 15, 2021 at 10:06:08AM -0600, jim.cromie@gmail.com wrote:
On Mon, Jun 14, 2021 at 1:20 AM Greg KH <greg@kroah.com> wrote:
On Mon, Jun 14, 2021 at 01:09:25AM -0600, jim.cromie@gmail.com wrote:
serio_raw is apparently tainting the kernel when its modprobed. why ? other modules load properly, no code changes to this module
bash-5.1# dmesg | grep -i taint [ 6.517150] serio_raw: module verification failed: signature and/or required key missing - tainting kernel
You did not build this with the correct module signing key that your kernel was built with. That is what this warning is showing you, try building all modules with the same key as your kernel had and you should be fine.
OK, I understand better now -
its nothing wrong with serio_raw, its just the 1st module to load, and warning comes just once. kernel/module.c 3962: pr_notice_once("%s: module verification failed: signature "
Whats odd is that the same module has a signature when modinfo'd in the kernel running the laptop, but not from the same kernel running inside a VM. Does this constitute a bug of some sort ?
I do not understand, what is different here and what is not working properly?
I have built and installed 5.13-rc6 onto my laptop, Im running it now. When I modinfo something, it shows a signature
[jimc@frodo ~]$ modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload sig_id: PKCS#7 signer: Build time autogenerated kernel key sig_key: 73:9F:4D:24:D7:05:0A:55:AE:5C:B1:F6:52:B1:BA:E0:5C:68:32:36 sig_hashalgo: sha512 signature: 47:10:D7:A0:79:BE:B5:24:B1:BE:7F:53:8D:EF:4E:73:BD:39:5C:B4: CB:7A:CD:3F:C8:96:E4:7A:72:17:A0:2B:42:63:5A:0F:F6:8B:70:7E: ...
when I run precisely the same kernel inside a virtme/kvm/qemu VM, the same modinfo lacks that sig stuff Note that vermagic matches exactly
bash-5.1# modinfo pcspkr filename: /lib/modules/5.13.0-rc6-lm1-00004-g28dc6f490a7f/kernel/drivers/input/misc/pcspkr.ko alias: platform:pcspkr license: GPL description: PC Speaker beeper driver author: Vojtech Pavlik <vojtech@ucw.cz> depends: retpoline: Y intree: Y name: pcspkr vermagic: 5.13.0-rc6-lm1-00004-g28dc6f490a7f SMP mod_unload bash-5.1#
Are you sure the modinfo you are running inside the vm knows to read the signature information? Odds are a busybox/toybox modinfo does not know this type of thing. Let me check:
$ ./busybox modinfo visor | grep signature $ modinfo visor | grep signature signature: 51:F5:13:E1:F9:49:FA:20:68:45:F8:32:67:E2:4D:9C:DD:B6:55:EA:
Yup, busybox does not know about these things.
that is interesting. Its not the reason here, IIUC.
Im using virtme for my VM, its big advantage is that it remounts the host system disks, so you get your whole home/host environment unchanged. Theres also no need to install a kernel to run it in the VM, I just happen to have installed the same kernel to run the whole laptop, (more to test my patches than to chase down this particular weirdness)
virtme-init: basic initialization done virtme-init: running systemd-tmpfiles Failed to create directory or subvolume "/var/spool/cups/tmp": Permission denied Failed to open directory 'portables': Permission denied Failed to open directory 'machines': Permission denied Failed to open directory 'private': Permission denied Failed to open directory 'private': Permission denied systemd-tmpfile (184) used greatest stack depth: 26448 bytes left virtme-init: starting udevd Starting version v248.3-1.fc34 Im gonna handwave a key visibilty problem causing my wierdness
participants (2)
-
Greg KH -
jim.cromie＠gmail.com