anyone aware of a high availability setup that relies on fully redundant install?
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive? i was having a discussion about the reliability of upgrading linux (any combination of kernel, basic rootfs and/or proprietary apps), and one approach that was suggested (not by me) was to have two entirely independent installations on the same hard drive so that, when one "upgraded" the system, the newer content was installed in the other partition and the system rebooted to that newer content, the rationale being that, if the upgraded system didn't work, one could revert back to the original install. the history of this particular approach lies in the fact that the original S/W running on the system in question was an embedded app programmed right on bare metal, so the entire app was basically a monolithic combination of boot loader, kernel and all the apps. in that case, i can certainly see how one wanted a reliable way to revert back to a known working system if the new S/W had flaws. but i don't think that logic holds with a linux installation. the major concern seems to be a fear that, if one upgrades the kernel even a little bit, user space apps might suddenly stop working because of some new incompatibility introduced by that upgrade. i'm well aware of gregkh's piece on kernel stable API nonsense: https://www.kernel.org/doc/Documentation/stable_api_nonsense.txt so i'm not overly worried, but folks in the discussion coming from the embedded programming space are using their experiences from that space to argue that a fully redundant install is the safest approach just in case something goes wrong with an upgrade. not surprisingly, my position is that this is not something they need to worry about a whole lot but, for the sake of fairness, i thought i'd look around to see if any linux folks *are* using that approach. anyone? thoughts? rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================
This depends on environment proposes. It,s aways a good thing to have a recovery boot option that can boot a minimum shell to dignoses whats going wrong but a full instalation for that is even useful?? IMHO is not a good idea, since you will have to maintain two instalations per hardware. You may be looking for something like update rollback. I know that yum does this and that smart package manager has this feature two, but I never tested it neither one :) Other option is rely on filesystem for such thing. btrfs has some nice snapshot features, you may want to take a look. For recovery boot option you can have a network boot, loading a complete distinct kernel and rootfs that can be chrooted to in drive instalation for recovery when everything goes wrong. Also, the "remote system" can be maintained independently of host instalation. With RO rootfs you can even boot multiple machines with same kernel/rootfs simultaneously, and maintain one recovery installation for multiple "host machines". The better aproach depends on your enviroment, what you have avaible for recovery and how you want to proceed when something goes wrong. Regards Enviado do meu smartphone BlackBerry 10. Mensagem original De: Robert P. J. Day Enviada: domingo, 17 de abril de 2016 11:48 Para: Kernel Newbies Assunto: anyone aware of a high availability setup that relies on fully redundant install? i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive? i was having a discussion about the reliability of upgrading linux (any combination of kernel, basic rootfs and/or proprietary apps), and one approach that was suggested (not by me) was to have two entirely independent installations on the same hard drive so that, when one "upgraded" the system, the newer content was installed in the other partition and the system rebooted to that newer content, the rationale being that, if the upgraded system didn't work, one could revert back to the original install. the history of this particular approach lies in the fact that the original S/W running on the system in question was an embedded app programmed right on bare metal, so the entire app was basically a monolithic combination of boot loader, kernel and all the apps. in that case, i can certainly see how one wanted a reliable way to revert back to a known working system if the new S/W had flaws. but i don't think that logic holds with a linux installation. the major concern seems to be a fear that, if one upgrades the kernel even a little bit, user space apps might suddenly stop working because of some new incompatibility introduced by that upgrade. i'm well aware of gregkh's piece on kernel stable API nonsense: https://www.kernel.org/doc/Documentation/stable_api_nonsense.txt so i'm not overly worried, but folks in the discussion coming from the embedded programming space are using their experiences from that space to argue that a fully redundant install is the safest approach just in case something goes wrong with an upgrade. not surprisingly, my position is that this is not something they need to worry about a whole lot but, for the sake of fairness, i thought i'd look around to see if any linux folks *are* using that approach. anyone? thoughts? rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
On Sun, Apr 17, 2016 at 10:47:55AM -0400, Robert P. J. Day wrote:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive?
ChromeOS and CoreOS do this. There's lots of documentation on the ChromeOS site for how this works and what is involved.
On Sun, 17 Apr 2016, Greg KH wrote:
On Sun, Apr 17, 2016 at 10:47:55AM -0400, Robert P. J. Day wrote:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive?
ChromeOS and CoreOS do this. There's lots of documentation on the ChromeOS site for how this works and what is involved.
interesting ... does the CoreOS functionality depend on containers? what little i know about CoreOS, i would think there's no way to use it *without* containers. rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================
On Sun, Apr 17, 2016 at 01:16:10PM -0400, Robert P. J. Day wrote:
On Sun, 17 Apr 2016, Greg KH wrote:
On Sun, Apr 17, 2016 at 10:47:55AM -0400, Robert P. J. Day wrote:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive?
ChromeOS and CoreOS do this. There's lots of documentation on the ChromeOS site for how this works and what is involved.
interesting ... does the CoreOS functionality depend on containers?
Nope, not at all, it's the same design as ChromeOS.
what little i know about CoreOS, i would think there's no way to use it *without* containers.
Not true at all, you can use it without containers, it's just not all that useful as odds are, the application you want to use isn't included in it so you need a container for that. greg k-h
On Sun, 17 Apr 2016 10:47:55 -0400, "Robert P. J. Day" said:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive?
IBM's AIX has for a long time had the concept of an 'alternate boot volume', which can be another logical volume on the same physical hard drive. But it's not intended for high-availability, it's for "if this software upgrade goes pear-shaped I have an easy backout procedure". And it avoids most of the "you have to keep two version" issues by providing a tool to copy your *current* system onto the alternate boot. I'm sure some Linux distros have stolen the concept. Most implementations of "high availability" would see the phrase "on the same hard drive" and start pointing and laughing at the single point of failure.
On Mon, 18 Apr 2016, Valdis.Kletnieks@vt.edu wrote:
On Sun, 17 Apr 2016 10:47:55 -0400, "Robert P. J. Day" said:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive?
IBM's AIX has for a long time had the concept of an 'alternate boot volume', which can be another logical volume on the same physical hard drive. But it's not intended for high-availability, it's for "if this software upgrade goes pear-shaped I have an easy backout procedure". And it avoids most of the "you have to keep two version" issues by providing a tool to copy your *current* system onto the alternate boot. I'm sure some Linux distros have stolen the concept.
that makes sense -- a *minimal* bootable system for recovery and troubleshooting. but not a fully independent previous install.
Most implementations of "high availability" would see the phrase "on the same hard drive" and start pointing and laughing at the single point of failure.
trust me, i'm aware of that. :-) perhaps i shouldn't have used the phrase "high availability", this proposal was more for just the ability to back out of a botched or flawed upgrade. rday -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ========================================================================
On Mon, 18 Apr 2016 06:29:21 -0400, "Robert P. J. Day" said:
that makes sense -- a *minimal* bootable system for recovery and troubleshooting. but not a fully independent previous install.
No, it's a *complete* system - your kernel boot image, /, /usr, /var, and whatever other file systems you specified to go into 'rootvg'. The use case is that before doing maintenance or whatever, you run a shell script that clones the entire rootvg over to alt_rootvg. Or you can apply system updates to the alternate boot, so instead of taking an outage for 2 hours to apply all your fixes, you apply fixes onto the alternate, and your outage window is only what it takes to reboot to the updated image - and you still have the old image to fall back to. Incredibly useful for that set of systems that you need to minimize downtime, but the application in question isn't one that you can run multiple instances behind a load balancer. We've used it for everything from a Listserv server to a TSM backup server. http://sureshaix.blogspot.com/2008/07/alternate-disk-installation.html http://www.drdobbs.com/aix-alternate-disk-installation/199101222 (Yes, that's an article from 1991.. :)
On 4/18/16 12:48 AM, Valdis.Kletnieks@vt.edu wrote:
On Sun, 17 Apr 2016 10:47:55 -0400, "Robert P. J. Day" said:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive? <snip>
Most implementations of "high availability" would see the phrase "on the same hard drive" and start pointing and laughing at the single point of failure.
Yup. That was my reaction. HA starts with either 2 servers and either a) mirrored drives (e.g., using DRBD) or b) a HA RAID cluster. Miles Fidelman
Hi Robert, On 04/17 10:47, Robert P. J. Day wrote:
i figure this is as good a place as any to ask ... is anyone here aware of anyone using a linux config and install that, for the purposes of reliability or high availability or whatever you want to call it, relies on a second, completely independent installation of linux on the same hard drive?
i was having a discussion about the reliability of upgrading linux (any combination of kernel, basic rootfs and/or proprietary apps), and one approach that was suggested (not by me) was to have two entirely independent installations on the same hard drive so that, when one "upgraded" the system, the newer content was installed in the other partition and the system rebooted to that newer content, the rationale being that, if the upgraded system didn't work, one could revert back to the original install.
the history of this particular approach lies in the fact that the original S/W running on the system in question was an embedded app programmed right on bare metal, so the entire app was basically a monolithic combination of boot loader, kernel and all the apps. in that case, i can certainly see how one wanted a reliable way to revert back to a known working system if the new S/W had flaws. but i don't think that logic holds with a linux installation.
the major concern seems to be a fear that, if one upgrades the kernel even a little bit, user space apps might suddenly stop working because of some new incompatibility introduced by that upgrade. i'm well aware of gregkh's piece on kernel stable API nonsense:
https://www.kernel.org/doc/Documentation/stable_api_nonsense.txt
so i'm not overly worried, but folks in the discussion coming from the embedded programming space are using their experiences from that space to argue that a fully redundant install is the safest approach just in case something goes wrong with an upgrade.
not surprisingly, my position is that this is not something they need to worry about a whole lot but, for the sake of fairness, i thought i'd look around to see if any linux folks *are* using that approach.
Yes, in one product I'm working on now we have the flash partitioned out with two separate root partitions for this reason. Doing atomic upgrades is not easy and this is one simple and effective way to avoid issues arising out of failed or botched upgrades. We're more worried about power going out in the middle of an upgrade moreso than anything else. A software upgrade for us wipes the not-booted partition and installs the entire new system into the not-booted partition, then updates the bootloader configuration to use that as the root file system. If the system fails to boot 3 times (watchdog induced resets) then the bootloader will fall back to the other partition. To retain configuration settings across upgrades, there's yet another partition with config data on it that both rootfs partitions use. At boot time the system validates that the config data is in the proper format and fixes it if it's not. Thanks, Andrew
participants (6)
-
Andrew Bradford -
Daniel -
Greg KH -
Miles Fidelman -
Robert P. J. Day -
Valdis.Kletnieks＠vt.edu