anyone aware of a high availability setup that relies on fully redundant install?

[Andrew Bradford]

Hi Robert,

On 04/17 10:47, Robert P. J. Day wrote:
> 
>   i figure this is as good a place as any to ask ... is anyone here
> aware of anyone using a linux config and install that, for the
> purposes of reliability or high availability or whatever you want to
> call it, relies on a second, completely independent installation of
> linux on the same hard drive?
> 
>   i was having a discussion about the reliability of upgrading linux
> (any combination of kernel, basic rootfs and/or proprietary apps), and
> one approach that was suggested (not by me) was to have two entirely
> independent installations on the same hard drive so that, when one
> "upgraded" the system, the newer content was installed in the other
> partition and the system rebooted to that newer content, the rationale
> being that, if the upgraded system didn't work, one could revert back
> to the original install.
> 
>   the history of this particular approach lies in the fact that the
> original S/W running on the system in question was an embedded app
> programmed right on bare metal, so the entire app was basically a
> monolithic combination of boot loader, kernel and all the apps. in
> that case, i can certainly see how one wanted a reliable way to revert
> back to a known working system if the new S/W had flaws. but i don't
> think that logic holds with a linux installation.
> 
>   the major concern seems to be a fear that, if one upgrades the
> kernel even a little bit, user space apps might suddenly stop working
> because of some new incompatibility introduced by that upgrade. i'm
> well aware of gregkh's piece on kernel stable API nonsense:
> 
> https://www.kernel.org/doc/Documentation/stable_api_nonsense.txt
> 
> so i'm not overly worried, but folks in the discussion coming from the
> embedded programming space are using their experiences from that space
> to argue that a fully redundant install is the safest approach just in
> case something goes wrong with an upgrade.
> 
>   not surprisingly, my position is that this is not something they
> need to worry about a whole lot but, for the sake of fairness, i
> thought i'd look around to see if any linux folks *are* using that
> approach.

Yes, in one product I'm working on now we have the flash partitioned out
with two separate root partitions for this reason.  Doing atomic
upgrades is not easy and this is one simple and effective way to avoid
issues arising out of failed or botched upgrades.  We're more worried
about power going out in the middle of an upgrade moreso than anything
else.

A software upgrade for us wipes the not-booted partition and installs
the entire new system into the not-booted partition, then updates the
bootloader configuration to use that as the root file system.  If the
system fails to boot 3 times (watchdog induced resets) then the
bootloader will fall back to the other partition.

To retain configuration settings across upgrades, there's yet another
partition with config data on it that both rootfs partitions use.  At
boot time the system validates that the config data is in the proper
format and fixes it if it's not.

Thanks,
Andrew