Implementing an Update/Upgrade System for Embedded Linux Devices

Implementing an update/upgrade system for embedded Linux devices

I believe you are looking wrong at the problem - any update which is non atomic (e.g. dd a file system image, replace files in a directory) is broken by design - if the power goes off in the middle of an update the system is a brick and for embedded system, power can go off in the middle of an upgrade.

I have written a white paper on how to correctly do upgrade/update on embedded Linux systems [1]. It was presented at OLS. You can find the paper here: https://www.kernel.org/doc/ols/2005/ols2005v1-pages-21-36.pdf

[1] Ben-Yossef, Gilad. "Building Murphy-compatible embedded Linux systems." Linux Symposium. 2005.

Step by Step walk through on how to use swupdate on Raspberry Pi or any Embedded board for system update

Here's a good example of OTA using SWUpdate on raspberry pi.
https://mkrak.org/2018/01/26/updating-embedded-linux-devices-part2/

I had to make a few changes to use the latest zeus release. Below is step by step commands on ubuntu 18.04. (This alway worked with master branch as of Mar-22-2020)

Install all required dependencies. (installation script below from https://medium.com/@shantanoodesai/run-docker-on-a-raspberry-pi-4-with-yocto-project-551d6b615c0b)

sudo apt-get update
sudo apt-get install \
     gawk wget git-core diffstat unzip texinfo gcc-multilib \
     build-essential chrpath socat cpio \
     python python3 python3-pip python3-pexpect \
     xz-utils debianutils iputils-ping \
     python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev

Clone all meta-layers

mkdir yocto && cd yocto
mkdir layers && cd layers
git clone git://git.yoctoproject.org/poky -b zeus
git clone git://github.com/openembedded/meta-openembedded.git -b zeus
git clone https://github.com/agherzan/meta-raspberrypi.git -b zeus
git clone https://github.com/sbabic/meta-swupdate -b zeus

git clone https://github.com/sbabic/meta-swupdate-boards.git -b master

cd ..
. layers/poky/oe-init-build-env build

Add layers. If this fails modify build/conf/bblayers.conf manually to include all the layers specified below

bitbake-layers add-layer ../layers/meta-openembedded/meta-oe
bitbake-layers add-layer ../layers/meta-openembedded/meta-python
bitbake-layers add-layer ../layers/meta-openembedded/meta-networking
bitbake-layers add-layer ../layers/meta-openembedded/meta-multimedia
bitbake-layers add-layer ../layers/meta-raspberrypi
bitbake-layers add-layer ../layers/meta-swupdate
bitbake-layers add-layer ../layers/meta-swupdate-boards

Add the following to build/conf/local.conf (Raspberry pi doesn't use uboot bootloader by default. swupdate requires ext4.gz image.)

RPI_USE_U_BOOT = "1"
IMAGE_FSTYPES = "rpi-sdimg ext4.gz"
PREFERRED_PROVIDER_u-boot-fw-utils = "libubootenv"

Now finally bake it. meta-swupdate-boards contains example for a few hardware. I was able to copy raspberrypi3 board implementation to support raspberrypi2 easily.

MACHINE=raspberrypi3 bitbake update-image

This should create core-image-full-cmdline-raspberrypi3.rpi-sdimg and update-image-raspberrypi3.swu files under build/tmp/deploy/image/raspberrypi3/.

Lets burn core-image-full-cmdline-raspberrypi3.rpi-sdimg image to sd card and use update-image-raspberrypi3.swu to update it.

Update to your flash using UI tool like Balena Etcher or via command line. Please note the target file system /dev/disk2 may be different.

sudo dd if=core-image-full-cmdline-raspberrypi3.rpi-sdimg of=/dev/disk2 bs=1m

Once the pi starts up, goto pi_ipaddress:8080. Drag and drop update-image-raspberrypi3.swu to update the firmware.

Linux-Based Firmware, how to implement a good way to update?

I used the following approach. It was somewhat based on the paper "Building Murphy-compatible embedded Linux systems," available here. I used the versions.conf stuff described in that paper, not the cfgsh stuff.

• Use a boot kernel whose job is to loop-back mount the "main" root file system. If you need a newer kernel, then kexec into that newer kernel right after you loop-back mount it. I chose to put the boot kernel's complete init in initramfs, along with busybox and kexec (both statically linked), and my init was a simple shell script that I wrote.
• One or more "main OS" root file systems exist on an "OS image" file system as disk image files. The boot kernel chooses one of these based on a versions.conf file. I only maintain two main OS image files, the current and fall-back file. If the current one fails (more on failure detection later), then the boot kernel boots the fall-back. If both fail or there is no fall-back, the boot kernel provides a shell.
• System config is on a separate partition. This normally isn't upgraded, but there's no reason it couldn't be.
• There are four total partitions: boot, OS image, config, and data. The data partition is for user application stuff that is intended for frequent writing. boot is never mounted read/write. OS image is only (re-)mounted read/write during upgrades. config is only mounted read/write when config stuff needs to change (hopefully never). data is always mounted read/write.
• The disk image files each contain a full Linux system, including a kernel, init scripts, user programs (e.g. busybox, product applications), and a default config that is copied to the config partition on the first boot. The files are whatever size is necessary to fit everything in them. As long I allowed enough room for growth so that the OS image partition is always big enough to fit three main OS image files (during an upgrade, I don't delete the old fall-back until the new one is extracted), I can allow for the main OS image to grow as needed. These image files are always (loop-back) mounted read-only. Using these files also takes out the pain of dealing with failures of upgrading individual files within a rootfs.
• Upgrades are done by transferring a self-extracting tarball to a tmpfs. The beginning of this script remounts the OS image read/write, then extracts the new main OS image to the OS image file system, and then updates the versions.conf file (using the rename method described in the "murphy" paper). After this is done, I touch a stamp file indicating an upgrade has happened, then reboot.
• The boot kernel looks for this stamp file. If it finds it, it moves it to another stamp file, then boots the new main OS image file. The main OS image file is expected to remove the stamp file when it starts successfully. If it doesn't, the watchdog will trigger a reboot, and then the boot kernel will see this and detect a failure.
• You will note there are a few possible points of failure during an upgrade: syncing the versions.conf during the upgrade, and touching/removing the stamp files (three instances). I couldn't find a way to reduce these further and achieve everything I wanted. If anyone has a better suggestion, I'd love to hear it. File system errors or power failures while writing the OS image could also occur, but I'm hoping the ext3 file system will provide some chance of surviving in that case.

---

Related Topics