Embedded Linux - Mechanism for Deploying Firmware Updates

What are techniques for allowing safe software upgrades in embedded systems

It all depends on how critical the application is. The two basic approaches (backup and bootloader) are also combined sometimes.

Many systems have a read only bootloader (like redboot), and then two banks of flash memory (on the same chip, most often). The bootloader then has a flag to choose which bank to boot from. The flag will then change based on events like upgrades (failed or successful), and so on.

So, when upgrading, the running version copies the new load into the backup bank, checks the checksum, toggles the boot flag, and then reboots the device. The device reboots on the new bank, with the new load. After the reboot, the new load can copy itself into the backup bank.

Often there is also a watchdog timer with a hardware reset. This way, if the firmware goes insane, it fails to kick the watchdog, the hardware reset will reboot the device, and the bootloader will look for a sane load.

The Open Mesh project is a good example of this approach.

Robust software update solutions for an OpenEmbedded/Yocto based system

I'm working on a metadata layer to integrate the Software Updater (swupd) from Clear Linux with the Yocto Project / OpenEmbedded Core.

swupd performs whole of OS updates, rather than package-based updates, using binary deltas to only update the files which change and to do so in an efficient manner.

I recently wrote some documentation (within the docs/Guide.md file in the meta-swupd repo) about adopting the "Clear Linux Way" to utilise meta-swupd from an OE/YP based distro. A wikified version of that guide, including a link to the layer git repository, are available on the Yocto Project wiki:
https://wiki.yoctoproject.org/wiki/Meta-swupd

I also have a sample layer on Github which demonstrates use of the layer (this is also the distro layer I test much of meta-swupd with):
https://github.com/incandescant/meta-myhouse

Microsoft is de-emphasizing Silverlight, should I pick another technology (Flash or HTML 5)?

I would point you to these two posts:

http://timheuer.com/blog/archive/2010/11/01/silverlight-is-dead-long-live-silverlight.aspx
http://team.silverlight.net/announcement/pdc-and-silverlight/

In summary, Silverlight is not going away. Microsoft is recognizing the true cross-platform potential of HTML 5, but the spec for that isn't even complete yet, and we've got years before there is consistent support for it across the most popular browsers. Silverlight is the basis for development on Windows Phone 7, which is a bet-the-farm play for Microsoft.

Speaking for myself, I'm getting ready to develop an entire ERP in Silverlight.

Software patching at a billion miles

I've been a developer on public telephone switching system software, which has pretty severe constraints on reliability, availability, survivability, and performance that approach what spacecraft systems need. I haven't worked on spacecraft (although I did work with many former shuttle programmers while at IBM), and I'm not familiar with VXworks, the operating system used on many spacecraft (including the Mars rovers, which have a phenomenal operating record).

One of the core requirements for patchability is that a system should be designed from the ground up for patching. This includes module structure, so that new variables can be added, and methods replaced, without disrupting current operations. This often means that both old and new code for a changed method will be resident, and the patching operation simply updates the dispatching vector for the class or module.

It is just about mandatory that the patching (and un-patching) software is integrated into the operating system.

When I worked on telephone systems, we generally used patching and module-replacement functions in the system to load and test our new features as well as bug fixes, long before these changes were submitted for builds. Every developer needs to be comfortable with patching and replacing modules as part of their daly work. It builds a level of trust in these components, and makes sure that the patching and replacement code is exercised routinely.

Testing is far more stringent on these systems than anything you've ever encountered on any other project. Complete and partial mock-ups of the deployment system will be readily available. There will likely be virtual machine environments as well, where the complete load can be run and tested. Test plans at all levels above unit test will be written and formally reviewed, just like formal code inspections (and those will be routine as well).

Fault tolerant system design, including software design, is essential. I don't know about spacecraft systems specifically, but something like high-availability clusters is probably standard, with the added capability to run both synchronized and unsynchronized, and with the ability to transfer information between sides during a failover. An added benefit of this system structure is that you can split the system (if necessary), reload the inactive side with a new software load, and test it in the production system without being connected to the system network or bus. When you're satisfied that the new software is running properly, you can simply failover to it.

As with patching, every developer should know how to do failovers, and should do them both during development and testing. In addition, developers should know every software update issue that can force a failover, and should know how to write patches and module replacement that avoid required failovers whenever possible.

In general, these systems are designed from the ground up (hardware, operating system, compilers, and possibly programming language) for these environments. I would not consider Windows, Mac OSX, Linux, or any unix variant, to be sufficiently robust. Part of that is realtime requirements, but the whole issue of reliability and survivability is just as critical.

UPDATE: As another point of interest, here's a blog by one of the Mars rover drivers. This will give you a perspective on the daily life of maintaining an operating spacecraft. Neat stuff!

---

Related Topics