Device Updates
One of the most complex aspects of building a device is remote updates. Updating an entire operating system image can be very complex, even when resorting to inefficient strategies like A/B partitions. Beyond the mechanics of picking the correct image to run and getting the image installed, there are challenges around the delivery of updates. How to avoid installing incompatible software, how to verify a USB release is not corrupt, and how to deliver releases over the air onto a device that may be turned off at any time are all complex problems. Extend this to multi-node-based devices, which is even more complex.
As KOS is assembly-based, a typical update doesn’t involve installing all KABs in the release, as it’s common that many of the required KABs are already available from the previous release. KOS handles identifying the KABS necessary for installation, collecting garbage from existing KABS to make space while ensuring rollback dependencies are preserved, and executing the actual process of installing and activating the update. KOS provides extensive support for both local updates, such as with a USB drive, and remote updates using one or more agents.
When working with USB updates, KOS is able to examine a disk of KABs, identify all the manifest chains that are compatible with the device, and provide descriptions of each chain suitable for display to a technician. A given manifest chain can then be installed with a single API call. Once installation on the primary node is complete, all secondary nodes are updated automatically from the primary. This is part of the standard KOS boot process, so even if nodes are turned off during installation or if a failed node is later replaced, they will all synchronize to the correct software version automatically.
Remote updates are somewhat different, however. Utilizing built-in KOS OTA support, an agent simply needs to specify the UUID of the manifest to be run. If KOS doesn’t have the KAB installed, it will request the KAB from registered agents. It will use this approach to fetch the entire manifest chain and then download all KABs that are missing from the device. Once complete, it will automatically install the update but not activate it. Since OTA can occur at any time of day, updates can disrupt customer operations. If an OTA completes and the customer reboots, they will still run the previous release until the new release is activated at a configured activation time in order to minimize disruption.
KOS support for software updates, combined with firmware update infrastructure, ensures that device updates are deterministic and controlled. The assembly model also ensures that rollbacks restore the entire device state, firmware included, to guarantee that every state the device is in is the result of a tested combination of components.
Device Agents
While KOS is agent agnostic, it provides much of the core functionality that agents typically provide. A centralized configuration system provides access to all settings, even in multi-node devices and with events that can be used to watch configuration changes. An analytics service provides a standard way to capture rich event data into different channels and notify agents when data needs to be sent off the device. Over-the-air updates are as simple as implementing the ability to download a single KAB at a time. This, along with many other core features in KOS, makes it a very friendly environment for agents.
Building device software can be challenging, but managing software at scale adds an entirely different level of complexity to every project. Software at scale needs to consider platform security, over-the-air updates, per-device content customization, rollback, software supply chain management, and so on. This is the DevOps side of building device software. While the huge evolution of cloud-based DevOps demonstrates what’s possible, device vendors, with their unique needs and complexities of managing remote devices, are typically left to solve this problem on their own. While KOS isn’t tied to any particular device management solution, it is architected to solve many software-at-scale problems from the start.