Building device software can be challenging, but managing software at scale adds an entirely different level of complexity to every project. Software at scale must consider platform security, over-the-air updates, per-device content customization, rollback, software supply chain management, etc. This is the DevOps side of building device software. While the huge evolution of cloud-based DevOps demonstrates what’s possible, device vendors, with their unique needs and complexities of managing remote devices, are typically left to solve these problems independently. While KOS isn’t tied to any particular device management solution, it is architected to solve many software at scale problems from the start.
Assemble vs Build
Most software systems are build-based. Developers create a source repository for a device; everyone contributes, and the source tree is built into deliverables. These deliverables are then gathered during another build step and packaged with larger components, such as the operating system image, a monolithic collection of components generated by a build. In such a system, you re-run the entire build to get the final installable image if you want to recreate a release.
KOS replaces this approach with an assembly-based model. A release is not built; it is assembled from known published parts. A KOS release is, in fact, simply a manifest that lists all the parts that make up the release. This is similar to systems like docker, which describe releases as collections of existing components. In the KOS model, recreating a release makes no sense as the manifest is the release, so having the manifest is the same as having the release.
The assembly-based system has many advantages:
- Components can be reused across different devices without copying code or modifying builds.
- Developers only need their small piece of the overall project, as all other components can be assembled from build servers.
- Any individual component can be replaced at any time for testing and diagnostics.
- Device management can add additional components, such as custom content, allowing easy per-device customization.
- Component builds are decoupled and manifests provide an audit trail of the components in a release.
- Assemblies are entirely deterministic, allowing automatic garbage collection of no longer needed components and avoiding problems with long-term disk usage.
The downside of assembly-based software is the infrastructure required to make it work seamlessly. KOS Studio exists to fill this gap and make assembly-based software a seamless experience.
Software Security
Security is complex, expensive, and a requirement for modern devices. Security doesn’t just apply to developer code; it applies to the device boot process and the entire software supply chain, including device management, over-the-air updates, and everything else. Considerable energy has been put into the underlying KOS security model so developers can focus on delivering functionality.
Security in KOS starts with KAB files. Everything from the Linux kernel to Java applications to data files are packaged as KABs. A KAB is a file of files, much like a zip file, that is digitally signed by developer keys and uniquely identified by UUID. Every KAB also has a mode, test, or production based on the keys used to sign it. Releases, defined as manifests, are also packaged as KABs. When KOS boots, it uses the manufacturer certificate to verify the manifest and all referenced KABs in the chain. If any file has been tampered with or doesn’t have the correct authority, the release will be rejected, and the device will roll back to the previous release. KOS can mount KABs directly into the native Linux filesystem, so the contents of KABs are never extracted. Thus, the entire system is immutable. KOS provides delegation models to allow 3rd party KABs to be trusted and support for chaining manifests, allowing external parties to add components to releases in developer-defined ways.
As KABs are signed, typed, uniquely identifiable, and can contain virtually anything, KOS uses KABs as the basic unit of assembly. KOS Studio has extensive support for working with KABs and a KAB registry so that organizations can publish their KABs within their organization, allowing releases to be defined and run without the need to build any of the KABs locally. KOS Studio also allows KABs to be published in a market, allowing organizations to collaborate and share components securely. KOS Studio handles downloading and verifying KABs and installing them onto devices, making the entire process transparent to developers.
Device Customization
Modern devices need to be tailored to customer needs. Consider a device in a hotel lobby that authenticates using a room key or a device located at a particular venue that needs themed content. This often goes beyond custom settings and may include custom content or even unique applications per device. Pulling down custom content or apps opens another channel for security concerns, and changing the state of a device in a way that impacts a rollback release can also impact overall stability. To address these needs, KOS provides a feature called chained manifests. While the developer can define a release with a single manifest, that manifest can be provided to upstream tooling, which can add more kabs in a developer-defined way. This doesn’t modify the original manifest, which is digitally signed and immutable; it creates a new manifest KAB that references the original manifest. Using this process, external systems can generate arbitrary chains of manifests that define the entire state of the device in aggregate. A KOS device can evaluate these chains and construct the final device state from the chain.
Device Updates
One of the most complex aspects of building a device is remote updates. Updating an entire operating system image can be very complex, even when resorting to inefficient strategies like A/B partitions. Beyond the mechanics of picking the correct image to run and getting the image installed, there are challenges around the delivery of updates. How to avoid installing incompatible software, verifying a USB release is not corrupt, and delivering releases over the air onto a device that may be turned off at any time are all complex problems. Extend this to multi-node-based devices, and this is even more complex.
As KOS is assembly-based, a typical update doesn’t involve installing all KABs in the release, as it’s common that many of the required KABs are already available from the previous release. KOS handles identifying the KABs necessary for an installation, garbage collecting existing KABs to make space while ensuring rollback dependencies are preserved, and installing and activating the update. KOS provides extensive support for local updates, such as with a USB drive and remote updates using one or more agents.
When working with USB updates, KOS can examine a disk of KABs, identify all the manifest chains compatible with the device, and provide descriptions of each chain suitable for display to a technician. A given manifest chain can then be installed with a single api call. Once installation on the primary node is complete, all secondary nodes are updated automatically from the primary. This is part of the standard KOS boot process, so even if nodes are turned off during installation or if a failed node is later replaced, they will all synchronize to the correct software version automatically.
Remote updates are somewhat different, however. Utilizing built-in KOS OTA support, an agent needs to specify the UUID of the manifest to be run. If KOS doesn’t have the KAB installed, it will request the KAB from registered agents. It will use this approach to fetch the entire manifest chain and then download all KABs that are missing from the device. Once complete, it will automatically install the update but not activate it. Since OTA can occur at any time of day, updates can disrupt customer operations. If an OTA completes and the customer reboots, they will still run the previous release until the new release is activated at a configured activation time to minimize disruption.
KOS support for software updates, combined with firmware update infrastructure, ensures that device updates are deterministic and controlled. The assembly model also ensures that rollbacks restore the entire device state, firmware included, to guarantee that every state the device is in is the result of a tested combination of components.
Device Agents
While KOS is agent agnostic, it provides much of the core functionality that agents typically provide. A centralized configuration system provides access to all settings, even in multi-node devices and with events that can be used to watch configuration changes. An analytics service provides a standard way to capture rich event data into different channels and notify agents when data needs to be sent off the device. Over-the-air updates are as simple as implementing the ability to download a single KAB at a time. This, along with many other core features in KOS, makes it a very friendly environment for agents.