Over-The-Air Update

Over-The-Air (OTA) update is a mechanism of distributing software updates over a wireless network without requiring physical access to a device. The target device needs to have a support for the OTA to be able to update wirelessly.

The Qt OTA offering provides tools that assist in enabling OTA update functionality in an embedded linux images build with meta-boot2qt. Generating new updates for OTA enabled devices is completely automated, given an ordinary linux sysroot as an input. This includes OTA updates for linux kernel, system libraries, user space applications, translation fixes, anything that is part of the sysroot. The offering includes Qt/C++ and QML APIs to make integration with your Qt-based application a breeze.

The OTA solution is based on OSTree. If you would like to learn more about OSTree workings refer to the OSTree Documentation. There you can read about the anatomy of an OSTree repository and the deployment system, booting, and other internals of the project, as well as how OSTree compares to other update solutions.

The following blog post series contain additional details on the Qt OTA product:

Over-the-Air Updates, Part 1: Introduction

Features of the Update System

  • Atomic Upgrades (all or nothing) - if an update did not fully complete, for example due to a power failure, the system will boot into an unmodified tree. The currently running tree is never modified, the update will become active on a system reboot.
  • Secure - GPG signing and pinned TLS with client and server side authentication support.
  • Efficient Handling of Disk Space - see the /ostree and the /var in Layout of an OTA Enabled Sysroot.
  • Snapshot-based - traditional package managers (dpkg/rpm) build filesystem trees on the client side. In contrast, the primary focus of OSTree is on replicating trees composed on a server.
  • Bandwidth Optimized - only files that have changed are downloaded. When resuming from an interrupted download, only the missing files are fetched.
  • Configuration Management - see the /etc in Layout of an OTA Enabled Sysroot.
  • Rollback Support - atomically rollback to the previous version (tree) if something goes wrong.
  • Updates Processing in Background - no unnecessary downtime for a user.
  • OS updates via OTA, with support for agnostic application delivery mechanism on top.

Requirements

  1. Filesystem.

    OSTree operates in userspace, and will work on top of any Linux filesystem that supports hard and symbolic links. For OSTree to function reliably, the filesystem needs to be able to recover to a consistent state after an unclean shutdown. Any journaling or log-structured filesystem, when configured properly, is capable of such recovery.

  2. The Linux Kernel.

    The kernel must be configured with initramfs enabled.

    CONFIG_BLK_DEV_INITRD=y

    There are ways around that, but we currently do not support this out of the box:
    https://mail.gnome.org/archives/ostree-list/2015-October/msg00015.html

  3. Boot Loader.

    Supported boot loaders are: U-Boot, GRUB 2.

Quick Start Guide

This guide will lead you through the full workflow of how to use the provided OTA tools.

  • Adding the OTA capability to a device before shipping it to a customer.
  • Generating an update from the new version of your product's sysroot.
  • Delivering this update to a customer device via OTA.
  • Securing a delivery of an update.

Installation

Currently, the OTA package is distributed with Qt Automotive Suite and Qt for Device Creation. The OTA-related files are installed under Tools/ota directory in the main SDK install location, referred to as SDK_INSTALL_DIR in this guide.

When executing scripts, we will refer to the current working directory as WORKDIR. We will be using the qt-ostree tool from the installation. To see a full list of available command line arguments run the following command:

./qt-ostree --help

Instead of providing a full path to qt-ostree each time we refer to it in the guide, we will assume to be already in the SDK_INSTALL_DIR/Tools/ota/qt-ostree directory.

Work on Your Product

Build your product on top of the Boot to Qt stack, or build your own custom embedded linux image. When the image is ready for the first release, continue to the Enabling OTA Functionality on a Device.

Enabling OTA Functionality on a Device

When preparing a device for shipping and subsequent updates are to be delivered via OTA, you first need to enable this feature in the sysroot:

  1. Generate OSTree boot compatible initramfs image.

    OSTree integration code parses the ostree= kernel command line argument in the initramfs to find the correct versioned tree. It setups the necessary mounts, notably the read-only mount on the /usr path, and makes the versioned tree to appear as a real "/" root directory in the booted system. The ostree= argument is passed via the boot loader integration code, explained in more detail later in the guide.

    To generate the initramfs image, run:

    cd SDK_INSTALL_DIR/Tools/ota/dracut/
    ./generate-initramfs

    Important: The device should be powered on, booted into your current product (the sysroot to be released), and connected to a machine from which you will run the generate-initramfs tool. Dracut generates the initramfs image based on the currently running kernel. You can, of course, provide your own (not necessarily dracut based) initramfs, as long as you include the required ostree logic from the provided dracut module.

    Note: The generate-initramfs works on systemd and System V init-based systems.

    This will produce an initramfs-${device}-${release} file in the working directory. The initramfs file will be needed in the later steps.

  2. Boot loader integration.

    OSTree maintains bootloader-independent drop-in configuration files in a format as defined by The Boot Loader Specification. Including a special ostree= kernel argument that allows the initramfs to find and boot the specified tree. Not all boot loaders support The Boot Loader Specification, so OSTree contains code to generate native configurations files from the bootloader-independent configurations.

    The boot script used by your device has to be changed to use the configurations that are managed by OSTree. This will ensure that, after OTA updates or rollbacks, the correct kernel version (and corresponding boot files) will be selected at boot time.

    • U-Boot

      U-Boot tools package is required. In Ubuntu, this can be installed with the following command:

      sudo apt-get install u-boot-tools

      OSTree maintains the uEnv.txt environment file, which the U-Boot environment should import. If custom changes to uEnv.txt are required, use the --uboot-env-file argument from the qt-ostree tool. The provided file will be appended to OSTree's uEnv.txt.

      The content of OSTree's uEnv.txt has the following format:

      kernel_image=/ostree/qt-os-590db09c66551670019a487992f4dae9cb2067e241f7c7fefd6b3d35af55895b/vmlinuz
      bootdir=/ostree/qt-os-590db09c66551670019a487992f4dae9cb2067e241f7c7fefd6b3d35af55895b/
      ramdisk_image=/ostree/qt-os-590db09c66551670019a487992f4dae9cb2067e241f7c7fefd6b3d35af55895b/initramfs
      bootargs=ostree=/ostree/boot.1/qt-os/590db09c66551670019a487992f4dae9cb2067e241f7c7fefd6b3d35af55895b/0

      Where:

      • ${kernel_image}: Path to the Linux kernel image.
      • ${ramdisk_image}: Path to the initramfs image.
      • ${bootargs}: Parameters passed to the kernel command line.
      • ${bootdir}: Path to other files that belong to the same release and should be accessible from U-Boot (DTBs, boot scripts).

      A sample U-Boot logic that uses the imported OSTree's environment variables:

        if ${fs}load ${dtype} ${disk}:${part} ${script} uEnv.txt ; then
          env import -t ${script} ${filesize}
        else
          echo "Error loading uEnv.txt"
          exit
        fi
      
        fdt_file=<device_tree_filename>
      
        ${fs}load ${dtype} ${disk}:${part} ${kernel_addr} ${kernel_image}
        ${fs}load ${dtype} ${disk}:${part} ${fdt_addr} ${bootdir}/${fdt_file}
        ${fs}load ${dtype} ${disk}:${part} ${initramfs_addr} ${ramdisk_image}
      
        # Don't overwrite bootargs set by OSTree in uEnv.txt.
        setenv bootargs ${bootargs} <additional_bootargs>
      
        bootz ${kernel_addr} ${initramfs_addr} ${fdt_addr}
        

      Enabling OSTree support requires minimal effort when using a default boot script as the base. A default boot script here means whatever the device is currently using for booting. The qt-ostree tool does not change the kernel image format, only the path and the file name changes. If the original script uses the bootm command for loading the kernel image, then the OSTree-enabled script should use bootm too.

      Note: You should expect to find all the files that are required for the boot process under the ${bootdir} path. Before starting to write U-Boot integration code, you can run the qt-ostree tool without providing the --uboot-env-file argument and examine the generated sysroot (see step 3).

    • GRUB 2

      Whenever the boot loader configuration files need to be updated on a GRUB 2 based system, OSTree executes ostree-grub-generator to convert bootloader-independent configuration files into native grub.cfg format. A default script, used by the qt-ostree tool is SDK_INSTALL_DIR/Tools/ota/qt-ostree/ostree-grub-generator. You can customize this script to match your requirements and provide it to qt-ostree via --grub2-cfg-generator. The ostree-grub-generator file contains additional details, the script itself is about 40 lines long.

    For more examples refer to Device Integration Examples

  3. Convert your sysroot into an OTA enabled sysroot.

    The conversion is done using the qt-ostree tool.

    sudo ./qt-ostree \
    --sysroot-image-path ${PATH_TO_SYSROOT} \
    --create-ota-sysroot \
    --initramfs ../dracut/initramfs-${device}-${release} \
    --uboot-env-file ../examples/device-integration/nitrogen6x/6x_bootscript

    The generated sysroot can be examined by mounting the boot.${BOOTFS_TYPE} and the rootfs.${ROOTFS_TYPE} filesystem images found in WORKDIR.

    In this guide we assume that the system is based on U-Boot boot loader.

    Notes on the arguments passed to qt-ostree:

    • --sysroot-image-path
      • A path to your sysroot image file(s). Binary images (*.img) and archive images (*.tar.gz) are accepted.
    • --create-ota-sysroot
      • This option tells qt-ostree to create a binary image that contains a bootable OTA enabled sysroot. You will have to deploy the generated image to a device; in this guide, we use an SD card as memory media (see step 4).
    • --initramfs
      • The initramfs image that we generated in the step 1.
    • --uboot-env-file
      • A custom U-Boot boot script or uEnv.txt file, see Boot loader integration. This argument is optional as U-Boot environment can be stored directly on the board's persistent storage dedicated for U-boot environment, or defined when building the U-Boot binary.
  4. Deploy the generated OTA image to an SD card.

    Plug in an SD card or a reader to the development host, and use the following command to find out its device name.

    lsblk -d

    Make sure to unmount all partitions on a device.

    sudo umount /dev/<device_name>?*

    And then deploy the image.

    sudo dd bs=4M if=<image> of=/dev/<device_name> && sync
  5. Test that everything went according to the plan.

    Boot from the SD card and run the following command from the device:

    ostree admin status

    The output should be something similar to:

    * qt-os 36524faa47e33da9dbded2ff99d1df47b3734427b94c8a11e062314ed31442a7.0
        origin refspec: qt-os:linux/qt

    This indicates that the deployment was successful.

    Note: You should also verify that application(s) are working as expected and do not write outside the permitted paths.

Preparing a New Update for an OTA Enabled Device

When preparing a new update for a device that already has OTA enabled, the workflow is as follows:

  1. Work on your sysroot as you normally would. When the product is ready for a release, continue to the next step.
  2. Generate an update.

    This is done by using the qt-ostree tool.

    sudo ./qt-ostree \
    --sysroot-image-path ${PATH_TO_SYSROOT} \
    --initramfs ../dracut/initramfs-${device}-${release} \
    --start-trivial-httpd

    The above command will create a new commit in the OSTree repository at WORKDIR/ostree-repo/, or create a new repository if one does not exist. Use the --ostree-repo argument to provide a custom path. This repository is the OTA update and can be copied to a production server at any time. OSTree repositories can be served via a static HTTP server.

    Notes on the arguments passed to qt-ostree:

    • --initramfs
      • When doing minor releases that do not update the kernel:

        Use the same initramfs that you already have generated for this kernel version in the earlier steps.

      • When doing a major release that updates a kernel:

        It is advised to regenerate initramfs for each new kernel release, so that the kernel and initramfs versions match. On U-Boot systems: If the new kernel/initramfs is not compatible with the boot script on a device, it must be updated as well (see the --uboot-env-file notes below).

    • --sysroot-image-path
      • Provide a path to the new version of your sysroot.
    • --uboot-env-file
      • Updating u-boot environment file is supported only for major releases - when kernel/initramfs versions change.

        The kernel/initramfs version is considered to change when bootcsum changes in the following expression:

        bootcsum=$(cat vmlinuz initramfs | sha256sum | cut -f 1 -d ' ')
    • --start-trivial-httpd
      • Starts a simple web server which you can access on the local host at address specified in WORKDIR/httpd/httpd-address file. Entering this address into a web browser lists the contents of your OSTree repository. This command line argument is useful for quick testing purposes, in production with more advanced requirements (TLS authentication) you will need to use a different web server solution, like Apache.
  3. Use Qt OTA APIs to update devices.
  4. Go back to step 1.

Securing a Delivery of an Update

OTA is a component of a system and not a security framework. If handled correctly (GPG and TLS are used properly, the keys are generated and handled properly and the servers in question are secure to known vulnerabilies and exploits) OTA is considered secure against realistic attacks.

GPG Signing

GPG signing helps to ensure that the data was transmitted in-full, without damage or file corruption and that the data was sent by a trusted party. A set of trusted keys is stored as keyring files on a device. Look for --gpg-* command line arguments in the output of ./qt-ostree --help.

In Ubuntu, the required packages can be installed with the following command:

sudo apt-get install gnupg gnupg2

TLS Authentication

TLS protects data from tampering and eavesdropping. System administrators use this to restrict the access to the server (client authentication) and client devices use this to verify the identitiy of an update server (server authentication). Look for --tls-* command line arguments in the output of ./qt-ostree --help.

It is advised to use both GPG and TLS in hostile environments. To learn more about the security topics from the above list, consult dedicated resources.

Layout of an OTA Enabled Sysroot

There are only two directories on a device for a safe storage of local files: /var and /etc. The sysroot generated by OTA tools adds convenience symbolic links in the / root directory, including symbolic links pointing to the /var.

Important:

  • Do not create or modify files in other locations, these files will be garbage collected on the next upgrade.
  • Do not directly modify the contents of the /ostree and the /boot directory. This can result in a system that fails to boot.
DirectoryDescription
/usr [read‑only]Everything that is part of the OS - mounted read-only to prevent inadvertent corruption. It's recommended that an operating system ships all of its content in /usr. Contents of this directory are updated via OTA.
/etcHost-specific system-wide configuration files. OTA preserves all local changes by doing a 3-way merge.

How a 3-way merge works:

First OSTree checks on the currently booted tree which configuration files have been changed, removed or added by a user by comparing the /etc with the read-only /usr/etc/. The /usr/etc is where OSTree stores default configurations of the tree as composed on the server (each tree has its own read-only copy of the /usr/etc). The advantage of having read-only /usr/etc is that you always have access to system defaults.

Then OSTree takes /etc of the OTA update, which is a separate copy from your running /etc (each tree has its own writable copy of the /etc) as a base and applies your local changes on top. It doesn’t attempt to understand the contents of files – if a configuration file on a device has been modified in any way, that wins.

/varThe only directory that is preserved and shared across upgrades. This is where user and application data should be stored.

Note: OSTree does not update the contents of /var, it is the responsibility of the OS to manage and upgrade /var if required.

/ostreeThe location of the OSTree repository on a device and where the bootable versioned filesystem trees are installed. These trees share all common files via hard links into the OSTree repository. This means each version is deduplicated; an upgrade process only costs disk space proportional to the new files, plus some constant overhead.

Note: /ostree is a symbolic link to /sysroot/ostree.

/bootContains the boot loader configuration files, kernel, initramfs, and other files that are required for the boot process.
/sysrootPhysical / root directory.
/Versioned filesystem tree's root.

Device Integration Examples

Find examples for real embedded devices in the SDK_INSTALL_DIR/Tools/ota/examples/device-integration/ directory.

© 2016 The Qt Company Ltd. Documentation contributions included herein are the copyrights of their respective owners. The documentation provided herein is licensed under the terms of the GNU Free Documentation License version 1.3 as published by the Free Software Foundation. Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.