Computing stuff tied to the physical world

Over-The-Air update

by Thorsten von Eicken

When building esp-link one of the first features added to the original esphttpd web server software was an over-the-air (OTA) update capability. This allows the firmware that the esp8266 runs to be updated via WiFi and makes firmware development much more pleasant.

Flash layout

The basic idea for OTA update is very simple: use only half the flash space with the firmware such that two partitions can be created. When the firmware is running out of partition 1 then partition 2 can be loaded with a new version and a restart can make the esp8266 run out of the new partition 2. At that point partition 1 can be updated, and so forth.

This scheme is built into the boot loader provided by Espressif. The bootloader enforces the following flash layout (all this assumes a 512KB flash chip and is described in Espressif’s 99C-ESP8266__OTA_Upgrade__EN_v1.5.pdf document):

  • 0x00000 4KB bootloader
  • 0x01000 236KB partition1
  • 0x3E000 16KB esp-link parameters
  • 0x40000 4KB unused
  • 0x41000 236KB partition2
  • 0x7E000 16KB system WiFi parameters
  • 0x7FFFF last byte of 512KB flash

Image

Essentially, the flash space is divided into two halves, the first 4KB of each half are reserved for the bootloader and the last 16KB are reserved to store settings. Everything in between is fair game to store the two copies of the firmware. The critical number to remember is that each partition leaves up to 236KB for the firmware.

Linker/loader magic

The firmware itself also consists of a number of different parts due to the way the esp8266 executes code. This is expressed as “linker segments” that are defined in linker scripts that the GCC linker uses to place all the data and code in the program into the correct memory ranges. The standard linker script provided by Espressif defines the following three segments:

dram0_0_seg : org = 0x3FFE8000, len = 0x14000
iram1_0_seg : org = 0x40100000, len =  0x8000
irom0_0_seg : org = 0x40201010, len = 0x2B000

In the esp8266 data and code are handled separately (but the are in the same address space, so it’s a von Neumann architecture, not a Harvard architecture). Data is represented by the dram0_0 segment and holds 80KB (0x14000). It contains some initialized data, the execution stack, and the heap for dynamic allocation. Instructions are stored in two segments. The first is a conventional read-only instruction ram represented by the iram1_0 segment and holds 32KB (0x8000). This RAM gets loaded by the bootloader from flash. The final segment is an instruction cache, which also holds 32KB but is loaded on-demand from flash by the hardware. The flash area that is cached is represented by the irom0_0 segment which is sized at 172KB (0x2B000).

The significance of the two instruction segments is that certain code, interrupt handlers in particular, must be loaded into the instruction RAM which is only 32KB. The bulk of the code will have to be in flash and demand-loaded into the cache when it is needed.

The segment sizes may seem to be incorrect because 80KB+32KB+172KB is more than 236KB! The reason this is OK is that not the entire dram0_0 segment needs to be loaded from flash, only the portion with statically initialized data, so while the segment is 80KB large, much less needs to actually be stored in flash.

It turns out that the esp-link firmware does not actually fit into these segments defined in Espressif’s loader scripts. The firmware uses much less than 80KB of initialized data but more than 32KB+172KB of code! In order to fit everything the linker script used by esp-link increases the irom0_0 segment to 224KB. Since this segment is demand-loaded into the cache it can be of arbitrary size as long as the entire firmware fits into the 236KB partition.

When building the firmware the Makefile prints a little informative line, which initially looked something like:

** user1.bin uses 218592 bytes of 241664 available

Here 241664 is 236KB and 218592 is the size of what’s getting flashed. This was early on in the development of esp-link and there were another 22KB to spend. Nowadays the line looks more concerning:

** user1.bin uses 237984 bytes of 241664 available

Meaning that with all the features added there are just a couple of KB left to play with!

There’s an additional twist to all these segments, which is the built-in flash filesystem. It needs to go somewhere too! The files are compressed and converted to a filesystem image in the Makefile. This image is a binary blob that is loaded at the end of irom0_0 (as if it were a bunch of instructions). Finally, a symbol is added to the linking step so the code can refer to the start of the filesystem blob using a C variable:

// address of espfs binary blob
uint32_t _binary_espfs_img_start;

The summary of all this is that the concept of the OTA update is quite simple, but the implementation requires diving into the details of how code is linked and loaded plus adding some tricks to store data like the flash filesystem into code areas.

POST handler

The actual upgrade process uses three simple HTTP requests that are geared to the needs of the firmware developer. For end-users that want to pull a ready-built firmware update from a web site and flash it a web page UI might be desirable. One big advantage of the simple version is that a lot parts have to work flawlessly for a web update page to function. This is often not the case while one is tweaking the firmware! The update process uses a simple shell script with the following steps:

  1. a GET request to esp-link establishes which of the two flash partitions can be updated (the firmware image loaded into each partition is slightly different)
  2. a POST request uploads the firmware to the appropriate partition
  3. another POST request causes esp-link to tell the bootloader to load the new partition at the next restart, and then it restarts the esp8266
  4. a final GET request issued in a loop waits for esp-link to come up again with the new firmware

This process is relatively simple and functions without the flash filesystem, without javascript, and without HTML. During the several hundred flash cycles used so far for the development of esp-link it has failed only once when a bug caused all HTTP requests to break. Fortunately, there is the serial upload to fall back on…

The next and final episode of this series will wrap-up with some loose ends.

[Back to article index]