Humble beginnings for thingy91 FOTA

 ### Summary

Get's a FOTA MVP up and running so we can get the latest and greatest on our nRF91 Thingy:91s

Change Highlights -- more details inline in the code:

  1. I was going insane trying to debug MCUboot when no logs were
     being emitted. Tinkered around with Kconfig options to get this
     working. Trade off is I disabled support for mcumgr serial DFU
     but I do not believe anyone is using this atm anyway.
  2. Fixing 1, pointed out that due to the size of the thingy91 image
     we are hitting an edge case with MCUboot's "swap move" algorithm
     blocking a FOTA from completing succesfully. I switched to OVERWRITE
     strategy to work around this.
  3. First pass at a FOTA check. The algo works as follows:
     - Check for FOTA upon first connection since boot
     - Check for FOTA every 12 hours there-after
     - If FOTA check on 12 hour mark fails, check upon next successful
       connection (to accomodate situation where we were not
       connected)
  4. There's some kind of corruption taking place when sending data to
     cloud _and_ doing a FOTA. We should figure out what's going wrong
     but for now I fixed it by adding retry logic in our own custom FOTA
     handler (`CONFIG_MEMFAULT_FOTA_DOWNLOAD_CALLBACK_CUSTOM=y`)

 ### Internal Documentation

https://www.notion.so/memfault/WeFault-Thingy91-Fleet-Quickstart-394ab5bd97ab4a49af86f40423f51982

 ### Test Plan

1. Build the new image

```
west build -b [email protected]  memfault-asset-tracker -- -DCONFIG_MEMFAULT_NCS_PROJECT_KEY=\"${PROJECT_KEY}\" -DOVERLAY_CONFIG=overlay-memfault.conf -DCONFIG_BUILD_OUTPUT_META=n -Dmcuboot_CONFIG_BUILD_OUTPUT_META=n -DCONFIG_MEMFAULT_FOTA=y -DCONFIG_MEMFAULT_NCS_FW_VERSION_PREFIX=\"0.0.2+\"
```

2. Flash the image on the board

```
west flash --erase
```

3. Re-run 1) but with `-DCONFIG_MEMFAULT_NCS_FW_VERSION_PREFIX=\"0.0.3+\"

4. Upload `app_update.bin` from `build/zephyr` folder into wefault project:
   https://app.memfault.com/organizations/memfault/projects/thingy91-fleet/releases/0.0.3%2Bd8aaaa

5. Confirm  FOTA completes successfully by watching console:

```
[00:00:26.520,416] <inf> mflt: FOTA Update Available. Starting Download!
[00:00:26.521,820] <inf> download_client: Downloading: https://ota-cdn.memfault.com/3993/6539/11764076539?token=rFQ0Owi7Cbbf7VP9AOQimNndKD16VO_DdjLDl9ANrYg&expires=1701064800 [0]
[00:00:26.521,881] <inf> mflt: FOTA In Progress
[00:00:26.714,202] <inf> download_client: Setting up TLS credentials, sec tag count 3
[00:00:26.714,324] <inf> download_client: Connecting to https://ota-cdn.memfault.com/3993/6539/11764076539?token=rFQ0Owi7Cbbf7VP9AOQimNndKD16VO_DdjLDl9ANrYg&expires=1701064800
[00:00:28.714,447] <inf> download_client: Downloaded 1024/424428 bytes (0%)
[...]
[00:07:33.470,214] <inf> download_client: Downloaded 316416/424428 bytes (74%)
[00:07:35.357,604] <inf> app_event_manager: LOCATION_MODULE_EVT_CLOUD_LOCATION_DATA_READY
[00:07:35.358,337] <inf> app_event_manager: DATA_EVT_DATA_READY
[00:07:35.364,715] <inf> app_event_manager: DATA_EVT_CLOUD_LOCATION_DATA_SEND
[00:07:35.382,873] <inf> app_event_manager: DATA_EVT_DATA_SEND_BATCH
[00:07:35.383,758] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
[00:07:35.384,094] <inf> app_event_manager: CLOUD_EVT_CLOUD_LOCATION_UNKNOWN
[00:07:35.384,857] <inf> app_event_manager: LOCATION_MODULE_EVT_INACTIVE
[00:07:35.386,108] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
[00:07:40.273,132] <inf> download_client: Downloaded 317440/424428 bytes (74%)
[00:07:41.878,082] <err> download_client: Unexpected HTTP response: 400 bad request
[00:07:41.878,112] <err> fota_download: Download client error
[00:07:41.878,143] <inf> dfu_target_mcuboot: MCUBoot image upgrade aborted.
[00:07:41.878,173] <inf> dfu_target_mcuboot: MCUBoot image upgrade aborted.
[00:07:41.879,608] <err> mflt: FOTA failed -- trying again ...
[00:07:50.251,220] <inf> mflt: FOTA Update Available. Starting Download!
[...]
[00:10:32.878,387] <inf> download_client: Downloaded 421888/424428 bytes (99%)
[00:10:33.554,199] <inf> download_client: Downloaded 422912/424428 bytes (99%)
uart:~$ *** Booting nRF Connect SDK dcaa0fbee306 ***
I: Starting bootloader
I: Image index: 0, Swap type: test
I: Image 0 upgrade secondary slot -> primary slot
I: Erasing the primary slot
I: Image 0 copying the secondary slot to the primary slot: 0x679f0 bytes
D: writing magic; fa_id=2 off=0x68ff0 (0x74ff0)
D: erasing secondary header
D: erasing secondary trailer
I: Bootloader chainload address offset: 0xc000
I: Jumping to the first image slot
```

child_image/mcuboot.conf

@@ -7,3 +7,29 @@
 # Increase the monotonic firmware version before building a firmware update
 CONFIG_FW_INFO=y
 CONFIG_FW_INFO_FIRMWARE_VERSION=1
+
+# Note: BOOT_SERIAL and UART_CONSOLE cannot be installed on the same uart
+# It looks like there is an unused UART_1 on the thingy where we could
+# potentially wire up CONFIG_MCUBOOT_SERIAL=y CONFIG_BOOT_SERIAL_UART=y
+# in the future
+
+CONFIG_MCUBOOT_SERIAL=n
+CONFIG_CONSOLE=y
+CONFIG_UART_CONSOLE=y
+CONFIG_MCUBOOT_LOG_LEVEL_DBG=y
+
+# Note: The image is nearly full causing some kind of edge case to be hit when swap upgrade
+# move strategy is used:
+#
+#   I: Boot source: none
+#   I: Image index: 0, Swap type: test
+#   I: Starting swap using move algorithm.
+#   W: Not enough free space to run swap upgrade
+#   W: required 430080 bytes but only 425984 are available
+#
+# For more info about mcuboot upgrade strategies see:
+#   https://interrupt.memfault.com/blog/mcuboot-overview#swap-mode
+#
+# For now let's flip on MCUBOOT_OVERWRITE_ONLY mode which has less wonky
+# flash requirements (and imo is better for long term flash health)
+CONFIG_BOOT_UPGRADE_ONLY=y

prj.conf

@@ -126,3 +126,7 @@ CONFIG_LOG_DEFAULT_LEVEL=3
 # instead of logs being separately viewed in a file via
 # RTT logger
 CONFIG_SHELL_LOG_BACKEND=y
+
+# Let's manage the memfault_fota_download_callback() from
+# the asset tracker application rather than using the Memfault default
+CONFIG_MEMFAULT_FOTA_DOWNLOAD_CALLBACK_CUSTOM=y

src/main.c

@@ -25,6 +25,9 @@
 #include "events/util_module_event.h"
 #include "events/modem_module_event.h"
 
+#include "memfault/components.h"
+#include "memfault/nrfconnect_port/fota.h"
+
 #include <zephyr/logging/log.h>
 #include <zephyr/logging/log_ctrl.h>
 
@@ -490,6 +493,74 @@ static void on_sub_state_active(struct app_msg_data *msg)
 	}
 }
 
+/* always check for a new FOTA on first boot and then every timer period */
+static bool s_run_fota_check_upon_lte_connect = true;
+
+static void prv_memfault_fota_work_handler(struct k_work *work) {
+	int rv = memfault_fota_start();
+	if (rv < 0) {
+		/* we may have just not been connected, try again on next connect */
+		s_run_fota_check_upon_lte_connect = true;
+	} else if (rv == 0) {
+		MEMFAULT_LOG_INFO("FOTA up to date");
+	}
+}
+K_WORK_DELAYABLE_DEFINE(s_memfault_fota_work, prv_memfault_fota_work_handler);
+
+static void prv_run_memfault_fota_check(void) {
+	k_work_schedule(&s_memfault_fota_work, K_SECONDS(5));
+}
+
+void memfault_fota_download_callback(const struct fota_download_evt *evt) {
+	switch (evt->id) {
+	case FOTA_DOWNLOAD_EVT_FINISHED:
+		MEMFAULT_LOG_INFO("OTA Complete, resetting to install update!");
+		memfault_platform_reboot();
+		break;
+	case FOTA_DOWNLOAD_EVT_ERROR:
+	  /*
+	   * FIXME: When multiple threads are send/read'ing from the modem
+	   * at the same time, we get intermittent corruption issues. We should
+	   * really figure out what is wrong here but for now we just keep retrying.
+	   *
+	   * Example error logs:
+	   *
+	   * [00:04:45.544,891] <wrn> location: GNSS timed out possibly due to too short GNSS time windows
+	   * [00:04:45.557,037] <inf> app_event_manager: LOCATION_MODULE_EVT_CLOUD_LOCATION_DATA_READY
+	   * [00:04:45.557,769] <inf> app_event_manager: DATA_EVT_DATA_READY
+	   * [00:04:45.562,713] <inf> app_event_manager: DATA_EVT_CLOUD_LOCATION_DATA_SEND
+	   * [00:04:45.580,718] <inf> app_event_manager: DATA_EVT_DATA_SEND_BATCH
+	   * [00:04:45.581,604] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
+	   * [00:04:45.581,909] <inf> app_event_manager: CLOUD_EVT_CLOUD_LOCATION_UNKNOWN
+	   * [00:04:45.582,672] <inf> app_event_manager: LOCATION_MODULE_EVT_INACTIVE
+	   * [00:04:45.583,923] <inf> app_event_manager: CLOUD_EVT_DATA_SEND_QOS
+	   * [00:04:46.846,618] <inf> download_client: Downloaded 346112/424124 bytes (81%)
+	   * [00:04:47.373,565] <err> download_client: Unexpected HTTP response: 403 forbidden
+	   * [00:04:47.373,596] <err> fota_download: Download client error
+	   * [00:04:47.373,596] <err> fota_download: Download client error
+	   * [00:04:47.373,596] <inf> dfu_target_mcuboot: MCUBoot image upgrade aborted.
+	   * [00:04:47.373,657] <inf> dfu_target_mcuboot: MCUBoot image upgrade aborted.
+	   * [00:04:47.375,183] <err> FOTA failed -- trying again ...
+	   */
+		MEMFAULT_LOG_ERROR("FOTA failed -- trying again ...");
+		prv_run_memfault_fota_check();
+		break;
+	default:
+		break;
+  }
+}
+
+static void prv_memfault_fota_timer_expiry_handler(struct k_timer *dummy) {
+	prv_run_memfault_fota_check();
+}
+
+K_TIMER_DEFINE(s_memfault_fota_timer, prv_memfault_fota_timer_expiry_handler, NULL);
+
+static void prv_memfault_fota_timer_start(void) {
+	/* Check and see if new FOTA is available every 12 hours */
+	k_timer_start(&s_memfault_fota_timer, K_HOURS(12), K_HOURS(12));
+}
+
 /* Message handler for all states. */
 static void on_all_events(struct app_msg_data *msg)
 {
@@ -517,6 +588,13 @@ static void on_all_events(struct app_msg_data *msg)
 	if (IS_EVENT(msg, sensor, SENSOR_EVT_MOVEMENT_IMPACT_DETECTED)) {
 		SEND_EVENT(app, APP_EVT_DATA_GET_ALL);
 	}
+
+	if (IS_EVENT(msg, modem, MODEM_EVT_LTE_CONNECTED)) {
+		if (s_run_fota_check_upon_lte_connect) {
+			prv_run_memfault_fota_check();
+			s_run_fota_check_upon_lte_connect = false;
+		}
+	}
 }
 
 int main(void)
@@ -547,6 +625,7 @@ int main(void)
 		LOG_ERR("Failed starting module, error: %d", err);
 		SEND_ERROR(app, APP_EVT_ERROR, err);
 	}
+	prv_memfault_fota_timer_start();
 
 	while (true) {
 		module_get_next_msg(&self, &msg);