Preventing Partial Serial Reads on micro:bit with Robust Framing

by

Summary

The system experienced a partial data ingestion failure during serial communication between a host PC and a micro:bit microcontroller. While the host successfully transmitted a long string, the micro:bit only processed and displayed the first segment of the message. This resulted in an incomplete state on the hardware device, even though the transmission logic appeared correct at the application layer.

Root Cause

The failure is caused by a buffer overflow/underrun mismatch and a misunderstanding of how asynchronous interrupt-driven serial handlers process incoming bytes. Specifically:

  • Buffer Size Limits: The micro:bit has a limited hardware and software buffer for serial RX (receive).
  • Race Conditions: The on_data_received function is triggered by a delimiter (Newline). However, the logic uses serial.read_string(), which attempts to grab what is currently in the buffer.
  • Non-Atomic Reads: In the original implementation, the Python script sends a large burst of data. The micro:bit’s interrupt fires as soon as the first chunk or newline is detected, but the read_string() call may execute before the entire payload has physically arrived over the wire.
  • Missing Termination Synchronization: While the user added a \n, the micro:bit logic was still attempting to read the string into a single variable without verifying if the complete byte stream had arrived.

Why This Happens in Real Systems

In high-performance or embedded production environments, this is known as a Streaming Data Fragmentation issue.

  • MTU Mismatches: Different layers of a network or serial stack have different Maximum Transmission Units. If a packet is larger than the receiver’s buffer, it is fragmented.
  • Interrupt Latency: In embedded systems, the CPU might be busy handling other tasks (like updating an LED matrix) when bits arrive. If the buffer fills up before the CPU can “drain” it, data is dropped.
  • Asynchronous Arrival: Data does not arrive as a single block; it arrives as a continuous stream of individual bytes. Developers often mistakenly treat serial ports like file systems (where you read a whole file) rather than streams (where you must collect pieces until a delimiter is found).

Real-World Impact

  • Data Corruption: Partial commands can lead to dangerous physical actions in industrial automation (e.g., receiving “MOVE_ARM_10” but only processing “MOVEARM“).
  • System Instability: Incomplete JSON or XML packets will cause parser exceptions, potentially crashing the service.
  • Security Vulnerabilities: Buffer overflow attacks exploit exactly this behavior, where an attacker sends more data than expected to overwrite adjacent memory.

Example or Code (if necessary and relevant)

import serial

# CORRECT PATTERN: Ensure the sender uses clear delimiters
ser = serial.Serial('COM5', 115200, timeout=1)
msg = "This is a full message that requires a delimiter\n"
ser.write(msg.encode('utf-8'))

# CORRECT PATTERN: The receiver must loop until the delimiter is found
# In the micro:bit context, we rely on the delimiter-based interrupt

How Senior Engineers Fix It

Senior engineers implement Robust Framing Protocols rather than relying on raw string reads.

  • Delimiter-Based Parsing: Instead of reading “a string,” we read “until the next \n or \r character.”
  • Length-Prefixing: A more professional approach is to send the length of the message first (e.g., [LEN:45]MESSAGE_HERE). The receiver reads the length, allocates a buffer, and waits until exactly that many bytes arrive.
  • Checksums/CRC: To ensure the partial message isn’t just “corrupted” but actually “complete,” we append a Cyclic Redundancy Check (CRC). If the calculated CRC doesn’t match the sent CRC, we discard the partial packet and request a retransmission.
  • State Machine Implementation: Instead of a single function call, use a Finite State Machine (FSM) that transitions from IDLE -> RECEIVING -> VALIDATING -> PROCESSING.

Why Juniors Miss It

  • Abstraction Fallacy: Juniors often assume that write() and read() are atomic operations. They believe that if they write 50 bytes, the receiver will see 50 bytes in one single “event.”
  • Focus on Happy Path: They test with small strings (like “Hello”) which fit entirely within the initial hardware buffer, and only encounter the bug when the “real” data is long enough to trigger fragmentation.
  • Lack of Hardware Awareness: They treat serial communication as a high-level software concept rather than a physical layer timing issue involving voltages, clock speeds, and buffer registers.

Leave a Comment