Is robot framework able to test that something is *not* happening?

by 
# Postmortem: Handling Negative Assertions in Robot Framework for MQTT Event Verification

## Summary
A critical gap was identified in automated tests for an IoT lighting system: Robot Framework had no explicit mechanism to confirm the **absence** of an expected MQTT message within a defined time window during night-mode operations. This led to incomplete verification of system behavior.

## Root Cause
The core issue stems from Robot Framework's lack of built-in keywords for asserting **non-occurrence** of events such as "no message on MQTT topic X for Y seconds". Existing keywords (`Wait For Message`, `Message Should Not Be Published`) validate state at a single point in time but can't guarantee non-occurrence over a continuous duration.

## Why This Happens in Real Systems
- **Asynchronous Events**: IoT/messaging systems often rely on async interactions, where the absence of an action is as critical as its presence.
- **State-Dependent Logic**: Features like "night mode" explicitly prohibit certain behaviors—testing requires proving something *doesn't* happen.
- **Legacy Testing Tools**: Frameworks built for HTTP/UI often lack primitives for real-time messaging validation.

## Real-World Impact
- **False Negatives**: Tests might pass if the system fails silently instead of blocking unwanted actions.
- **Flaky Tests**: Crude workarounds (e.g., fixed `Sleep`) increase execution time and cause race conditions.
- **Masked Regressions**: Unexpected messages could go undetected, allowing bugs in business logic (e.g., lights turning on illegally during curfew).

## Example and Solution Code

### Broken Approach Using Fixed Delays
```robotframework
Test Night Mode (Flawed)
    Publish "NIGHT" to mysystem/mode
    Subscribe to mysystem/lights/lamp1
    Publish "PRESS" to mysystem/buttons/button1
    Sleep    5s  # Arbitrary delay
    Message Should Not Be Published    mysystem/lights/lamp1

Problem: Message could arrive immediately after the check.

Fixed Approach with Time-Bounded Verification

# Custom Python keyword
def verify_no_messages(seconds: int):
    end_time = time.time() + seconds
    while time.time() < end_time:
        messages = mqtt_client.get_new_messages(topic)
        if messages: 
            raise AssertionError(f"Unexpected messages: {messages}")
        time.sleep(0.1)
Test Night Mode (Fixed)
    ${topic}=    Set Variable    mysystem/lights/lamp1
    Publish "NIGHT" to mysystem/mode
    Subscribe to ${topic}
    Publish "PRESS" to mysystem/buttons/button1
    Verify No Messages in ${topic} for 5s  # Custom keyword

How Senior Engineers Fix It

  1. Build Time-Bounded Checks: Create keywords that actively monitor for events continuously throughout the verification window.
  2. Leverage Callbacks: Use MQTT client hooks to capture messages asynchronously during the wait period.
  3. Avoid Static Delays: Replace Sleep with dynamic checking loops with tight polling intervals.
  4. Boundary Testing: Explicitly verify edge cases (“last allowed delay failed”).
  5. Instrumentation: Add logging for detected messages during negative checks to aid debugging.

Why Juniors Miss It

  • Positive-Test Bias: Focus on verifying what should happen vs. what shouldn’t.
  • Over-Reliance on Built-ins: Assuming framework keywords cover all scenarios without customization needs.
  • Async Blindness: Underestimating race conditions in message-driven systems.
  • Time Handling: Using Sleep as a quick fix without understanding temporal non-determinism.
  • Debugging Difficulty: Intermittent message leaks are harder to diagnose than constant failures.