User Safety: safe

by

Summary

Detecting success or failure icons in Power Automate Desktop (PAD) should rely on UI element selectors rather than image matching. By capturing the icons as distinct UI elements (desktop or web, depending on the application) and using actions such as Wait for UI element, If UI element exists, and Get attribute, a flow becomes far more stable across resolution changes and UI updates.

Root Cause

  • The icons are rendered as individual UI controls with their own properties (e.g., AutomationId, ClassName, Name, src attribute for web images).
  • Relying on image detection treats the icon as a bitmap, causing false negatives when DPI, theme, or screen scaling changes.
  • The underlying UI framework (Win32, UIA, Selenium) already provides reliable selectors that PAD can query.

Why This Happens in Real Systems

  • Dynamic resolution: Users may switch monitors or change scaling, breaking pixel‑perfect matches.
  • Theme variations: Dark/light mode alters colors, making image hashes obsolete.
  • UI redesign: Minor HTML/CSS updates change the visual representation but keep the element’s logical identifier intact.
  • Performance: Image detection scans the entire screen each time, adding latency compared to a selector lookup.

Real-World Impact

  • Flaky automations: Frequent false failures cause loss of trust and increased maintenance overhead.
  • Increased runtime: Image searches are CPU‑intensive, inflating execution time.
  • Higher support cost: Support tickets spike when end‑users change display settings.
  • Scalability limits: Large fleets of bots cannot afford the instability introduced by image matching.

Example or Code (if necessary and relevant)

# No code needed for this approach; actions are configured via PAD UI.

How Senior Engineers Fix It

  • Capture the icon as a UI element

    1. Open UI element picker → hover over the green check or red X.
    2. Save each as a separate element (e.g., SuccessIcon, FailureIcon).

  • Use a reliable selector

    • For desktop apps: keep AutomationId or Name in the selector.
    • For web pages: keep the src attribute or a unique CSS class.

  • Build the decision logic

    1. Wait for UI element (SuccessIcon) or FailureIcon with a timeout.
    2. If UI element exists (SuccessIcon) → treat as success.
    3. Else If UI element exists (FailureIcon) → treat as failure.
    4. Else → log timeout / unknown state.

  • Add fallback verification (optional): retrieve the element’s src or alt attribute and compare to expected values.

  • Log and handle: write outcome to a variable, send a notification, or branch the flow accordingly.

Why Juniors Miss It

  • Overreliance on tutorials that showcase image detection as the “quick” solution.
  • Lack of knowledge about PAD’s UI element picker and selector tuning.
  • Skipping selector validation: they often accept the default selector without pruning unstable attributes (e.g., Idx, Depth).
  • Ignoring environment variance: they test on a single monitor/resolution and assume the flow will work everywhere.

By focusing on stable UI element selectors and the built‑in If UI element exists actions, PAD automations become robust, maintainable, and performant.

Leave a Comment