Production Postmortem: Tag-Based Task Execution Failure in Ansible Molecule Scenario

Summary

Root Cause

• Molecule applies --tags/--skip-tags globally to both role tasks and its internal action playbooks (create.yml/destroy.yml).
• The role’s always-tagged task (Make preparations) runs indiscriminately due to Ansible’s built-in always tag behavior.
• Tag inheritance wasn’t isolated to the role. Action playbooks lacking explicit tagging inherited global CLI/molecule.yml tags, conflicting with scenario intentions.

Why This Happens in Real Systems

• Action Playbook Coupling: Internal Molecule playbooks execute within the same Ansible context as role tasks, inheriting tag flags.
• always Tag Ambiguity**: Ansible’s undocumented always tag behavior (runs unless explicitly skipped) contradicts scenario-specific filtering.
• Imperfect Isolation: Multi-scenario repositories often share action playbooks/vars, propagating tag conflicts across environments.
• Tool Assumptions: Molecule treats tags as global runtime args, not scoped to role execution.

Real-World Impact

• Unsafe State Changes: Non-remove tasks (e.g., configuration/installation) executed during destructive operations, risking data loss or misconfiguration.
• Scenario Reliability: Delete scenarios became non-deterministic, preventing safe infrastructure teardown.
• Debugging Overhead: Engineers spent hours verifying task execution logs due to unexpected tag behavior.

Example or Code

Attempted to create a moleculed `delete` scenario for an Ansible role that should exclusively run tasks tagged `remove`. Despite configuring scenario actions and CLI tag flags, Molecule executed unwanted tasks because tag filters applied globally to all playbooks (including Molecule's action playbooks), causing unintended task inclusion.

yaml

Role task (tags create conflicts)

• name: Make preparations
  tags: [always] # Runs in EVERY scenario unless explicitly skipped

• name: Remove
  tags: [never, remove] # Intended to run ONLY in delete scenario

yaml

Non-working molecule.yml approach

provisioner:
name: ansible
playbooks:
create: ../shared/create.yml # Lacks tags → Inherits global –tags=remove
options:
tags: remove # Applied to ALL playbooks!

How Senior Engineers Fix It

1. Explicit Skip-Tags

• Use --skip-tags always in CLI for delete scenario:
  bash
  molecule converge -s delete — –skip-tags always

• Ensures always-tagged tasks don’t force-execute during removal.

2. Isolate Action Playbooks

• Modify create.yml/destroy.yml with scenario-specific tags:
  yaml

  destroy.yml

• name: Destroy infrastructure
  tags: [“never”, “destroy”]
  hosts: all
  tasks: […]

• Call with --tags=destroy,remove during molecule destroy.

3. Decouple Role Tasks

• Split main.yml into dedicated files:
  yaml
  tasks/
  main.yml: # Default scenario
• include_tasks: setup.yml
• include_tasks: configure.yml
• include_tasks: remove.yml
  when: scenario == “delete”

remove.yml:

• name: Remove
  tags: [remove] # Remove ‘never’ tag

4. Conditional Logic

Replace tags with Molecule variables:

yaml

• name: Remove
  when: scenario == “delete” # Explicit scenario guard

Why Juniors Miss It

• Assumed Tag Locality: Expected --tags to apply only to the role, not Molecule’s internal playbooks.
• Overlooked Implicit Tags: Unaware that always runs unless skipped, even with custom tags.
• Tool Abstraction: Didn’t inspect Molecule’s underlying Ansible command generation.
• Undocumented Nuances: Ansible’s tag precedence (always > CLI tags) isn’t intuitive.
• Scenario Coupling: Underestimated shared action playbook side effects across scenarios.