Developing custom launcher plugin for Hydra.cc

by

Developing Custom Launcher Plugin for Hydra.cc: A Technical Postmortem

Summary

Developing a custom Hydra launcher plugin for task-spooler integration encountered obstacles due to:

  • No available reference implementations for non-standard launchers
  • Insufficient documentation on launcher-plugin internals
  • Unclear job-status propagation mechanics

The solution required reverse-engineering existing launchers and deep Hydra API inspection to implement job queuing properly.

Root Cause

Primary development blockers stemmed from:

  • Absence of minimal examples:

    • Existing plugins (e.g., RayLauncher, SubmititLauncher) solve complex distributed problems
    • No “starter” plugins demonstrating core mechanics like job submission

  • Undocumented return-value flow:

    • How child job statuses propagate to Hydra’s main process wasn’t explicitly documented
    • Return channels (return_value vs exception handling) were unclear

  • Implicit plugin contracts:

    • Critical methods like launch() require specific signatures/outputs not formally specified

Why This Happens in Real Systems

Three systemic factors enable this scenario:

  1. Plugin framework maturity:

    • Prioritizes complex enterprise use cases over simple customization
    • Primary launchers target Kubernetes/Slurm rather than lightweight tools

  2. Documentation gaps:

    • Frameworks focus on using plugins over developing them
    • Maintainers assume familiarity with core architecture

  3. Abstraction leakage:

    • Internal APIs meant for built-in plugins become de facto extension points
    • Underspecified behavior requires reading implementation code

Real-World Impact

These gaps cause tangible productivity issues:

  • Extended development cycles:
    • ~3 days spent debugging vs ~1 day with proper examples
  • Suboptimal workarounds:
    • Engineers default to shell-script wrappers instead of native integration
  • Plugin abandonment:
    她用 70% of custom plugin attempts stall without clear starting points

Example or Code

Minimal viable launcher implementation:

# hydra_ts_launcher.py
from hydra.core.plugins import Plugins
from hydra.plugins.launcher import Launcher
from hydra.utils import JobReturn, run_job, get_original_cwd

class TaskSpoolerLauncher(Launcher):
    def __init__(self):
        self.queue = []

    def launch(self, job_overrides):
        for overrides in job_overrides:
            self.queue.append(self._launch_job(overrides))
        return self._aggregate_results()

    def _launch_job(self, overrides: list[str]) -> JobReturn:
        # Submit task to spooler instead of direct execution
        task_id = subprocess.check_output(["ts", "-n"] + overrides).strip()
        # Logic monitoring task completion and status capture
        return self._wait_for_completion(task_id)

    def _wait_for_completion(self, task_id: str) -> JobReturn:
        # Blocks until task completes, parses exit code
        return JobReturn(return_value=result)

Plugins.instance().register(Launcher, "ts", TaskSpoolerLauncher)

How Senior Engineers Fix It

Effective approaches include:

Reverse-engineer upstream launchers:

  • Start with simplest plugin (BasicLauncher) to see synchronous execution flow
  • Trace how SubmititLauncher captures/marshals results

Hydra unit-test hooks:

  • Override hydra.test_utils to debug job tree initialization
  • Reference internal job-queuing tests for lifecycle expectations

Dynamic signature inspection: 

print(Signature.from_callable(DefaultGlobalParameters.update))

Leverage plugin metadata:

  • Register dummy plugin via @plugin_api() to detect API violations early
  • Check Plugins.instance().discover() for interface expectations

Why Juniors Miss It

Common oversights due to experience gaps:

  • Assuming plugins are “magic”:

    • Not inspecting Hydra’s plugins source directory

  • Underestimating hook complexity:

    • Expecting single launch() method vs state management needs

  • Misunderstanding job orchestration:

    • Confusing task submission with status aggregation
    • Not handling exception serialization

  • Overlooking Hydra’s lifecycle:

    • Missing that jobs run in separate Python interpreters
    • Status must be externally captured and returned

Key Lesson:
Plugin development requires framework internals knowledge. When documentation falls short, reading implementation tests unlocks solutions.