Fixing witadminAssembly Version Mismatch in Azure DevOps Server

by

Summary

A production engineer encountered a critical failure when attempting to export work item templates from an on-premise Azure DevOps 2022 server using the witadmin CLI tool. Despite having the required assembly (Microsoft.Identity.Client) present in the local NuGet cache, the tool threw a version mismatch error (HRESULT: 0x80131040). The core issue is a conflict between the assembly version requested by the application’s manifest and the version currently loaded in the runtime environment, often caused by DLL Hell or global assembly redirection conflicts.

Root Cause

The error is a classic case of a Binding Redirect Failure. The specific mechanics are:

  • Manifest Mismatch: The witadmin.exe binary contains a manifest that explicitly requests Microsoft.Identity.Client version 4.70.0.0.
  • Runtime Environment Pollution: Because the user has multiple versions of Visual Studio (2022 and 2026) installed, the system’s global assembly cache or the application’s local search path is providing a different version of the same DLL.
  • NuGet Misconception: The presence of the DLL in %AppData%/.nuget/packages is irrelevant to the CLI tool. The tool does not automatically scan the user’s NuGet cache; it looks in the Application Directory or the Global Assembly Cache (GAC).
  • Dependency Shadowing: A higher version of the identity library installed by a different developer tool is being prioritized by the OS, but it lacks the specific metadata expected by the witadmin manifest.

Why This Happens in Real Systems

In complex enterprise environments, this occurs due to:

  • Side-by-Side Installations: Installing multiple versions of heavy IDEs (like Visual Studio) often updates shared shared-component paths or environment variables.
  • Implicit Dependencies: Many CLI tools rely on the .NET Framework/Core runtime which may attempt to resolve dependencies from shared system folders before local folders.
  • Transitive Dependency Conflicts: A tool might depend on Library A, which depends on Microsoft.Identity.Client v4.70, while another installed tool forced a system-wide update to v5.x.

Real-World Impact

  • Automation Failure: CI/CD pipelines or maintenance scripts using witadmin will fail unexpectedly during scheduled tasks.
  • Operational Bottlenecks: Engineers are unable to perform routine administrative tasks like template exports or bulk work item updates.
  • Debugging Overhead: High-seniority engineers are pulled away from feature work to troubleshoot “ghost” dependency issues that appear non-deterministic.

Example or Code (if necessary and relevant)

To diagnose the exact version being loaded versus the version requested, an engineer would use a tool like fuslogvw.exe (Assembly Binding Log Viewer).

# Example: Checking which versions of the assembly exist in the local tool directory
Get-ChildItem -Path "C:\Program Files\Azure DevOps Server\Tools\" -Filter "Microsoft.Identity.Client.dll" -Recurse

How Senior Engineers Fix It

A senior engineer ignores the NuGet cache and focuses on Isolation and Explicit Redirection:

  1. Local Assembly Injection: Copy the specific version of Microsoft.Identity.Client.dll (v4.70.0.0) directly into the same folder as witadmin.exe. This forces the runtime to prioritize the local file over the GAC or system paths.
  2. App.config Modification: Locate the witadmin.exe.config file and manually add a <bindingRedirect> to map any requested version to the version actually present on the disk.
  3. Environment Isolation: Use a Virtual Environment or a dedicated “Tools” directory that is stripped of conflicting PATH entries.
  4. Dependency Walking: Use dumpbin /dependents or Dependencies.exe to identify exactly which parent assembly is requesting the incorrect version.

Why Juniors Miss It

  • The “NuGet Trap”: Juniors often assume that if a package exists in their nuget folder, it is available to the entire OS. They fail to understand the Application Base Directory priority.
  • Treating Symptoms, Not Causes: A junior might try to install every version of the library they see online, effectively making the “DLL Hell” worse by adding more versions to the system.
  • Ignoring Config Files: They often overlook .config files, assuming the error is a “code bug” rather than a deployment/environment configuration issue.

Leave a Comment