Gitlab Registry Error pulling Image within Cluster

by

Summary

A deployment inside a MicroK8s cluster failed to pull images from an internal GitLab Registry (registry.learning.com) even though pushing from GitLab Runner and pulling from external machines worked. The failure occurred only inside the cluster, specifically when ArgoCD attempted to deploy workloads.

The core issue was internal DNS resolution inside the cluster failing to resolve the registry hostname, causing image pulls to fail.

Root Cause

The failure stemmed from cluster‑internal DNS not resolving the ingress‑exposed registry hostname. Common underlying triggers include:

  • Ingress hostname not added to CoreDNS search domains
  • Registry service not exposed internally (ClusterIP) but only via Ingress
  • Pods using cluster DNS cannot resolve external ingress hostnames
  • MicroK8s DNS misconfiguration or missing wildcard DNS entries
  • ArgoCD and workloads relying on internal DNS, not external resolvers

Why This Happens in Real Systems

This pattern is extremely common in self‑hosted GitLab + Kubernetes setups because:

  • Kubernetes DNS only resolves Services, not arbitrary hostnames.
  • Ingress hostnames require external DNS, which pods do not use unless explicitly configured.
  • Single‑node clusters often rely on local /etc/hosts hacks, which pods cannot see.
  • GitLab Registry is typically accessed via public hostname, but workloads inside the cluster need internal access paths.

Real-World Impact

When DNS resolution fails inside the cluster, you see:

  • ImagePullBackOff on deployments
  • ArgoCD sync failures
  • Pods stuck in Pending or ContainerCreating
  • GitLab CI/CD succeeds but runtime deployments fail
  • Confusing behavior where external machines work but cluster workloads do not

Example or Code (if necessary and relevant)

A common fix is to create an internal DNS entry using a Kubernetes ConfigMap patch for CoreDNS:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        hosts {
            10.0.x.x registry.learning.com
            fallthrough
        }
        forward . 8.8.8.8
    }

Or expose the registry internally:

apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: gitlab
spec:
  type: ClusterIP
  ports:
    - port: 5000
      targetPort: 5000
  selector:
    app: registry

How Senior Engineers Fix It

Experienced engineers solve this by ensuring multiple access paths to the registry:

  • Create an internal ClusterIP service for the registry
  • Patch CoreDNS to resolve the registry hostname internally
  • Use a registry mirror or internal alias (e.g., registry.gitlab.svc.cluster.local)
  • Configure imagePullSecrets for private registry authentication
  • Avoid relying on ingress hostnames for internal traffic
  • Validate DNS resolution from inside a pod using: 
    kubectl exec -it test-pod -- nslookup registry.learning.com

They ensure the registry is reachable both internally and externally, eliminating dependency on ingress DNS for internal workloads.

Why Juniors Miss It

Junior engineers often overlook this because:

  • They assume ingress hostnames work everywhere, including inside pods.
  • They don’t realize pods use CoreDNS, not the host machine’s DNS.
  • They test from their laptop, not from inside the cluster.
  • They confuse successful pushes from GitLab Runner with cluster‑internal pull capability.
  • They are unaware that Kubernetes does not resolve arbitrary hostnames unless explicitly configured.

The result is a classic scenario: everything works externally, CI works, but deployments fail—because internal DNS was never configured.

Leave a Comment