How to configure portainer and reverse nginx on my VPS?

by

Summary

A user attempted to set up container orchestration on an Ubuntu VPS using Docker, Portainer, and Nginx Proxy Manager (NPM). The goal was to expose these management interfaces via custom subdomains. The configuration resulted in 502 Bad Gateway errors for Portainer and SSL_ERROR_UNRECOGNIZED_NAME_ALERT errors for NPM. The root causes were improper upstream configuration within NPM (pointing to the VPS IP rather than the Docker internal network) and recursive proxy loops caused by exposing NPM through itself without distinct ports.

Root Cause

The failures stem from a misunderstanding of how Docker networking interacts with a reverse proxy running on the same host.

  • Portainer 502 Bad Gateway: NPM was configured to proxy traffic to Portainer, but the upstream URL was likely set to http://<vps-ip>:9000. If the VPS firewall blocked this external loopback or if Portainer was binding only to 127.0.0.1, the proxy could not establish a connection.
  • NPM SSL_ERROR_UNRECOGNIZED_NAME_ALERT: The user attempted to proxy npm.my-domain.com to itself via https://<vps-ip>. This creates a recursive proxy loop. NPM receives a request for npm.my-domain.com, resolves the IP, sends the request back to itself, and expects the “Host” header to match a valid SSL configuration it owns. The handshake fails because the proxy is essentially trying to SSL-wrap itself without handling the SNI correctly, leading to the unrecognized name alert.
  • Domain Redirections: Setting redirections at the domain provider (cPanel) to the bare IP address strips the Host header information required by NPM to route to the correct service. This forces NPM to default to the first configured host or fail SSL verification.

Why This Happens in Real Systems

  • Host vs. Container Resolution: In Docker, a container has its own IP address inside a bridge network. Using the VPS public IP as an upstream target forces traffic out to the public interface and back in, which is often blocked by firewalls (UFW) or internal routing rules. Senior engineers always use internal Docker DNS names (e.g., http://portainer:9000) for upstreams.
  • Proxying the Proxy: Running a reverse proxy and then proxying traffic to that same proxy on the standard ports creates a cycle. The request enters NPM, NPM looks up the destination, sees it is itself, and attempts to forward the traffic. The SSL handshake breaks because the proxy cannot negotiate a certificate for a request that is technically terminating inside its own process loop.

Real-World Impact

  • Management Lockout: The primary 502 error prevents access to Portainer, rendering the visual management layer for Docker containers useless.
  • Insecure Fallbacks: If the user attempts to bypass the proxy by accessing ports directly (e.g., http://<vps-ip>:81), they may expose sensitive management interfaces to the public internet without the protection of SSL termination.
  • DNS Thrashing: Improper redirections cause the browser to constantly redirect between HTTP and HTTPS or resolve to the wrong IP, leading to browser security warnings and poor user experience.

Example or Code

To fix this, the Docker Compose file must expose the correct ports, and NPM must target the Docker internal network.

1. Correct Docker Compose Setup (docker-compose.yml):

version: '3.8'

services:
  nginx-proxy-manager:
    image: jc21/nginx-proxy-manager:latest
    container_name: nginx-proxy-manager
    restart: unless-stopped
    ports:
      - "80:80"
      - "81:81"  # Admin UI
      - "443:443"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    networks:
      - app_net

  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer
    restart: unless-stopped
    command: -H unix:///var/run/docker.sock
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    ports:
      - "9000:9000" # Exposed for direct access if needed, but NPM will route internally
    networks:
      - app_net

volumes:
  portainer_data:

networks:
  app_net:
    driver: bridge

2. Correct NPM Proxy Host Configuration (Conceptual):

  • Domain Names: portainer.my-domain.com, npm.my-domain.com
  • Scheme: http
  • Forward Hostname / IP: portainer (Use the Docker service name, NOT the VPS IP)
  • Forward Port: 9000
  • SSL: Let’s Encrypt (Wildcard or specific)

How Senior Engineers Fix It

  1. Isolate Management Networks: The reverse proxy should ideally run on a dedicated network or handle traffic strictly for application containers. Management tools like Portainer and NPM are often exposed on localhost only (bound to 127.0.0.1:81) or a separate internal network, secured by a VPN or SSH tunnel, rather than exposing them directly to the web.
  2. Use Docker DNS: Configure NPM upstreams using the internal Docker service name (e.g., http://portainer:9000). This keeps traffic inside the host, bypassing the external firewall and ensuring connectivity.
  3. Decouple Provider DNS: Point the domain DNS A records directly to the VPS IP. Do not use cPanel redirections. Let NPM handle all 80/443 traffic and perform the HTTP-to-HTTPS redirection internally.

Why Juniors Miss It

  • Mental Model of “IP Address”: Juniors often view the VPS as a single entity (the IP address) rather than a host running a virtual network. They target the external IP because they see the server as a “server,” not a “router for containers.”
  • Confusing Proxying with Forwarding: They assume that because NPM listens on port 80/443, it can also accept traffic on port 81 and proxy it to itself. They fail to recognize the recursive loop involved in proxying npm.my-domain.com to the same software handling the request.
  • Over-reliance on UI Setup: Following tutorials blindly often leads to copying incorrect upstream configurations (using <vps-ip>) without understanding why localhost or the container_name is the correct choice.