Merge n8n and Docling in Docker Compose without OOM on M1

by

Summary

The user attempted to transition from a standardized starter kit to a customized production environment by manually injecting new services (Docling) into an existing Docker Compose orchestration. The core challenge is moving from a “one-click” deployment model to a modular architecture without creating dependency hell or service outages.

Root Cause

The underlying issue is a misunderstanding of container orchestration principles. The user views the “self-host folder” as a monolithic entity rather than a collection of decoupled services. The technical hurdles include:

  • Orchestration Coupling: Attempting to merge two different docker-compose.yaml files without reconciling network definitions and volume mappings.
  • Environment Divergence: The two GitHub repositories use different versions of base images, environment variables, and networking drivers.
  • Resource Contention: On an M1 Mac, adding heavy ingestion tools like Docling without adjusting resource constraints can lead to kernel panics or OOM (Out of Memory) kills.

Why This Happens in Real Systems

In professional DevOps environments, this is known as Configuration Drift. It happens when:

  • Shadow IT: Teams adopt “all-in-one” scripts that are easy to start but impossible to extend.
  • Vendor/Tutorial Lock-in: Developers rely on pre-configured stacks (like the n8n starter kit) which are optimized for ease of use rather than extensibility.
  • Lack of Modular Design: Systems are built as “Big Balls of Mud” where adding a single microservice requires rewriting the entire infrastructure definition.

Real-World Impact

  • Service Downtime: Incorrectly merging YAML files often leads to container_name collisions or port conflicts.
  • Data Corruption: If two different compose files attempt to mount the same host directory with different permissions, local databases (PostgreSQL/Qdrant) can suffer filesystem corruption.
  • Deployment Fragility: A “working” setup that relies on manual folder merging cannot be version-controlled effectively via Git, making disaster recovery impossible.

Example or Code

To add Docling to an existing n8n stack, one should not copy-paste files, but rather extend the existing compose file or create a unified orchestration layer.

services:
  n8n:
    image: n8nio/n8n:latest
    networks:
      - ai-network
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres

  docling:
    image: docling/docling:latest
    networks:
      - ai-network
    deploy:
      resources:
        limits:
          memory: 4G

networks:
  ai-network:
    driver: bridge

How Senior Engineers Fix It

A senior engineer treats infrastructure as Code (IaC) and follows these steps:

  • Decoupling: They treat every tool (n8n, Qdrant, Docling) as an independent microservice.
  • Unified Networking: They define a single External Docker Network so that containers can communicate via service names (e.g., http://docling:8080) regardless of which compose file they belong to.
  • Environment Abstraction: They use .env files to manage credentials and versions, ensuring that merging two projects doesn’t overwrite critical database passwords.
  • Modular Compose: They use the extends keyword or multiple -f flags in Docker Compose to layer configurations:
    docker-compose -f docker-compose.yml -f docker-compose.docling.yml up -d

Why Juniors Miss It

  • The “Copy-Paste” Trap: Juniors tend to look for a single “correct” file to copy, rather than understanding the relationships between services.
  • Ignoring the Network Layer: They focus on the image and the port, but forget that for services to talk to each other in Docker, they must inhabit the same virtual network.
  • Monolithic Thinking: They see a “folder” as a single unit of deployment, whereas seniors see a folder as a repository of orchestrated components.

Leave a Comment