How to inject a complex YAML config (arrays & nested objects) into a Go application running on Kubernetes (using Helm)?

by

Summary

The problem at hand is injecting a complex YAML configuration with arrays and nested objects into a Go application running on Kubernetes, deployed using Helm. The application uses koanf to load configuration from YAML files and environment variables. The goal is to find a clean and idiomatic Kubernetes solution to inject this configuration without flattening it manually into environment variables.

Root Cause

The root cause of the problem is that environment variables are flat strings, which makes it difficult to inject a complex YAML configuration with arrays and nested objects. Additionally, Helm templating for arrays into env vars becomes very verbose and error-prone.

Why This Happens in Real Systems

This problem occurs in real systems because:

  • Environment variables are not designed to handle complex configurations with arrays and nested objects.
  • Helm templating can become verbose and error-prone when dealing with complex configurations.
  • Kubernetes and Helm provide alternative solutions, such as ConfigMaps, to handle complex configurations.

Real-World Impact

The real-world impact of this problem is:

  • Increased complexity in managing and injecting complex configurations into applications.
  • Error-prone configurations that can lead to application failures or unexpected behavior.
  • Difficulty in scaling applications that require complex configurations.

Example or Code

package main

import (
    "fmt"
    "io/ioutil"
    "path/filepath"

    "github.com/kubernetes/client-go/util/homedir"
    "k8s.io/client-go/rest"
)

func main() {
    // Create a ConfigMap
    configMap := map[string]string{
        "cron": `
jobs:
  - job_kind: daily
    topic: Topic1
    interval: 1
    day_of_month_to_start: 1
    day_of_month_to_stop: 31
    at_times:
      - hour: 05
        minute: 00
        second: 00
      - hour: 11
        minute: 00
        second: 00
  - job_kind: monthly
    topic: Topic2
    interval: 1
    days_of_the_month:
      - -1
    at_times:
      - hour: 21
        minute: 30
        second: 0
`,
    }

    // Create a Kubernetes client
    kubeConfig := filepath.Join(homedir.HomeDir(), ".kube", "config")
    config, err := rest.InClusterConfig()
    if err != nil {
        config, err = rest.LoadKubeConfig(kubeConfig)
        if err != nil {
            fmt.Println(err)
            return
        }
    }

    // Create a ConfigMap object
    configMapObj := &corev1.ConfigMap{
        ObjectMeta: metav1.ObjectMeta{
            Name: "my-config",
        },
        Data: configMap,
    }

    // Apply the ConfigMap to the cluster
    configMapClient := kubeclientset.CoreV1().ConfigMaps("default")
    _, err = configMapClient.Create(context.TODO(), configMapObj, metav1.CreateOptions{})
    if err != nil {
        fmt.Println(err)
        return
    }

    // Load the ConfigMap into the application
    configMap, err = configMapClient.Get(context.TODO(), "my-config", metav1.GetOptions{})
    if err != nil {
        fmt.Println(err)
        return
    }

    // Unmarshal the ConfigMap into a Go struct
    var cronConfig CronConfig
    err = yaml.Unmarshal([]byte(configMap.Data["cron"]), &cronConfig)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(cronConfig)
}

type CronConfig struct {
    Jobs []Job `yaml:"jobs"`
}

type Job struct {
    JobKind          string    `yaml:"job_kind"`
    Topic            string    `yaml:"topic"`
    Interval         int       `yaml:"interval"`
    DayOfMonthToStart int       `yaml:"day_of_month_to_start"`
    DayOfMonthToStop  int       `yaml:"day_of_month_to_stop"`
    AtTimes          []AtTime  `yaml:"at_times"`
    DaysOfTheMonth   []int     `yaml:"days_of_the_month"`
}

type AtTime struct {
    Hour   int `yaml:"hour"`
    Minute int `yaml:"minute"`
    Second int `yaml:"second"`
}

How Senior Engineers Fix It

Senior engineers fix this problem by:

  • Using ConfigMaps to store complex configurations with arrays and nested objects.
  • Creating a Kubernetes client to apply the ConfigMap to the cluster.
  • Loading the ConfigMap into the application using the Kubernetes client.
  • Unmarshaling the ConfigMap into a Go struct using a YAML unmarshaller.

Why Juniors Miss It

Juniors may miss this solution because:

  • They may not be familiar with ConfigMaps and their use cases.
  • They may not know how to create a Kubernetes client and apply a ConfigMap to the cluster.
  • They may not understand how to load and unmarshal a ConfigMap into a Go struct.
  • They may not be aware of the best practices for handling complex configurations in Kubernetes and Helm.

Leave a Comment