As I’ve mentioned in my post about Pulumi, I don’t like helm template approach. In my opinion, it’s better to stick with the tool rather that mimic it’s behaviour. In case of helm “sticking with the tool” also means out of the box support for the standard helm tool, including plugins.

My tool of choice is Helmsman

I appreciate the fact that building abstraction on top of the tool such as helm is not ideal. That’s why there are no major players in this field.

And there are even less of these which were not abandoned by their authors, and support changes introduced in helm 3.

Alternatives

  • Pulumi - didn’t like it for use of helm template.
  • Ship - concentrated on a single application lifecycle.
  • helmfile - quite popular, but I didn’t like the fact that the configuration file (helmfile.yaml) was treated as a Go template file.
  • Landscaper - deprecated in favour of helmfile just couple days ago.
  • Reckoner - Python-based, quite slow in my experience, very brief documentation, lack of secrets or environment variables management.

Installation (on macOS)

brew install helm helmsman sops
helm plugin install https://github.com/databus23/helm-diff --version master
helm plugin install https://github.com/fabmation-gmbh/helm-whatup
helm plugin install https://github.com/futuresimple/helm-secrets

Usage

  • Create Desired State Specification file (helmsman.yaml in this example)
  • helmsman --apply -p 3 -f helmsman.yaml, where -p is parallelism level. I set it to 1 on my Raspberry PI k3s cluster, otherwise it can’t cope with the load; my intel-based cluster works better, and -p 3 speeds things up.
  • …
  • Profit!

Secrets Management

Use helm-secrets and sops (installed in my example) for secret management.

With GPG key it’s as easy as:

export SOPS_PGP_FP="<KEY_ID>"
sops /path/to/new/or/existing/secrets-file.yaml
  • Each value is encoded separately, which makes it perfect for GitOps and change management.
  • Cloud KMS services are supported.

Extracting common variables into dotenv

.env files are supported, and they are ideal for reusable values like URLs, hostnames, IP addresses, versions.

ohmyzsh has even a dotenv plugin to load them up automatically into your shell.

Simple .env looks like:

IP_ETCD=192.168.0.100
# Some comment
VERSION_KIBANA=7.6.2

Desired State Definition (DSD) tips

Both environment variables can secrets can then be referenced in the Desired State Specification file.

You can (and should) also extract value files for individual charts into separate files.

I use the following folder structure:

.
├── charts
│   ├── kibana
│   └── prometheus-operator
├── helmsman.yaml
├── secrets
│   ├── kibana.yaml
│   └── prometheus-operator.yaml
└── values
    ├── kibana-default.yaml
    ├── kibana.yaml
    ├── prometheus-operator-default.yaml
    └── prometheus-operator.yaml
  • charts/ hold unpacked chart archives (.gitignore’d) - I use it for updating to latest versions, more on that later.
  • secrets/ hold files with sensitive information, encrypted by sops
  • values/ hold value files used to configure charts, as well as default value files extracted from charts/. Default value files are stored in git, that’s how I find out what changed between currently installed and latest chart release. Then I upgrade values used by installed apps accordingly.

Helmsman DSD file looks like this:

settings:
  # I use that to control chart removal order - use it rarely :)
  reverseDelete: true

# Namespaces observed by helmsman. It will remove any charts from these namespaces that are not managed by helmsman.
namespaces:
  kibana: {}
  prometheus-operator: {}

# All your helm repos go here
helmRepos:
  stable: 'https://kubernetes-charts.storage.googleapis.com'

apps:
  # That's pretty self-descriptive IMHO :)
  kibana:
    namespace: kibana
    chart: stable/kibana
    version: 3.2.6 # Chart version
    enabled: true # Set to false, and release will be removed
    secretsFile: secrets/kibana.yaml # Path to yaml file with secrets managed by sops. It will be decrypted, and later used as a regular values file (helm -f)
    valuesFile: values/kibana.yaml # Yaml file with unencrypted configuration values  (helm -f)
    set:
      image.tag: '${VERSION_KIBANA}' # Here you can use variables defined in .env (helm --set key=value)
      'ingress.hosts[0]': '${DNS_NAME_KIBANA}' # And even address array indices

  prometheus-operator:
    namespace: prometheus-operator
    chart: stable/prometheus-operator
    version: 8.13.0
    enabled: true
    valuesFiles: # You can use multiple values files if necessary
      - values/prometheus-operator.yaml
      - values/prometheus-operator-rules.yaml
    secretsFile: secrets/prometheus-operator.yaml
    priority: -50 # And also set priority. The lower, the sooner it will be executed when installing. And the later during deletion (due to reverseDelete)
    wait: true # You can also wait...
    timeout: 600 #... for a specific timeout until chart is fully deployed
    noHooks: true # You can disable Helm hooks
    helmFlags: ['--skip-crds'] # Or pass additional flags to helm command line, if necessary

Upgrading charts

I use helm-whatup plugin to find updates for the charts:

helm whatup --all-namespaces -a

And I wrote a script that supplements the helmsman upgrade process:

  • Run helm whatup
  • For each outdated chart run helm fetch <chart_name> --version=<chart_new_version> --untar --destination charts/
  • Then I copy charts/<chart_name>/values.yaml to values/<chart_name>-default.yaml

I am too ashamed of the script to share it though :smile:.

Once updated charts are fetched, I update chart versions in helmsman.yaml, make necessary updates to my overridden values, and run helmsman apply... again.