Kubernetes CD with Flux

Deploying resources to a cluster is one of the most basic tasks in using Kubernetes. There are a lot of tools out there designed to help with this task, each with their own advantages and drawbacks.

Flux CD, developed by Weaveworks, is a tool that tries to bring a Gitops workflow to Kubernetes operations in a relatively simple way. In my opinion it does this rather well, though it can be somewhat challenging to get the service up and running in your cluster. In this article I’ll try to explain Flux’s functionality and give a short tutorial in installing it in a Kubernetes cluster.

How it works

is fairly simple. It makes it so that you don’t need to give production credentials to your CI tool, and puts the border between CI and CD at the Docker repository.

First, Flux lives inside a Kubernetes cluster and has permissions to manage resources in that cluster. There are no commands coming in from outside.

You provide Flux with the URI of a Git repo which contains Kubernetes manifests in YAML files. Flux needs to be able to read and write to this repo, and will apply whatever manifests are found in it to the cluster Flux is running in.

Whenever you change or add a manifest and commit the change to Git, Flux will notice the change and apply it to the cluster.

In addition to this, you can annotate your Kubernetes deployments to indicate to Flux that it should keep the deployment’s Docker image up to date with the Docker repository from which the image hails. You can even specify image tag regex so that Flux will only install new images which match the given pattern, eg: only update the deployment with new images whose tags match “develop-.*”.

1. Flux is running in a Kubernetes cluster, watching a Git repo that has two YAML files defining resources for app-1 and app-2.
2. The manifest for app-1 contains an annotation indicating that Flux should install any new images that are pushed to app-1’s Docker repository.
3. Flux runs kubectl apply -f sync/ on the Git repository containing the manifests. The resources are now running in the cluster as specified in the Git repo.
4. The CI pipeline for app-1 pushes a new image to its Docker repository.
5. Flux finds out that there’s a new image for app-1.
6. Flux runs kubectl patch on app-1’s deployment, changing the container image to this new one.
7. Finally, Flux modifies the image tag to the new one in the Git repository, and commits and pushes the change.

This hopefully illustrates how Flux can be helpful for managing your pile of Kubernetes manifests and rolling out new versions of your app. Now let’s look at how to set it up in Kubernetes.

Installing Flux

First let’s give Flux its own namespace, as well as cluster wide permissions. We’re going to give it free reign in the cluster, but you could just as easily limit it to a single namespace.

---
apiVersion: v1
kind: Namespace
metadata:
  name: flux
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flux
  namespace: flux
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: flux
rules:
  - apiGroups: ['*']
    resources: ['*']
    verbs: ['*']
  - nonResourceURLs: ['*']
    verbs: ['*']
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: flux
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flux
subjects:
- kind: ServiceAccount
  name: flux
  namespace: flux

Now any pod in the flux namespace that is given the service account named flux will get admin access to the entire cluster. Now we’ll define memcached, a dependency of Flux which caches container image to speed things up.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memcached
  namespace: flux
spec:
  replicas: 1
  selector:
    matchLabels:
      name: memcached
  template:
    metadata:
      labels:
        name: memcached
    spec:
      containers:
      - name: memcached
        image: memcached:1.5.15
        imagePullPolicy: IfNotPresent
        args:
        - -m 512   # Maximum memory to use, in megabytes
        - -I 5m    # Maximum size for one item
        - -p 11211 # Default port
        ports:
        - name: clients
          containerPort: 11211
        securityContext:
          runAsUser: 11211
          runAsGroup: 11211
          allowPrivilegeEscalation: false
---
apiVersion: v1
kind: Service
metadata:
  name: memcached
  namespace: flux
spec:
  ports:
  - name: memcached
    port: 11211
  selector:
    name: memcached

Now Flux is going to need an SSH known_hosts file to help it recognize the Git server to which it’s going to be connecting. In the following command replace github.com with whichever Git server you’ll be using: ssh-keyscan github.com . Use the results of this in the following manifest, replacing the line starting with github.com .

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: flux-ssh-config
  namespace: flux
data: # known_host lines generated from output of `ssh-keyscan github.com` and such commands
  known_hosts: |
    github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGm...

Let’s also create an empty secret for Flux to use:

---
apiVersion: v1
kind: Secret
metadata:
  name: flux-git-deploy
  namespace: flux
type: Opaque

Now let’s finish off by defining the Flux daemon itself:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flux
  namespace: flux
spec:
  replicas: 1
  selector:
    matchLabels:
      name: flux
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: flux
    spec:
      serviceAccountName: flux
      containers:
      - name: flux
        image: docker.io/fluxcd/flux:1.14.2
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
        args:
        - --memcached-service= # this can be left blank
        - --ssh-keygen-dir=/var/fluxd/keygen
        - --git-url=ssh://git@github.com/j18e/k8s-deployment.git
        - --git-branch=master
        - --git-path=sync
        - --git-user=flux-cd
        - --git-email=flux-cd@j18e.com
        - --git-poll-interval=1m0s
        - --sync-interval=1m0s
        volumeMounts:
        - name: git-key
          mountPath: /etc/fluxd/ssh
        - name: git-keygen
          mountPath: /var/fluxd/keygen
        - name: ssh-config
          mountPath: /root/.ssh
      volumes:
      - name: git-key
        secret:
          secretName: flux-git-deploy
          defaultMode: 0400
      - name: git-keygen
        emptyDir:
          medium: Memory
      - name: ssh-config
        configMap:
          name: flux-ssh-config

Make sure you change the --git-url option to the SSH URI of the Git repo you’re using, and the --git-branch to your chosen branch. --git-path tells Flux to only look for manifests under the given path in the Git repo. The --git-user and --git-email are relevant for when Flux commits changes to the Git repo. I’ve set the poll and sync intervals to one minute each, ensuring that Flux will be pretty quick to pick up new changes.

Take all these manifests and apply them to your cluster with kubectl apply -f flux-init/ (wherever your files are).

Check that both Flux and memcached are running and healthy with kubectl -n flux get pods .

Giving Flux permissions on your Git repo

Now that Flux is up and running in your cluster, you’ll need to allow it to read/write to your Git repo. If you don’t plan on using the continuous deployment feature, you won’t need the write part of that, but it will at least need to be able to check out the repo over SSH.

To find Flux’s public key, we’ll need to get the log output of the container.

POD_NAME=$(kubectl -n flux get pods -o name | grep 'pod/flux')
kubectl -n flux logs $POD_NAME | grep 'identity.pub='

Now from the output, copy the public key. It’s between the quotation marks after identity.pub= .

Log into your Git server and go to the settings of the repository you’ll be using to sync manifests. There should be a section to add public keys and permissions. Add the copied public key and give it write permissions (or read if you don’t want to use continuous deployment).

If you wait a couple of minutes and look at Flux’s log again you should be able to see that it’s successfully checked out the Git repo. If there are errors, you’ll need to do some troubleshooting. I’ve discovered it can be fairly finicky with the Git URL.

Synchronize resources between Git and Kubernetes

Let’s try committing a manifest to our Git repo, and seeing if Flux picks up on it. Add the following contents to a new file resource.yml and commit it to the sync directory of your Git repo.

--- 
kind: Namespace
apiVersion: v1
metadata:
  name: mytestns

Now wait a few minutes and run kubectl get ns . Does mytestns come up? If not, you have a problem, and may want to check Flux’s logs again. But if it does come up, you’ve successfully set up Flux!

Synchronize a deployment with a Docker repository

Let’s try out the final key feature of Flux. To do this, we’re going to need a Kubernetes deployment and a Docker image. So first I’m going to create a new image and push it to Docker hub. I’m using my own Docker account so you should switch out j18e with your own Docker username:

mkdir -p mynewapp; cd mynewapp 
echo 'FROM nginx' > Dockerfile
docker build -t j18e/mynewapp:0.1.1 .
docker push j18e/mynewapp:0.1.1

Now let’s create a deployment that uses this new image. Put the following contents into a new YAML file and check it into the sync directory of your Git repo:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mynewapp
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.mynewapp: "glob:*"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mynewapp
  template:
    metadata:
      labels:
        app: mynewapp
    spec:
      containers:
      - name: mynewapp
        image: j18e/mynewapp:0.1.1
        ports:
        - containerPort: 80

Notice the two annotations in our new resource. The first tells Flux that it should be automating the deployment’s container images, and the second provides a pattern to match against the tags of any new images that get pushed to the repository. Currently it’s set up to take any new images, but you could just as easily set the tag to "glob:master-*" to make sure it only matches images whose tags begin with master- . This is the connection between your continuous integration server and Flux.

Wait a couple minutes and make sure the deployment has been applied with kubectl get deployment mynewapp -o yaml . You should be able to see the image mynewapp:0.1.1 listed in the output.

Now push a new version of your app to Docker hub. You don’t need to actually change the image; a new tag will do just fine:

docker tag j18e/mynewapp:0.1.1 j18e/mynewapp:0.1.2
docker push j18e/mynewapp:0.1.2

Wait another couple of minutes and again run kubectl get deployment mynewapp -o yaml . Do you see the new tag in the deployment? You can look at Flux’s logs as well to see it picking up on the change.

Run git pull in your repo and see if Flux managed to commit the change to Git. If all of the above worked, you’ve successfully tested the main features of Flux and can start managing your operations environment using Gitops!