Earlier this month I presented at the Rust Zürich meetup group about how we re-implemented a critical piece of code used in our workflows. In this presentation I walked the audience through the migration of a key component of Project Syn (our Kubernetes configuration management framework) from Python to Rust.

We tackled this project to address the longer-than-15-minute CI pipeline runs we needed to roll out changes to our Kubernetes clusters. Thanks to this rewrite (and some other improvements) we’ve been able to reduce the CI pipeline runs to under 5 minutes.

The related pull request, available on GitHub, was merged 5 days ago, and includes the mandatory documentation describing its functionality.

I’m also happy to report that this talk was picked up by the popular newsletter „This Week in Rust“ for its 538th edition! You can find the recording of the talk, courtesy of the Rust Zürich meetup group organizers, on YouTube.

Our customers trust us with their most precious resource: their information systems. Our job is to keep the underlying systems running, updated, and most importantly, secure.

Project Syn with its Commodore Components is one of the primary weapons in our arsenal to configure and thus protect those systems. Thanks to its GitOps approach, we can ensure that all Kubernetes clusters are always running the latest and (hopefully) most secure version possible.

But just like any other software package, Project Syn brings its complexity: we must keep it safe and sound, which means watching over its container images, its Helm charts, and all of the Commodore Components we use every day.

As you can imagine, juggling so many different software packages is a considerable task; now, think about all of their upstream dependencies (most of them are container images and helm charts, but also Go and Python are a part of the mix). The complexity of the task exponentially increases.

How do we cope with this? Well, as usual, standing on the shoulder of giants. In this case, Renovate.

Renovate has been created to manage this complexity, whether container images, Helm charts, or upstream dependencies. But understandably enough, Renovate per se does not know anything about Commodore Components (at least not yet!), and in particular, it does not know about the Project Syn configuration hierarchy and how to find dependencies within that hierarchy.

So, what’s an Open Source developer to do? We forked Renovate, of course, and adapted it to our needs. How?

1. We added the Project Syn configuration hierarchy as a new Manager.
2. We reused the existing datasource to detect new versions of our Commodore Components.

Then we configured our own Renovate fork on all the repositories holding our source code and started getting notified via pull requests whenever there was a new dependency version. Voilà!

With this approach, we have been able to automate much work and avoid using outdated software by automatically being notified of new versions. No more forgotten updates!

We also decided to use „golden files“ to test our Commodore Components; this, in turn, meant that we could not merge PRs created by Renovate in case of failure. For those cases, we also taught Renovate how to update those golden files if needed.

The pull request „Update dependency ghcr.io/appuio/control-api to v0.8.1 – autoclosed #29“ is a live example of this mechanism in action, and you’re most welcome to check it out.

We’re very happy to announce a new member of the Project Syn family: the new Commodore Components Hub. This is the new central point of reference and information for all Commodore components available in GitHub.

Not only does the Commodore Component Hub list all existing components on GitHub, it also automatically imports and indexes the documentation of each and every one, written in Asciidoc and formatted as an Antora documentation site. This makes it very easy to find the perfect component that suits your needs.

The source code used to generate the Commodore Component Hub was born during our recent VSHN HackDay; it’s written in Python and 100% open source (of course!). Check it out in GitHub.

Get your Component on the Hub!

To add your own Commodore Component to the Hub, it’s very easy: just add the commodore-component tag to your project on GitHub, and voilà! The Commodore Component Hub is rebuilt every day at every hour from 6 AM to 7 PM (CET time).

We recommend that you write the documentation of your component with Asciidoc in the docs folder of your component. This will ensure that users will be able to find your component, and most important, also learn how to use it properly.

We look forward to featuring your Commodore Components on the Hub!

Without further ado, we’re announcing the release 0.2 of the Project Syn tools.
Since the first public release mid-March this year (read more about it in First Pre-Release of Project Syn Tools) we used the tools on a daily basis, in particular for the development of our new product „VSHN Syn Support“. And of course we have incorporated all of that experience in the source code. The main features are now in place, and are getting better and better on a daily basis.

New Features and Improvements

When reading the announcement of a new version, engineers are always interested in new features and improvements. So these are the most important new additions since 0.1:

• Everything required for setting up a new cluster (GitOps repository, cluster config file in Tenant configuration, Vault secrets, and more) is now fully automated. One API call to register a new cluster and you’re done.
• In parallel to the creation of clusters, we have also automated all steps required to decommission them (Repo deletion, Vault secret cleanup, and more). Just delete it and everything is gone (of course, there are preventive measures in place to not make this an Uh-oh moment).
• Commodore got a lot of improvements: for local development, and for developing new components with a comprehensive cookiecutter template.

Document All The Things

Besides implementing new features and fixing bugs we put a lot of effort into the documentation. The main documentation page https://syn.tools/ got a completely new structure and a huge amount of new content. We’re in the process of adding new pages frequently, so make sure to check it out every so often.
Before 0.2 it was hard to get started with Project Syn and to understand what it was all about. To solve that issue we wrote the following introductions:

• An overview of all the features offered by Project Syn
• The architecture and how all the tools are working together
• A comprehensive getting started guide in a tutorial format

Our next goal is to document the concepts behind configuration management with Commodore in detail.

Commodore Components on GitHub

An important building block of Project Syn are Commodore Components. Over the past months we’ve written and open sourced more than 10 Commodore Components on GitHub. They offer the flexibility to install and configure Kubernetes system services, adapted to their respective distribution and infrastructure.
These Commodore Components can be found by searching for the „commodore-component“ topic on GitHub.
We are writing and refining more and more Components every day. We are going to publish some guidelines about how to write Commodore Components (one specifically for OpenShift 4 Components is already available) and eventually enforce them via CI jobs and policies.
An upcoming Component Writing Tutorial will help beginners to start writing own Components or contribute to existing ones.

The Road to 1.0 and Contributions

What we learnt while working on Project Syn over the last few months gave us a very clear picture of what we want to achieve in version 1.0. The roadmap contains the most important topics:

• Documentation! We have to and will put a lot of effort into documentation, be it tutorials, how-to guides, or explanations.
• Full Commodore automation to automate and decentralize the cluster catalog compilation process.
• Developer experience improvements for simplifying the development of Commodore Components even further.
• Engineering of a new tool helping users to launch managed services on any Kubernetes cluster.
• Cluster provisioning automation integration, to leverage third party tools for automatically bootstrapping Kubernetes clusters.

This is not all; check the more detailed roadmap on the Project Syn page for more. The GitHub project will grow with issues over the next few weeks.
If you think that this sounds interesting and you would like to contribute, we now have an initial Contribution Guide available and are very open to suggestions and pull requests. Just get in contact with us if you’re interested.

Our Product: VSHN Syn Support

Besides the Open Source project we were also working on defining what added value you can get from VSHN. We call this product „VSHN Syn Support.“ If you’re interested in getting commercial support from VSHN for Project Syn on a Managed Kubernetes Cluster based on OpenShift 4 or Rancher, get in touch with us. More information about VSHN Syn Support can be found here.

One of the most common questions we got from companies moving to Kubernetes has always had to do with backups: how can we ensure that the information in our pods and services can be quickly and safely restored in case of problems?
This situation is so common that we VSHN decided to tackle it with our own Kubernetes operator for backups, which we called K8up.
Note: This tutorial is available in three versions, each in its own branch of the GitHub repository bundled with this text:

• Minikube in the master branch.
• k3s in the k3d branch.
• Red Hat CodeReady Containers (OpenShift 4.3) in the openshift branch.

1. What is K8up?

K8up (pronounced „/keɪtæpp/“ or simply „ketchup“) is a Kubernetes operator distributed via a Helm chart, compatible with OpenShift and plain Kubernetes. It allows cluster operators to:

• Backup all PVCs marked as ReadWriteMany or with a specific annotation.
• Perform individual, on-demand backups.
• Schedule backups to be executed on a regular basis.
• Schedule archivals (for example to AWS Glacier), usually executed in longer intervals.
• Perform „Application Aware“ backups, containing the output of any tool capable of writing to stdout.
• Check the backup repository for its integrity.
• Prune old backups from a repository.
• Based on top of Restic, it can save backups in Amazon S3 buckets, and Minio (used we’ll see in this tutorial.)

K8up is written in Go and is an open source project hosted in GitHub.

2. Introduction

This tutorial will show you how to backup a small Minikube cluster running on your laptop. We are going to deploy Minio, MariaDB and WordPress on this cluster, and create a blog post in our new website. Later we’re going to „deface“ it, so that we can safely restore it later. Through this process, you are going to learn more about K8up and its capabilities.
Note: All the scripts and YAML files are available in GitHub: github.com/vshn/k8up-tutorial.

2.1 Requirements

This tutorial has been tested in both Linux (Ubuntu 18.04) and macOS (10.15 Catalina.) Please install the following software packages before starting:

• Make sure PyYAML 5.1 or later is installed: pip install PyYAML==5.1
• The kubectl command.
• The Restic backup application.
• The latest version of Minikube (1.9 at the time of this writing.)
• Helm, required to install K8up in your cluster.
• k9s to display the contents of our clusters on the terminal.
• jq, a lightweight and flexible command-line JSON processor.

3. Tutorial

It consists of six steps to be executed in sequence:

1. Setting up the cluster.
2. Creating a blog.
3. Backing up the blog.
4. Restoring the contents of the backup.
5. Scheduling regular backups.
6. Cleaning up.

Let’s get started!

3.1 Setting up the cluster

Note: The operations of this step can be executed at once using the scripts/1_setup.sh script.

1. Start your minikube instance with a configuration slightly more powerful than the default one:
   • minikube start --memory 4096 --disk-size 60g --cpus 4
     Note: On some laptops, running Minikube on battery power severely undermines its performance, and pods can take really long to start. Make sure to be plugged in to power before starting this tutorial.
2. Copy all required secrets and passwords into the cluster:
   • kubectl apply -k secrets
3. Install and run Minio in your cluster:
   • kubectl apply -k minio
4. Install MariaDB in your cluster:
   • kubectl apply -k mariadb
5. Install WordPress:
   • kubectl apply -k wordpress
6. Install K8up in Minikube:
   • helm repo add appuio charts.appuio.ch
   • helm repo update
   • helm install appuio/k8up --generate-name --set k8up.backupImage.tag=v0.1.8-root

After finishing all these steps, check that everything is running; the easiest way is to launch k9s and leave it running in its own terminal window, and of course you can use the usual kubectl get pods.
Tip: In k9s you can easily delete a pod by going to the „Pods“ view (type :, write pods at the prompt and hit Enter), selecting the pod to delete with the arrow keys, and hitting the CTRL+D key shortcut.

The asciinema movie below shows all of these steps in real time.

 

3.2 Viewing Minio and WordPress on a browser

Note: The operations of this step can be executed at once using the scripts/2_browser.sh script.

1. Open WordPress in your default browser using the minikube service wordpress command. You should see the WordPress installation wizard appearing on your browser window.
2. Open Minio in your default browser with the minikube service minio command.
   • You can login into minio with these credentials: access key minio, secret key minio123.

3.2.1 Setting up the new blog

Follow these instructions in the WordPress installation wizard to create your blog:

1. Select your language from the list and click the Continue button.
2. Fill the form to create new blog.
3. Create a user admin.
4. Copy the random password shown, or use your own password.
5. Click the Install WordPress button.
6. Log in to the WordPress console using the user and password.
   • Create one or many new blog posts, for example using pictures from Unsplash.
7. Enter some text or generate some random text using a Lorem ipsum generator.
8. Click on the „Document“ tab.
9. Add the image as „Featured image“.
10. Click „Publish“ and see the new blog post on the site.

3.3 Backing up the blog

Note: The operations of this step can be executed at once using the scripts/3_backup.sh script.
To trigger a backup, use the command kubectl apply -f k8up/backup.yaml. You can see the job in the „Jobs“ section of k9s.
Running the logs command on a backup pod brings the following information:

$ kubectl logs backupjob-1564752600-6rcb4
No repository available, initialising...
created restic repository edaea22006 at s3:http://minio:9000/backups
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
Removing locks...
created new cache in /root/.cache/restic
successfully removed locks
Listing all pods with annotation appuio.ch/backupcommand in namespace default
Adding default/mariadb-9588f5d7d-xmbc7 to backuplist
Listing snapshots
snapshots command:
0 Snapshots
backing up via mariadb stdin...
Backup command: /bin/bash, -c, mysqldump -uroot -p"${MARIADB_ROOT_PASSWORD}" --all-databases
done: 0.00%
backup finished! new files: 1 changed files: 0 bytes added: 4184711
Listing snapshots
snapshots command:
1 Snapshots
sending webhook Listing snapshots
snapshots command:
1 Snapshots
backing up...
Starting backup for folder wordpress-pvc
done: 0.00%
backup finished! new files: 1932 changed files: 0 bytes added: 44716176
Listing snapshots
snapshots command:
2 Snapshots
sending webhook Listing snapshots
snapshots command:
2 Snapshots
Removing locks...
successfully removed locks
Listing snapshots
snapshots command:
2 Snapshots

If you look at the Minio browser window, there should be now a set of folders that appeared out of nowhere. That’s your backup in Restic format!

3.3.1 How does K8up work?

K8up runs Restic in the background to perform its job. It will automatically backup the following:

1. All PVCs in the cluster with the ReadWriteMany attribute.
2. All PVCs in the cluster with the k8up.syn.tools/backup: "true" annotation.

The PVC definition below shows how to add the required annotation for K8up to do its job.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wordpress-pvc
  labels:
    app: wordpress
  annotations:
    k8up.syn.tools/backup: "true"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Just like any other Kubernetes object, K8up uses YAML files to describe every single action: backups, restores, archival, etc. The most important part of the YAML files used by K8up is the backend object:

backend:
  repoPasswordSecretRef:
    name: backup-repo
    key: password
  s3:
    endpoint: http://minio:9000
    bucket: backups
    accessKeyIDSecretRef:
      name: minio-credentials
      key: username
    secretAccessKeySecretRef:
      name: minio-credentials
      key: password

This object specifies two major keys:

• repoPasswordSecretRef contains the reference to the secret that contains the Restic password. This is used to open, read and write to the backup repository.
• s3 specifies the location and credentials of the storage where the Restic backup is located. The only valid option at this moment is an AWS S3 compatible location, such as a Minio server in our case.

3.4 Restoring a backup

Note: The operations of this step can be executed at once using the scripts/4_restore.sh script.
Let’s pretend now that an attacker has gained access to your blog: we will remove all blog posts and images from the WordPress installation and empty the trash.

Oh noes! But don’t worry: thanks to K8up you can bring your old blog back in a few minutes.
There are many ways to restore Restic backups, for example locally (useful for debugging or inspection) and remotely (on PVCs or S3 buckets, for example.)

3.4.1 Restoring locally

To restore using Restic, set these variables (in a Unix-based system; for Windows, the commands are different):

export KUBECONFIG=""
export RESTIC_REPOSITORY=s3:$(minikube service minio --url)/backups/
export RESTIC_PASSWORD=p@ssw0rd
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123

Note: You can create these variables simply running source scripts/environment.sh.
With these variables in your environment, run the command restic snapshots to see the list of backups, and restic restore XXXXX --target ~/restore to trigger a restore, where XXXXX is one of the IDs appearing in the results of the snapshots command.

3.4.2 Restoring the WordPress PVC

K8up is able to restore data directly on specified PVCs. This requires some manual steps.

• Using the steps in the previous section, „Restore Locally,“ check the ID of the snapshot you would like to restore:

$ source scripts/environment.sh
$ restic snapshots
$ restic snapshots XXXXXXXX --json | jq -r '.[0].id'

• Use that long ID in your restore YAML file k8up/restore/wordpress.yaml:
  • Make sure the restoreMethod:folder:claimName: value corresponds to the Paths value of the snapshot you want to restore.
  • Replace the snapshot key with the long ID you just found:

apiVersion: backup.appuio.ch/v1alpha1
kind: Restore
metadata:
  name: restore-wordpress
spec:
  snapshot: 00e168245753439689922c6dff985b117b00ca0e859cc69cc062ac48bf8df8a3
  restoreMethod:
    folder:
      claimName: wordpress-pvc
  backend:

• Apply the changes:
  • kubectl apply -f k8up/restore/wordpress.yaml
  • Use the kubectl get pods commands to see when your restore job is done.

Tip: If you use the kubectl get pods --sort-by=.metadata.creationTimestamp command to order the pods in descending age order; at the bottom of the list you will see the restore job pod.

3.4.3 Restoring the MariaDB pod

In the case of the MariaDB pod, we have used a backupcommand annotation. This means that we have to „pipe“ the contents of the backup into the mysql command of the pod, so that the information can be restored.
Follow these steps to restore the database:

1. Retrieve the ID of the MariaDB snapshot:
   • restic snapshots --json --last --path /default-mariadb | jq -r '.[0].id'
2. Save the contents of the backup locally:
   • restic dump SNAPSHOT_ID /default-mariadb > backup.sql
3. Get the name of the MariaDB pod:
   • kubectl get pods | grep mariadb | awk '{print $1}'
4. Copy the backup into the MariaDB pod:
   • kubectl cp backup.sql MARIADB_POD:/
5. Get a shell to the MariaDB pod:
   • kubectl exec -it MARIADB_POD — /bin/bash
6. Execute the mysql command in the MariaDB pod to restore the database:
   • mysql -uroot -p"${MARIADB_ROOT_PASSWORD}" < /backup.sql

Now refresh your WordPress browser window and you should see the previous state of the WordPress installation restored, working and looking as expected!

3.5 Scheduling regular backups

Note: The operations of this step can be executed at once using the scripts/5_schedule.sh script.
Instead of performing backups manually, you can also set a schedule for backups. This requires specifying the schedule in cron format.

backup:
  schedule: '*/2 * * * *'    # backup every 2 minutes
  keepJobs: 4
  promURL: http://minio:9000

Tip: Use crontab.guru to help you set up complex schedule formats in cron syntax.
The schedule can also specify archive and check tasks to be executed regularly.

archive:
  schedule: '0 0 1 * *'       # archive every week
  restoreMethod:
    s3:
      endpoint: http://minio:9000
      bucket: archive
      accessKeyIDSecretRef:
        name: minio-credentials
        key: username
      secretAccessKeySecretRef:
        name: minio-credentials
        key: password
check:
  schedule: '0 1 * * 1'      # monthly check
  promURL: http://minio:9000

Run the kubectl apply -f k8up/schedule.yaml command. This will setup an automatic schedule to backup the PVCs every 5 minutes (for minutes that are divisors of 5).
Wait for at most 2 minutes, and run the restic snapshots to see more backups piling up in the repository.
Tip: Running the watch restic snapshots command will give you a live console with your current snapshots on a terminal window, updated every 2 seconds.

3.6 Cleaning up the cluster

Note: The operations of this step can be executed at once using the scripts/6_stop.sh script.
When you are done with this tutorial, just execute the minikube stop command to shut the cluster down. You can also minikube delete it, if you would like to get rid of it completely.

4. Conclusion

We hope that this walkthrough has given you a good overview of K8up and its capabilities. But it can do much more than that! We haven’t talked about the archive, prune, and check commands, or about the backup of any data piped to stdout (called „Application Aware“ backups.) You can check these features in the K8up documentation website where they are described in detail.
K8up is still a work in progress, but it is already being used in production in many clusters. It is also an open source project, and everybody is welcome to use it freely, and even better, to contribute to it!