Tech

Happy 30th Birthday, Linux!

25. Aug 2021

On the evening of August 25th, 1991, Linus Torvalds wrote a short message on the comp.os.minix Usenet newsgroup:

I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones.

Fast-forward 30 years, and to make a long story short, well: Linux is now both big and professional. It’s nearly everywhere, hidden behind the shiny graphical user interface of your smartphone or tablet (well, except for iOS users), in your gaming box, in your car, and even on Mars.

I discovered Linux as a young boy still at school, in 1998. It was in a book store where I was attracted by a shiny package (I had no idea what Linux was) called “SuSE Linux 6.0” (nowadays available for download from archive.org), and since then I couldn’t stop working with it.

Over time, I got more and more into Linux, and it became an important hobby, with all my PCs running Linux, and also on my home server under my desk at home. Many years later, in 2011, I could finally start working full-time as a Linux administrator. My desktop computer has been only powered by Linux since around 2004, and I’ve been a big fan of KDE ever since.

The author’s original SuSE Linux 6.0 CDs, originally released in December 1998.
More memorabilia from early Linux versions.

Today, the Linux Kernel is more present, yet less visible than ever. We interact with it on containers and in VMs on the cloud, and it gets more and more abstracted away, deep down in serverless architectures, making the Kernel even more invisible than ever before. Albeit out of sight of most users, Linux has become much more solid, mature, and pervasive, and does its great job behind the scenes without interruption.

Linux empowers VSHN at all levels; not only it is the dominating technology we use every day, empowering Kubernetes, containers, and cloud VMs, but it is also the operating system that the majority of VSHNeers (around 66%, according to an internal poll) use for their daily job, the remaining third using macOS or Windows.

Some numbers: of those two thirds of VSHNeers that use Linux every day in their laptops, 61% chose Ubuntu (or one of its various derivatives); 17% use Arch, 11% Fedora, and others use DebianMint, and other distributions. Some even contemplate switching to Qubes OS soon! As for desktop environments, around 35% use GNOME, 25% use KDE, 20% use i3, and 6% use Cinnamon.

Each one of us VSHNeers has a unique feeling about Linux; here are some thoughts about what Linux means to us:

Before using Linux, I was primarily focused on how to use and work with computer systems. With the switch to Linux I started to understand how they actually work.

What I really appreciate about Linux is that it’s (still relatively) lightweight, powerful, transparent and adaptable. I do heavyweight gaming and video livestreaming on the same OS that runs my file servers and backup systems (not all on the same machine, don’t worry). Even my car and my television run Linux! This universality combined with the permissive licenses means that whenever one industry improves Linux (the kernel), every other industry profits.

I originally just wanted to play Minecraft with my friends. Suddenly I had to learn how to host this on a Linux server, which then turned into a fascination on how things work on the backstage. It’s the urge to see how our modern society works!

Linux is the operating system of our software-driven world.

On to the next 30 years of Linux!

Tobias Brunner

Tobias Brunner is working since over 20 years in IT and more than 15 years with Internet technology. New technology has to be tried and written about.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Press Tech

VSHN Top Ten Open Source Contributor in Switzerland

4. Aug 2021

Open Source is without any doubt one of the most important forces shaping the business of software in the 21st century. And Switzerland, after some hesitant first steps in the 2000’s, has now fully embraced it; many organizations are now contributing lots of code for the community, collaborating with one another, and helping shape an open future for our craft. And Open Source is undoubtedly a major factor contributing to Switzerland topping the UN innovation ranking for the past 10 years in a row.

For VSHN, Open Source is part of our culture. We release new projects and products continuously, and gladly collaborate with projects all over the country, providing pull requests, bugs reports, and feedback. We actively advocate for our customers to adopt Open Source as well; they can learn more about the Open Source advantage to their businesses in our website.

And our commitment pays off; we’ve been consistently figuring in the top ten of Swiss organizations contributing to Open Source. We are thrilled and proud to share the stage with such prestigious names as Camptocamp, CERN, Exoscale, Swisscom, the University of Bern, Zühlke and many others!

Our major contributions at this moment are:

  • Project Syn, our suite of tools to manage large numbers of Kubernetes clusters from a central location.
    • Commodore, one of the members of the Project Syn family, can be extended through components. The full component ecosystem is featured in our recently released Commodore Components Hub, including not only our own components, but also those created by third parties.
  • K8up, our Kubernetes backup operator, which recently released its version 1.0.
  • But there is more, so much more! Check out our GitHub profiles: VSHNAPPUiO, and Project Syn.

We are also contributing and sponsoring the Linux Foundation and Cloud Native Computing Foundation, CH Open, Parldigi, Opendata.ch and many more and we support the Swiss Open Source Study 2021.

On The Press

These contributions are making the headlines:

Markus Speth

Markus is VSHN's CEO and one of the General Managers.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Project Syn Tech

The New Commodore Components Hub

28. Jul 2021

We’re very happy to announce a new member of the Project Syn family: the new Commodore Components Hub. This is the new central point of reference and information for all Commodore components available in GitHub.

Commodore Components Hub

Not only does the Commodore Component Hub list all existing components on GitHub, it also automatically imports and indexes the documentation of each and every one, written in Asciidoc and formatted as an Antora documentation site. This makes it very easy to find the perfect component that suits your needs.

The source code used to generate the Commodore Component Hub was born during our recent VSHN HackDay; it’s written in Python and 100% open source (of course!). Check it out in GitHub.

Get your Component on the Hub!

To add your own Commodore Component to the Hub, it’s very easy: just add the commodore-component tag to your project on GitHub, and voilà! The Commodore Component Hub is rebuilt every day at every hour from 6 AM to 7 PM (CET time).

We recommend that you write the documentation of your component with Asciidoc in the docs folder of your component. This will ensure that users will be able to find your component, and most important, also learn how to use it properly.

We look forward to featuring your Commodore Components on the Hub!

Tobias Brunner

Tobias Brunner is working since over 20 years in IT and more than 15 years with Internet technology. New technology has to be tried and written about.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Benchmarking Kubernetes Storage Solutions

23. Jul 2021

One of the most difficult subjects in the world of Kubernetes is storage. In our day-to-day operations we’ve often had to choose the best storage solution for our customers, but in a changing landscape of requirements and technical offerings, such choice becomes a major task.

Faced to many options, we decided to benchmark storage solutions in real life conditions, to generate the data required for a proper decision. In this article we’re going to share with you our methodology, our results, and our final choice.

The chosen storage providers for this evaluation were:

All of these benchmarks (except Gluster) were run on an OpenShift 4.7 cluster on Exoscale VMs.

We benchmarked Ceph both with unencrypted and encrypted storage for the OSDs (object-storage daemons). We included Gluster in our evaluation for reference and comparison only, as that’s the solution we offered for storage on OpenShift Container Platform 3.x. We never intended to use Gluster as the storage engine for our new Kubernetes storage cluster product.

Methodology

We first created a custom Python script driving kubestr, which in turn orchestrates Fio. This script performed ten (10) iterations for each benchmark, each of which included the following operations in an isolated Fio run:

  • Read iops
  • Read bandwidth
  • Write iops, with different frequencies of calls to fsync:
    • no fsync calls during each benchmark iteration
    • an fsync call after each operation (“fsync=1”)
    • an fsync call after every 32 operations (“fsync=32”)
    • an fsync call after every 128 operations (“fsync=128”)
  • Write bandwidth, with different frequencies of calls to fsync:
    • no fsync calls during each benchmark iteration
    • an fsync call after each operation (“fsync=1”)
    • an fsync call after every 32 operations (“fsync=32”)
    • an fsync call after every 128 operations (“fsync=128”)

This is the Fio configuration used for benchmarking:

[global]
randrepeat=0
verify=0
ioengine=libaio
direct=1
gtod_reduce=1
[job]
name=JOB_NAME     (1)
bs=BLOCKSIZE      (2)
iodepth=64
size=2G
readwrite=OP      (3)
time_based
ramp_time=5s
runtime=30s
fsync=X           (4)
  1. We generate a descriptive fio job name based on the benchmark we’re executing. The job name is generated by taking the operation (“read” or “write”) and the measurement (“bw” or “iops”) and concatenating them as “OP_MEASUREMENT”, for example “read_iops”.
  2. The blocksize for each operation executed by fio. We use a blocksize of 4K (4kB) for IOPS benchmarks and 128K (128kB) for bandwidth benchmarks.
  3. The IO pattern which fio uses for the benchmark. randread for read benchmarks. randwrite for write benchmarks.
  4. The number of operations to batch between fsync calls. This parameter doesn’t have an influence on read benchmarks.

If writing to a file, issue an fsync(2) (or its equivalent) of the dirty data for every number of blocks given. For example, if you give 32 as a parameter, fio will sync the file after every 32 writes issued. If fio is using non-buffered I/O, we may not sync the file. The exception is the sg I/O engine, which synchronizes the disk cache anyway. Defaults to 0, which means fio does not periodically issue and wait for a sync to complete. Also see end_fsync and fsync_on_close.

Fio documentation

Results

The following graph, taken from the full dataset for our benchmark (available for download and study) shows the type of comparison performed across all considered solutions.

Figure 1. Read IOPS (higher is better)

The table below gives an overview over the data gathered during our evaluation (each column shows mean ± standard deviation):

Storage solutionread IOPSread bandwidth MB/swrite IOPS, no fsyncwrite bandwidth MB/s, no fsyncwrite IOPS, fsync=1write bandwidth MB/s, fsync=1
OCS/Rook.io Ceph RBD (unencrypted OSDs)42344.21 ± 885.521585 ± 32.6559549.14 ± 371.11503.208 ± 12.544305.18 ± 15.6535.591 ± 1.349
OCS/Rook.io CephFS (unencrypted OSDs)44465.21 ± 1657.911594 ± 82.5229978.00 ± 456.97512.788 ± 8.0498808.47 ± 357.87452.086 ± 10.154
OCS/Rook.io Ceph RBD (encrypted OSDs)36303.06 ± 2254.871425 ± 59.7206292.75 ± 424.91310.520 ± 63.047225.00 ± 12.1122.804 ± 1.031
OCS/Rook.io CephFS (encrypted OSDs)36343.35 ± 1234.931405 ± 92.8686020.49 ± 251.16278.486 ± 49.1015004.28 ± 152.01291.729 ± 17.367
Longhorn (unencrypted backing disk)11298.36 ± 664.99295.458 ± 25.458111.197 ± 10.3225975.43 ± 697.14391.57 ± 26.1129.993 ± 1.544
Gluster22957.87 ± 345.40976.511 ± 45.2682630.89 ± 69.21531.88 ± 48.22133.563 ± 11.45543.549 ± 1.656

Unencrypted Rook/OCS numbers are from OCS, encrypted Rook/OCS numbers from vanilla Rook.

Conclusion

After careful evaluation of the results shown above, we chose Rook to implement our APPUiO Managed Storage Cluster product. Rook allows us to have a single product for all the Kubernetes distributions we’re offering at VSHN.

We have released the scripts on GitHub for everyone to verify our results, and published even more data in our Products documentation. Feel free to check the data, run these tests in your own infrastructure, and of course, your pull requests are more than welcome.

Simon Gerber

Simon Gerber is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Networking Security Part 3: Trust and Certificate Chains

16. Jun 2021

Welcome to the third and last installment of our “Networking Security” blog post series. If you didn’t read the previous ones yet, go ahead and read the first two parts:

In this article we’ll talk about trust, and how it’s implemented thanks to certificate chains.

Chain of Trust

As explained in the previous posts, a server certificate can be signed by a CA certificate in order to confirm their identity. We trust that the CA has verified the contents of the server certificate before signing it, and hence we trust that the contents of the server certificate are valid.

Now, since a certificate that is used to sign other certificates, can in turn be signed by another certificate, we can build a whole “chain of trust”:

  • At the top, there is the Root Certificate (also called the Anchor). It is signed by itself (self-signed).
  • In between, there may be one or more Intermediate Certificates. They are signed by either other intermediates or the root certificate.
  • On the bottom is our server (or client) certificate, called the Leaf Certificate.

In order to verify trust of a leaf certificate, we follow the “certificate chain” upwards until we reach a root certificate we already know and trust.

Root Certificates

In order to know which root certificates to trust, we must have a “trust store” containing all the root certificates. This store is usually provided by the operating systems, but some clients (most notably web browsers) have their own “trust store”.

In order for a CA to have its root certificate accepted into the trust store of an operating system or browser, they must meet certain quality criterias. They must prove that they take CSR validation seriously and that they have measures put in place to keep their private key confidential.

Having their private key stolen by an attacker is the absolute worst-case scenario for any CA. It would mean that the attacker could create and sign ANY certificate using the CA’s cert! And since the CA’s certificate is trusted by all operating systems, clients connecting to the attacker would be none the wiser!

The only way to react to a compromised key is to rotate it, that is, to create a new one using a new private key. This however would mean that the certificate also has to be replaced in all trust stores. Since this can take a long time for some operating systems or clients, CA’s usually only use Intermediate Certificates to sign your certificates, and store the root key somewhere safe (and by safe I mean printed out on a piece of paper and stored in an actual safe.)

Intermediate Certificates

As explained, CA’s usually use an Intermediate Certificate to sign your CSR’s. An Intermediate Certificate, in turn, can be signed by either the Root Certificate or another Intermediate Certificate. This means that, in theory, there could be any number of intermediate certificates between your server (leaf) certificate and the Root. In practice, however, there is usually only one or two intermediates.

CA Bundles

In order to verify a certificate, a client must possess the certificate that was used to sign it. So in order to verify the whole chain, it must have each intermediate certificate as well as the root certificate. The root certificate we already have in our trust store. But what about the intermediates?

When a CA has signed your CSR and sends you back the certificate, they will also send you all the intermediate certificates between your server cert and the Root. This is called the “CA bundle” or “CA chain”.

When we install our Server certificate in our web server, we also provide it the CA bundle. And when a client connects to the server, the server will send along the bundle with the certificate. This is safe because as long as one of the certificates in the chain is signed by a trusted root certificate, we know that we can trust the whole chain.

Note that the root certificate itself must NOT be part of the chain. It MUST come from the client’s trust store! Unfortunately some CA’s have the bad habit to include their root in the CA bundle, so make sure to remove it before configuring your webserver.

If you check your website against SSL Labs’ SSL Server Test, it will warn you if either intermediates are missing or the root (“anchor”) is included.

Closing Words

Congratulations, you’ve made it! You should now have some basic understanding about TLS/SSL certificates, what they do, and how they work!

Thanks to various browser vendors’ efforts, HTTPS has become the norm when surfing the web in recent years. And now that you’re armed with knowledge, there is no excuse anymore to serve your websites unencrypted!

Manuel Hutter

Manuel Hutter is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Networking Security Part 2: Verifying and Connecting

28. May 2021

Welcome to the second article of a series of three, about the subject of security in networking applications, TLS, certificates, keys, and more.

In the first part, we learnt about TLS/SSL, certificates, CAs and Keys. If you haven’t read it yet, please do so now.

Now that we know all about the pieces of the puzzle, how do they all fit together? Let’s have a closer look!

Issuing Certificates

To begin with, we need to obtain a certificate. The process works something like this:

First, we need a public/private key pair. With it, we can now create a new CSR (short for “Certificate Signing Request”). The CSR contains all the details we want to include in the certificate (the “Subject”) as well as the public key. We then send the CSR to the CA (the “Certificate Authority”).

The CA then verifies the contents of our CSR, and signs it with their own certificate and private key. This results in a new certificate with the details from our CSR, its “Issuer” field set to the “Subject” of the CA’s certificate, and our public key embedded. The CA then sends the new certificate back to us.

And that’s it! We now have a certificate that is signed by the CA, and we have the accompanying private key. We can now install the certificate in our web server.

Establishing Connections

The whole purpose of certificates and keys is to create secure channels through which devices can exchange information, without the risk of snooping by third parties. In other words, security experts call this “being able to send secrets on a postcard”, and all things considered, the analogy is quite appropriate.

So, how do we establish a secure connection with SSL? The following things are required:

  1. A (web) server with:
    • A CA certificate;
    • A server certificate;
    • And a server certificate key (with its corresponding passphrase).
  2. And a client (usually a web browser) with a list of trusted CA certificates.

With all this in place, we can establish a trusted, secure connection between both:

  1. The client opens a TLS connection to the server.
  2. The server responds with its certificate.
  3. The client verifies that the server certificate was indeed signed by a trusted CA.
  4. The client verifies that the server certificate’s CN attribute (part of the “Subject”, remember?) matches the requested URL.
  5. The client verifies that the server certificate is not expired.
  6. The client can optionally check that the server’s certificate has not been revoked, either by using a CRL (a “Certificate Revocation List”), or a mechanism like OCSP (an “Online Certificate Status Protocol”).

After these steps, the client has established an encrypted connection, and verified that the server it is connected to is indeed the one it claims to be.

The client and the server can now exchange data securely with one another, without the risk of a third party being able to read their interactions, or even worse, manipulating them!

Client Certificates

There is however another use case for certificates, and that is Client Authentication. Remember how in the example above the server’s certificate was used to verify its identity? This works the other way around too!

If we have a client certificate that is signed by a CA known to the server, we can establish a mutually trusted, secured connection.

To achieve this, we need the following things:

  1. A server with:
    • A CA certificate;
    • A server certificate;
    • And a server certificate key (with its corresponding passphrase).
  2. A client with:
    • A CA certificate;
    • A client certificate;
    • And a client certificate key (again, with its corresponding passphrase).

Once all the things are in place, a mutually trusted, secured connection can be established:

  1. The client opens a TLS connection to the server, sending its certificate.
  2. The server responds with its own certificate.
  3. The client verifies that the server certificate was indeed signed by a trusted root.
  4. The client verifies that the server certificate’s CN attribute matches the requested URL.
  5. The client verifies that the server certificate is not expired.
  6. The client can optionally ensure the server’s certificate was not revoked.
  7. The server verifies that the client certificate was indeed signed by a trusted root (our CA certificate in this case).
  8. The server verifies that the client certificate is not expired.
  9. The server can optionally ensure the client’s certificate was not revoked.

After these steps, both client and server have established an encrypted connection, one in which both parties are who they claim they are. The server can now read the client’s certificate details (mostly the Subject) to identify the client.

And just like previously, our connection is now fully encrypted and all communications flowing through it are confidential.

Coming Up

In the last part of this series, we are going to talk about some technical details about chains, roots, and intermediates. Stay tuned!

Manuel Hutter

Manuel Hutter is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Self-Service Provisioning with the Crossplane Service Broker

6. May 2021

Tobias Brunner’s lightning talk at Crossplane Community Day Europe 2021.

Watch the recording

The Crossplane Service Broker exposes Crossplane primitives via the Open Service Broker API. Tobias will introduce the concepts behind the Crossplane Service Broker and demonstrate to the audience how it all works together. By leveraging the Open Service Broker API while coupling it with the powerful concept of Crossplane Compositions it’s very easy to enable users of a platform which exposes Open Service Broker API integration (like Kubernetes Service Catalog or Cloudfoundry) to provision services fully automated. In a demonstration a real use case will be shown how a Redis service can be provisioned using the Open Service Broker API, leveraging the Crossplane Compositions and the Helm provider.

Slides: https://speakerdeck.com/tobru/self-service-provisioning-with-the-crossplane-service-broker

Note: This talk was held at the Crossplane Community Day Europe 2021

Also see: Crossplane – The Control-Plane of the future

Markus Speth

Markus is VSHN's CEO and one of the General Managers.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
General Tech

Crossplane – The Control-Plane of the future

29. Apr 2021

VSHN has been a fan of Crossplane since its very early days in 2019. Since then the project has matured a lot and is now used in production by VSHN and many others. In mid 2020 Crossplane became a CNCF Sandbox project and lately applied to be promoted to be a CNCF Incubation project. It’s time for an introduction to Crossplane, why it matters to VSHN and talk about our production usage.

This blog article is also available as a video talk (with corresponding slides).

Use Case: Self-Service Marketplace with Crossplane Service Broker

The very first use case we were able to fulfill by using Crossplane is a project for a VSHN customer who provides a self-service marketplace to their internal customers (developer and service teams). This marketplace is available in their internal Cloudfoundry environment, presented as a one-click service provisioning web-interface. Via this interface, the end-user can order a MariaDB Galera Cluster or a Redis Cluster with different characteristics (e.g. available storage or memory), called Service Plans, with one click. This infrastructure runs in an on-premise datacenter which doesn’t provide any of the well-known hyperscaler services and APIs. We were able to use the Crossplane Helm Provider to deploy services which are specified by Crossplane Composite Resources and Compositions.

In the world of Cloudfoundry the Open Service Broker API is used to provision and manage services. To have a native integration in to the marketplace we developed a Crossplane Service Broker which maps the concepts of the Open Service Broker API specification to the concepts of Crossplane. As they match very well, the integration and translation between these two APIs is very easy.

The Crossplane Service Broker is Open Source and available under https://github.com/vshn/crossplane-service-broker.

This concept lays the foundation for many upcoming new services of VSHN, under the name “Application Catalog”. Watch out this space for more articles about this topic!

What is Crossplane?

In short:

Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume, without having to write any code. 

(Source: crossplane.io)

To achieve this promise, Crossplane brings three main features with it:

  • Providers: These are the pluggable building blocks to provision and manage infrastructure. Each provider enables the use of an upstream or third-party API, for example of a cloud provider, and manages the abstraction to it by bringing Kubernetes custom resources (CRDs) with it. These custom resources are called “managed resources” in the Crossplane world and resemble the upstream API as closely as possible. As each upstream provider has its own opinionated API, Crossplane aligns these interface by providing its own opinionated structure, the Crossplane Resource Model, abbreviated XRM. This allows for example to have a unified API for things like status conditions and references.
    Crossplane itself brings already a lot of providers out-of-the-box and under the crossplane-contrib GitHub organization a lot of third-party providers are being developed.
  • Compositions: This is a Crossplane specific construct which enables the possibility to define new custom APIs – called “composite resources” (XR) – which provide a pre-configured set of managed resources. By predefining a set of managed resources – called “composition” –  the end-user of the platform (e.g. the developer) is being enabled to get infrastructure in an actually usable self-service way. The user doesn’t have to care about the inner details of for example an AWS RDS instance which most of the time needs a lot of configuration and other objects (VPC, networking, firewalling, access control). This work is done by the platform team.
  • Packages: Sharing opinionated infrastructures is done by packaging up all the resources (XRD, Provider, Compositions) in to a package and re-distributing it as a standard OCI image.

The Crossplane developer Nic Cope has written a very good overview in Crossplane vs Cloud Provider Infrastructure Addons and the Crossplane documentation gives another good introduction.

The project overview slides Crossplane – CNCF Project Overview walks the reader through the project (As of March 2021).

Why do we care?

The three core features described above in itself are already very cool, but we feel that there is much more behind it.

As Crossplane leverages the Kubernetes API and concepts, it enables a lot of possibilities:

  • Usage of a well-known API. There is no need to learn a completely new API.
  • This allows to re-use tooling like GitOps to declaratively manage infrastructure.
  • The infrastructure is defined in the same language (Kubernetes API style) as the application deployment is described. No need to learn a new domain-specific language.
  • With that there can be one place which describes everything needed to run an application, including all the infrastructure needs (databaes, caches, indexes, queues, …).
  • As Crossplane is a Kubernetes operator, it has reconciliation built into its heart and therefore all the time actively makes sure that the infrastructure adheres to the defined state. Manual changes to the infrastructure will be rolled-back immediately. No configuration drift is possible this way.
  • Battle-tested Kubernetes RBAC rules help to control access to infrastructure provisioning and management.

Kubernetes is much more than “just” container orchestration. It’s the platform aspect that counts, the well-defined API and the concepts of the control-loop. Crossplane brings this to the next level, making Kubernetes more independent of containers than ever before.

VSHN lately updated its company beliefs in which we set out to bet on Kubernetes as the technical foundation for everything we do in the future. With Crossplane we can now even provision all the infrastructure we need straight out of Kubernetes, no need to use another, disconnected tool anymore.

One API to rule them all:

Core Kubernetes APIs to orchestrate application containers
Crossplane APIs to orchestrate infrastructure

Comparison to other Infrastructure as Code tools

The above description might already shed some light where the differences to other Infrastructure as code tools like Terraform, Pulumi or Ansible are.

Why not Terraform?

When directly comparing Crossplane with Terraform the most important aspect is that Terraform is a command-line (CLI) tool acting on control-planes, where Crossplane is a control-plane itself which is active all the time.

To configure infrastructure with Terraform, the user has to declare the intended architecture in the domain specific HashiCorp Configuration Language (HCL). After that the CLI has to be invoked manually which starts Terraform to plan and actually apply the configuration. After that the current state is stored in a state file to represent the current infrastructure. When something changes in the infrastructure without telling Terraform about it, the stored state differs from the actual state and on the next CLI invocation no-one knows what happens. Also, when only wanting to change something on one of the probably many provisioned services, Terraform always configures all services which could take a long time and affect other services as well, unintended. Automating Terraform is very hard because of these and many other issues. Many more aspects are discussed in Crossplane vs Terraform.

That doesn’t mean Terraform is bad, it’s just a completely different approach to manage infrastructure.

Why not cloud specific Kubernetes operators?

Many, if not all, of the big cloud providers (Hyperscaler) are providing a native Kubernetes operator to manage their cloud of out Kubernetes: Google Cloud has their Config Connector, Azure the Service Operator and AWS the Controllers for Kubernetes. All these operators are specific to the cloud they are engineered for and are providing low-level access to their services. While it’s perfectly fine to use them, Crossplane provides an abstraction layer where the same APIs can be used cross-cloud and presents the platform user the same API, independent on which cloud the cluster is running on and how all the services are named. By leveraging the Composition feature of Crossplane the user doesn’t have to care what all is needed to properly provision a service: For example a production-grade AWS RDS instance has hundreds of configuration values and needs networking, security group, API connection secret, user, schema and grants. This all can be easily abstracted by Crossplane Compositions. An in-depth discussion can be found in Crossplane vs Cloud Provider Infrastructure Addons.

Keeping up with infrastructure providers

Cloud providers are constantly adding and changing services. The Crossplane providers somehow have to keep up with that. As it’s nearly impossible and also impractical to manually keep up with the changes, the Crossplane developers have engineered tools to generate Crossplane providers out of already existing and well maintained SDKs.

Examples:

Crossplane Terminology

To better understand Crossplane, a few Crossplane specific terms will need an explanation:

  • Providers: Extends Crossplane to enable infrastructure resource provisioning. In order to provision a resource, a Custom Resource Definition (CRD) needs to be registered in the Kubernetes cluster and its controller should be watching the Custom Resources those CRDs define.
  • Managed Resources: The Crossplane representation of the cloud provider resources and they are considered primitive low level custom resources that can be used directly to provision external cloud resources for an application or as part of an infrastructure composition.
  • Crossplane Resource Model (XRM): Standardization between all providers, defining for example the status properties and resource references.
  • Composite Resource (XR): A special kind of custom resource that is composed of other resources. Its schema is user-defined.
  • Composite Resource Definition (XRD): Defines a new kind of composite resource, and optionally the claim it offers.
  • Composite Resource Claim (XRC): Declares that an application requires a particular kind of infrastructure, as well as specifying how to configure it.
  • Composition: Specifies how Crossplane should reconcile a composite infrastructure resource – i.e. what infrastructure resources it should compose. It can be used to build a catalog of custom resources and classes of configuration that fit the needs and opinions of an organization.

Video introduction

Tobias Brunner

Tobias Brunner is working since over 20 years in IT and more than 15 years with Internet technology. New technology has to be tried and written about.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
General Tech

Download the “DevOps in Switzerland 2021” Report

9. Apr 2021

We are thrilled to announce the second edition of our “DevOps in Switzerland” report!

From autumn 2020 until early this year we conducted a study to learn how Swiss companies implement and apply DevOps principles.

We compiled the results into a 30-page PDF file (only available in English), and just like in the previous edition, we provided a short summary of our findings in the first pages.

You can download the report directly from our website. Enjoy reading and we look forward to your feedback!

Adrian Kosmaczewski

Adrian Kosmaczewski is in charge of Developer Relations at VSHN. He is a software developer since 1996, a trainer, and a published author. Adrian holds a Master in Information Technology from the University of Liverpool.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Networking Security Part 1: What is TLS?

6. Apr 2021

Welcome to this first article of a series of three, about the subject of security in networking applications, TLS, certificates, keys, and more.

We have noticed that many of our customers struggle to understand how TLS certificates work on a conceptual level. For example, how does “signing” work? What does it even mean? What are a “chain”, an “intermediate” or a “root”? Why do intermediates belong to a chain, but not the root? What is a CSR? How do CSR, Key, and certificate work together? And so on.

This series is an attempt to explain these critical concepts to our customers, starting this first part with the basic vocabulary and knowledge to get started.

HTTPS – TLS – SSL

Let’s talk about what we want to achieve first. When you visit a website via plain old HTTP, an attacker could intercept your request and grab any private information like usernames and password. Additionally, there is no way for you to verify that you indeed connected to the server you intended. An attacker could have modified the DNS response (which is also unencrypted) to send your browser to their server instead. See Man-in-the-middle attack on Wikipedia for more examples.

So in order to verify the identity of the server we connected to, and to make sure nobody except the server and our browser can read the data we exchange, websites these days use TLS (“Transport Layer Security”, or its predecessor SSL “Secure Sockets Layer”) to both authenticate the server and encrypt the traffic.

From a technical perspective, TLS sits between TCP and HTTP on the protocol stack, and while it’s mostly known for being the S (for Secure) part in HTTPS, it’s noteworthy that it can be used for other protocols as well. Some examples:

  • Email protocols: IMAPS, POP3S, SMPTS, …​
  • Database connections: MySQL, PostgreSQL, MongoDB, etcd, …​
  • Data transfer: FTPS
  • Telephony: SIPS
  • Chat: XMPPS, IRCS, …​

The last thing to note here is that there are different versions of TLS (and SSL), and some of them are not considered secure anymore!

From oldest to newest, at the time of writing (March 2021):

  • SSLv1: Insecure, not supported anymore
  • SSLv2: Insecure, not supported anymore
  • SSLv3: Insecure, not supported anymore
  • TLS 1.0: Insecure, deprecated
  • TLS 1.1: Insecure, deprecated
  • TLS 1.2: OK
  • TLS 1.3: OK

Keep this in mind when planning your environment! You wouldn’t want to protect your brand new microservice with outdated security protocols!

By the way, if you want to check which TLS versions a website supports, use SSL Labs’ SSL Server Test. It’s a great debugging tool and will show you a lot of information about the topics of this Blog post series!

Certificates

The next thing we should have a look at are Certificates, or more specifically X.509 v3 public key certificates.

A Certificate is a cryptographically signed piece of information, of which the most important part is the Subject, which identifies who this certificate belongs to, as well as the Issuer. Other attributes include details about when the certificate is about to expire as well as a lot of technical information about the key and signature algorithms used, and so on.

The Subject and Issuer of a certificate are characterized by a set of attributes:

  • CN – Common Name
  • C – Country
  • ST – State
  • L – Location (City)
  • O – Organisation
  • OU – Organisational Unit

Together, those attributes form a Distinguished Name (DN). Most attributes are optional, except for the Common Name. In the case of Server Certificates, the CN must match the address used to connect, for example www.vshn.ch.

The Certificate also contains a Public Key (embedded in the certificate) and its matching Private Key.

The last important feature of X.509 certificates is that they are signed by other certificates (or itself). Once a certificate is signed, its contents cannot be changed anymore.

Signing CAs

The Internet is a big place, so how do we know whom to trust? To solve this issue, the concept of Certificate Authorities came up.

A Certificate Authority (commonly referred to as CA) is a central trusted source of certificates. They have a CA Certificate (also known as the “root” certificate) that is well known and trusted. This certificate is only used to sign other certificates. By signing another certificate, the CA confirms “yes, I trust this certificate and the information in it is correct.”

A related concept is a Certificate revocation list, also known as “CRL”, which is a list of digital certificates that have been revoked by the CA before their scheduled expiration date, and should therefore no longer be trusted.

Now that we have explained what a certificate is, let’s talk about keys.

Keys

Each certificate has its own key. The key makes certificates actually useful; without them, they would not work.

Each key consists of two parts: a public key and a private key. While the public key is embedded as part of the certificate itself, the private key is stored in a different file altogether.

In order to sign another certificate, you need the Private key of the signing certificate (but not of the certificate you want to sign).

When you open a connection to a server, the server also needs the private key of the server certificate in order to authenticate itself and establish encrypted communication.

This whole concept is called Public-key cryptography.

Confidentiality

A very important thing to understand is that the key is the only confidential piece in the puzzle; needless to say, it is the most important piece! So please, keep this always in mind: do not ever exchange keys over unsecured channels, under any circumstances!

On the other hand, the certificate itself is not confidential. The CA certificate in particular must be provided to all clients and servers, in order to verify other certificates.

How can a client ask a CA to sign a certificate? Through a CSR (also known as Certificate Signing Request) which, as the name implies, performs exactly this task.

Coming Up

Speaking about verification and issuing new certificates, we are going to talk about these subjects in detail, respectively in the second and third parts of this series. Stay tuned!

Manuel Hutter

Manuel Hutter is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Press Project Syn Tech

K8up Version 1.0 Released

16. Mar 2021

We are thrilled to announce the general availability of K8up version 1.0!

New K8up Logo
New K8up Logo

K8up (pronounced /keɪtæpp/ or simply “ketchup”) is a Kubernetes Operator distributed via a Helm chart, compatible with OpenShift and plain Kubernetes. It allows cluster operators to backup PVCs; to perform on-demand backups; or to schedule regular backups. K8up is written in Go and is an Open Source project hosted at GitHub.

This new version is a full rewrite of the operator, based on the Operator SDK. This provided us with a more stable code base, with extended testing, paving the way for future improvements and new features.

Speaking about which; some of the new features in version 1.0 are:

  • Support for Kubernetes 1.20.
  • K8up status printed when using kubectl get or oc get.
  • Run multiple replicas of K8up for high availability.
  • Specify Pod resources (CPU, memory, etc.) from different levels.
  • New random schedules (e.g. @daily-random) to help distribute the load on the cluster.
  • New end-to-end and integration tests.
  • Docker image mirroring in Docker Hub and Quay.
  • More and better documentation, including a new logo!

Current K8up users: please check the upgrade guide with all the information you need to start using the latest and greatest version of K8up.

Would you like to know more or to contribute to the project? Check the K8up GitHub project and backup your clusters!

Christian Сremer

Christian Cremer is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Diagrams with Draw.io

6. Jan 2021

I think it can be said that at VSHN we have a certain obsession with documentation. This goes beyond the requirements set by our customers, or by those imposed by certifications such as the ISO 27001 or ISAE-3402. We have found that as a team we can work better when we can refer to written documentation at all times.

In the past few years we have standardized our documentation toolkit: we have adopted Asciidoctor as the lingua franca for our documentation. We have put in production various documentation websites based on Antora: our Handbook, our Knowledge Base, and the Project Syn documentation are some examples.

That works great for text. But what about diagrams? Our documentation has lots of them, and although embedding PNG files works fine, it does not “scale” very well.

For that reason we have set up an instance of Kroki in production (in APPUiO, of course!), to enable the generation of SVG diagrams based on various formats: PlantUML for UML diagrams, Nwdiag for network diagrams, and Ditaa for simple box-and-arrow kind of diagrams, all produced and shown in a crisp and neat SVG format. We love Kroki! It is such an awesome idea, that it has even been integrated to GitLab as a supported diagramming tool. How cool is that? In general, the Asciidoctor ecosystem has very good integration of diagrams into workflows, and this extends to Antora as well.

I am not going to dive into the benefits of vector vs. raster images; I think the readers of this article are well aware of the bonuses the former has over the latter. SVG diagrams look great in both online and print, but they can be a bit complicated to draw by hand (duh!).

But here’s a confession: I’m not a big fan of editing text-based diagrams either. But that’s just a personal choice. And the truth is, many other non-technical VSHNeers find the whole “diagrams as text” thing a bit arcane.

So that means that we needed a more accessible tool for them. The ideal diagram drawing tool for VSHN must have the following requirements:

  • Cross-platform: Most of us at VSHN use Linux, but a sizable number also use Macs and Windows laptops.
  • Accessible to non-technical VSHNeers: Not everyone at VSHN is a DevOps engineer.
  • Simple to use: If possible, using a point-and-click interface; for example, in our Confluence wiki we have enabled the Gliffy extension, which allows non-technical users to create stunning diagrams.

So we started our search. Leaving aside tools like Inkscape, that are too generic, like Asciiflow, that are too limited, or like Dia, that are too platform-specific, it took us a while to find a tool that would fit our requirements. But we found it!

Please welcome Diagrams.net (previously also known as Draw.io). It is a web application geared specifically to the generation of diagrams, very easy to use and ticking all the boxes that we needed.

And even better, there is a Visual Studio Code extension that allows to edit diagrams locally, directly on your laptop, in Linux, Mac and Windows.

How do we integrate this with Antora? Very simple. In the assets/images folder of our documentation components, create your diagrams using the *.drawio.svg extension. These are automatically associated by Visual Studio Code with the extension, and provides a live editor with all the bells and whistles you might expect.

And then, well, just image::diagram.drawio.svg[] in your documents, git commit and push.

This approach is definitely more accessible to non-technical VSHNeers, making it super easy to edit diagrams online, or locally in Visual Studio Code.

To finish, here’s a Pro Tip™®©: switch the Draw.io extension to the “Atlas” theme, since some lines in the extension lack contrast, and aren’t easily seen when using a dark theme. You can change the Draw.io extension theme with the selector at the bottom right of the screen.

PS: there is an open issue in Kroki to add support for Draw.io diagrams. Can’t wait to see it in action!

Christian Сremer

Christian Cremer is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

About Our ISO 27001 Certification And ISAE 3402 Type 1 Report

11. Nov 2020
One of our core value propositions consists in making sure that our infrastructure responds to the highest levels of security, confidentiality, and availability.
It is with this goal in mind that we passed our first ISO 27001 certification in 2017, and this year we added an ISAE 3402 audit, initially requested by one of our customers in the financial sector.

ISO 27001

ISO 27001 is a worldwide applied standard for the certification of an information security management system (ISMS). Its aim is to protect information based on an analysis of business risks regarding confidentiality, integrity and availability.
The certification is valid for three years. Every year there is a monitoring audit done by the certification authority. This year the three year period for the first certificate ended. In spite of the ongoing worldwide pandemic we were able to renew this certification earlier this year with excellent results.
The audit required for the renewal not only didn’t raise any concerns, but rather praised positively our KPIs for ISMS, our thorough integration of information security into every single step of our operations, and the overall commitment of management to the whole process.
Our certification is valid for the next three years, and you can download it for review.

ISAE 3402

Service providers are recommended to use a recognized auditing company on their own behalf, which annually reports in a standardized form on the functioning of their internal controls.
This is done in order not to have to repeatedly expose its confidential internal processes, procedures and methods.
The most important international standard for companies in the financial sector in Switzerland is ISAE 3402.
The International Standard on Assurance Engagements (ISAE) 3402 is an internationally accepted audit standard issued and maintained by the International Auditing and Assurance Standards Board (IAASB). The ISAE 3402 is divided into two categories: ISAE 3402 Type I only assesses the suitability or design of controls, i.e. the control design and implementation. While the ISAE 3402 Type II additionally also evaluates the effectiveness of the controls during the test period, i.e. their definition and concrete implementation.
The ISAE 3402 Assurance Report examines the definition and implementation of control objectives as well as the existence and effectiveness of controls. The basic prerequisite for a successful audit is a complete and up-to-date documentation of the company organization, the IT organization and ICS-related issues. This includes an assessment of the existence and effectiveness of the internal control system, including the IT organization and IT security.
ISAE 3402 reports provide substantial added value to FINMA regulated firms, which must ensure conformity to circulars regarding outsourcing, such as Rundschreiben 2008/21 and Rundschreiben 2018/3.
On June 2020 our ISAE 3402 Report Type 1 got audited by ERM Solution AG. We are currently planning the establishment of our Type 2 report by January next year. This report ensures and supports the legal audits of our financial customers.
If your company requires a yearly ISAE 3402 report for audit or revision, please contact our sales and marketing team.

More Information

If you would like to know more about the differences between ISO 27001 and ISAE 3402, please check this link.
We remain at your service for any enquiry. Contact us if you need more information.

Daniel Hauswirth

Daniel Hauswirth is CISO and DevOps Engineer in VSHN. He holds a BSc Hochschule Luzern/FHZ in Computer Science, and is working towards his MAS in Information and Cyber Security at HSLU.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Assessing the Impact of New Docker Pull Rate Limits in APPUiO

3. Nov 2020

Docker Inc., the company behind Docker Hub, has recently announced an enforcement of image pull rate limits for Docker Hub users. This change affects all cloud native installations currently configured to consume container images stored in Docker Hub, one of the most popular image repositories available today. This measure has a direct impact in the deployment of many customer applications currently running on APPUiO Public and Private Clusters.

We are following the situation closely. Following recent announcements, Docker Hub is slowly starting to enforce a pull rate limit of 100 pulls per 6 hours for anonymous (unauthenticated) IP-Addresses, and 200 pulls per 6 hours for authenticated non-paying users. This means that a Kubernetes or OpenShift cluster can only pull 100 images in 6 hours without authentication for all of its users from Docker Hub. During maintenance periods, most of the images of a cluster are pulled at some point, and the reduction of pull rate limits can cause downtime when the images can’t be pulled due to the limit.

This situation should not affect today’s maintenance window. Even though the announced start date of the rate limit was Monday, October 2nd, 2020 the analysis of responses from Docker Hub indicates that the new limits are not yet applied, and will only be enforced during a 3-hour window from 18:00 to 21:00 (CET) tomorrow. Unfortunately at this point we don’t know when the new pull rate limits will be enforced fully.

Also noteworthy, these restrictions do not apply to the internal OpenShift registry included with APPUiO Public or Private, which is completely independent of Docker Hub, nor to Docker users with Pro or Team accounts. Authenticated users with Pro or Team accounts enjoy unlimited data transfer to and from Docker Hub.

VSHN is currently evaluating measures to prevent downtime, and reduce the impact of this situation for our customers. The most appropriate solution at this moment consists in switching to the aforementioned Pro or Team Docker account types. Even simpler, the use of an authenticated user account in a pull secret instead of an anonymous account will double the pull rate limit, and will reduce the risk of downtime considerably. Another possibility consists in migrating images to a different public registry, like Red Hat Quay. Another option is the use of a private registry, such as the OpenShift integrated registry, AWS ECR, GitLab Container Registry, Harbor, or other similar technologies. In particular, AWS has recently announced the future availability (“within weeks”) of a free public registry.

We remain at your service for any enquiry. Contact us if you need more information.

Gabriel Mainberger

Gabriel Mainberger is a DevOps engineer in VSHN.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
General Tech

How to migrate OpenShift 3 to OpenShift 4

9. Oct 2020

Migrating OpenShift Container Platform (OCP) 3 to 4: learn about possible migration paths and why you should make the move to the new version of OpenShift.

Recap: what is OpenShift?

Red Hat OpenShift is a hybrid cloud, enterprise Kubernetes application platform for developers and business applications.
OpenShift provides a secure and stable platform for container-based deployments without the need for big IT investments. This means that much needed legacy applications can be used alongside new cloud native and container-based ones.
Red Hat OpenShift is a Kubernetes distribution. An extensive article with more details about Kubernetes and how Kubernetes compares to OpenShift can be found here.
OpenShift consists of platform, application, and developer services. It is used to manage workloads, build cloud-native apps, and to boost the productivity of developers.

(image: © Red Hat https://www.openshift.com/learn/what-is-openshift)

What’s new in OpenShift 4?

OpenShift 4 was introduced at the Red Hat Summit in May 2019. In this major release, Red Hat redefines Kubernetes for the enterprise through full stack automation. OpenShift 4 includes new technologies and functionality that results in self-managing, flexible, and automated clusters.
New key elements of OpenShift 4 include:

  • Immutable Red Hat Enterprise Linux CoreOS;
  • Operator framework;
  • OpenShift service mesh; and
  • Knative framework.

OpenShift 4 is Kubernetes at its core and in this release, Red Hat has completely re-architected how OpenShift is installed, upgraded and managed, introducing innovations such as Kubernetes Operators. Operators automate life cycle management of containerized applications with Kubernetes, and drives installation and upgrades of OpenShift and all of its services. This includes Kubernetes core services, along with Prometheus, Grafana, Elasticsearch, software-configured networking, storage, registry and other components of the OpenShift Kubernetes platform. OpenShift 4 is an Operator-driven platform that delivers full-stack automation from top to bottom.
OpenShift 4 also offers multiple cluster management across multiple clouds, and enables hybrid cloud services with Operators & OperatorHub. OpenShift 4 is designed to deliver a unified experience across hybrid cloud by driving automated installation and updates across Kubernetes deployments everywhere, all powered by Kubernetes Operators. OpenShift 4 is still built on top of Red Hat Enterprise Linux, and delivered in a new immutable form as Red Hat Enterprise Linux CoreOS.
OpenShift 4 also brings a number of new developer services and capabilities needed to build cloud-native applications, to deploy them consistently across any supported on-premises, private, or public cloud infrastructure.
Red Hat OpenShift 4 is Kubernetes for the Enterprise, designed to power businesses’ digital transformation, and to unite development teams on a single platform.

What are the advantages of migrating to OpenShift 4

OpenShift 3 will only be supported until 2021, so besides the need to migrate to the new version, we also think that OpenShift 4 brings many advantages for both operators and users.
The level of maturity of operators leaves little to be desired. For the operation of a cluster, updates are much faster and at the same time, easier to handle and more stable. This also simplifies operations, as operators can react to changes and correct errors if needed. Operators are a new concept also for the operation of applications for users of OpenShift.
The switch from Docker to Buildah for container builds now allows to build container images in a more secure way. This is a welcome innovation for all multitenant clusters, such as public platform providers and corporate clusters with multiple users. With the introduction of the OpenShift Service Mesh, developers in particular will gain new insights and new possibilities to visualize, secure and optimize the communication of their applications.
OpenShift 4 represents a significant change in the way that OpenShift Container Platform clusters are deployed, managed, and developed upon. OpenShift 4 includes new technologies and functionality, for both developers and cluster administrators. Operators, Serverless, Service Mesh, and Red Hat Enterprise Linux CoreOS (RHCOS), are all new to OpenShift 4. They are redefining what’s possible with the next generation of Kubernetes platforms. This technology shift enables clusters to self-manage functions previously performed by administrators, and empower developers to innovate on a consistent and stable platform that is simple to install and scale.
The advantages of OpenShift 4 include:

  • Operators: implement and automate common Day-1 (installation, configuration, etc) and Day-2 (re-configuration, update, backup, failover, restore, etc.) activities in a piece of software running inside your OpenShift cluster, by integrating natively with Kubernetes concepts and APIs.
  • Red Hat Serverless: enables an application to consume only the compute resources necessary, scaling up or down automatically depending on usage. This removes the overhead of server provisioning and maintenance from the developer, letting them focus on application development instead.
  • Red Hat Service Mesh: controls how different parts of an application share data with one another through a built-in dedicated infrastructure layer. This visible infrastructure layer can track different parts of an app and their interactions, to optimize communications and to avoid downtime as it grows.
  • Red Hat Enterprise Linux CoreOS: combines the ease of over-the-air updates from Container Linux with the Red Hat Enterprise Linux kernel, to deliver a more secure, easily managed container host.

OpenShift’s goal is to help developers to innovate more rapidly, to address the needs of the business quicker. Cloud native application development brings new challenges. As developers adopt microservice architectures, managing the communication between each service, securing those services and getting better service to service traceability to debug issues is an absolute necessity. These are the challenges that the Istio open source project seeks to address.
The OpenShift 4 Service Mesh takes Istio and combines it with other key projects, like Jaeger for tracing and Kiali for visualization, to provide better manageability and traceability to microservices deployments. Developers can focus on building the business logic, letting the service mesh manage how each microservice communicates based on policies they define. They can also leverage the tracing and visualization capabilities to debug issues when they occur.
Development approaches haven’t stopped evolving, and serverless is yet another way developers are looking to build applications, by leveraging function as a service based offerings. The Serverless model enables scaling down to zero, so as to only consume compute resources when functions execute. This can be an effective way to control operational costs, particularly in the public cloud. FaaS offerings were first pioneered by public cloud providers like AWS, but have the potential to lock your applications into a single cloud environment. This is why Red Hat is working to bring these capabilities to a hybrid cloud environment via Knative.
Red Hat is an active member of the Knative open source project, collaborating with the Kubernetes community to drive upstream development that enables hybrid serverless capabilities. Using the Knative framework enabled in OpenShift, users can extend Kubernetes to build, deploy and manage serverless applications, supporting containerized and serverless deployments from a single Kubernetes control plane.

Who is OpenShift 4 for?

Almost a year after the initial release of OpenShift 4, we believe that OpenShift Container Platform 4 is ready for productive workloads.
If you are currently running OpenShift 3, you have to evaluate and plan migrating to OpenShift 4.
What are your options?

  • Do you want to run and operate OpenShift yourself?
    • This way you “keep full control” and you decide on every detail with all the implications
    • Do you have enough staff with the needed know-how to run OpenShift operations yourself 24/7? This might mean that you have to re-train your staff and also re-engineer all operations tools.
      • You might need to hire new people or train your existing staff.
      • External consultants might be needed for specific tasks such as the setup or migration part.
  • Or do you want to work with a partner who takes care of your OpenShift installation and ensure 24/7 operations?
    • Work with a hyperscaler like AWS/Azure/Google Cloud etc.:
      • You are super flexible, you can spin up your own cluster and the additional tools you need and you can do everything yourself “at a fingertip”
      • But you are also bound to one platform and you have to ensure the actual 24/7 management and operations
    • Work with a specialized partner like VSHN / APPUiO:
      • You can be sure that you have the latest OpenShift know-how and enough people who take care of your operations, both on premises or in any cloud of your choice.
      • A certified, Swiss based Red Hat partner who knows how to run OpenShift even in the most sensitive areas and industries such as banking & finance with a focus on the business-relevant application, not the “generic platform”
      • You get Managed OpenShift, 24/7 operations, additional Managed Services for a recurring monthly fee, at the end you save time & money so that your people can focus on developing your product and / or services.

At the end, it’s not a question of if, but when to migrate to OpenShift 4.

How to migrate from OpenShift 3 to OpenShift 4

There is no planned update path from OpenShift 3 to 4. Red Hat provides migration tools, however, that can migrate not only the Kubernetes resources, but also the data from persistent volumes, where S3 Storage is used as a cache. The migration tool supports migrations from version 3 to 4 as well as migrations between different OpenShift 4 clusters.
According to the website openshift.com, the migration process from OpenShift 3 to 4 can be completed in 5 steps:

  1. Spin up a new cluster running OpenShift 4.
  2. Configure the new OpenShift 4 cluster.
  3. Create a migration plan which includes how to handle data from the applications that are being migrated.
  4. Run migration plan.
  5. Move your DNS or Load-balancer configuration to your new cluster.

To successfully transition from OpenShift Container Platform 3 to OpenShift Container Platform 4, it is important that you review the following information:

  • Planning your transition: Learn about the differences between OpenShift Container Platform versions 3 and 4. Prior to transitioning, be sure that you have reviewed and prepared for storage, networking, logging, security, and monitoring considerations.
  • Performing your migration: Learn about and use the tools to perform your migration:
    • Control Plane Migration Assistance Tool (CPMA): the Control Plane Migration Assistance tool helps you configure the new cluster OpenShift 4 cluster that will be the destination for the applications that are being migrated from your existing OpenShift 3 cluster. This tool reduces the possibility for human error in the configuration process, matching when possible the existing settings in the source cluster. It also allows you to review the resources that will be applied to the new cluster.
    • Cluster Application Migration Tool (CAM): the Cluster Application Migration tool (CAM) migrates stateful and stateless applications from the source cluster on OpenShift 3 to a destination cluster running the latest OpenShift. It also can migrate applications between OpenShift 4 clusters. It is installed on your destination cluster via an Operator. Through the rich user interface take advantage of the simplified, customizable workflows. Decide which applications to start with and which migration approach best fits each application and your needs.

If you need detailed instructions about your OpenShift migration, head over to this page to learn more about the migration process and how to utilize the migration tooling. And of course, if you need help with your migration, we will be very happy to help you.

VSHN and APPUiO

VSHN is a Red Hat Advanced CCSP partner and we offer Managed OpenShift since 2016 through our brand APPUiO, both in the cloud of your choice or on premises. In 2020, we are operating OpenShift clusters for 350 customers and partners in 16 clouds around the globe.
If you want to learn how we helped acrevis Bank in Switzerland with the move and how we operate OpenShift for them, check out our Success Story.

APPUiO OpenShift 4 Fact Sheet

Our APPUiO OpenShift Fact Sheet explains the benefits of OpenShift 4 in an easy to read 2-pager.

Download the Fact Sheet below.

OpenShift 4 Tech Labs

Join our free OpenShift 4 Tech Labs.
Need help migrating and operating OpenShift 4? Contact us, we are here to help.

Markus Speth

Markus is VSHN's CEO and one of the General Managers.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Project Syn Tech

Second Beta Release of Project Syn Tools

23. Jul 2020

Without further ado, we’re announcing the release 0.2 of the Project Syn tools.
Since the first public release mid-March this year (read more about it in First Pre-Release of Project Syn Tools) we used the tools on a daily basis, in particular for the development of our new product “VSHN Syn Support”. And of course we have incorporated all of that experience in the source code. The main features are now in place, and are getting better and better on a daily basis.

New Features and Improvements

When reading the announcement of a new version, engineers are always interested in new features and improvements. So these are the most important new additions since 0.1:

  • Everything required for setting up a new cluster (GitOps repository, cluster config file in Tenant configuration, Vault secrets, and more) is now fully automated. One API call to register a new cluster and you’re done.
  • In parallel to the creation of clusters, we have also automated all steps required to decommission them (Repo deletion, Vault secret cleanup, and more). Just delete it and everything is gone (of course, there are preventive measures in place to not make this an Uh-oh moment).
  • Commodore got a lot of improvements: for local development, and for developing new components with a comprehensive cookiecutter template.

Document All The Things

Besides implementing new features and fixing bugs we put a lot of effort into the documentation. The main documentation page https://syn.tools/ got a completely new structure and a huge amount of new content. We’re in the process of adding new pages frequently, so make sure to check it out every so often.
Before 0.2 it was hard to get started with Project Syn and to understand what it was all about. To solve that issue we wrote the following introductions:

Our next goal is to document the concepts behind configuration management with Commodore in detail.

Commodore Components on GitHub

An important building block of Project Syn are Commodore Components. Over the past months we’ve written and open sourced more than 10 Commodore Components on GitHub. They offer the flexibility to install and configure Kubernetes system services, adapted to their respective distribution and infrastructure.
These Commodore Components can be found by searching for the “commodore-component” topic on GitHub.
We are writing and refining more and more Components every day. We are going to publish some guidelines about how to write Commodore Components (one specifically for OpenShift 4 Components is already available) and eventually enforce them via CI jobs and policies.
An upcoming Component Writing Tutorial will help beginners to start writing own Components or contribute to existing ones.

The Road to 1.0 and Contributions

What we learnt while working on Project Syn over the last few months gave us a very clear picture of what we want to achieve in version 1.0. The roadmap contains the most important topics:

  • Documentation! We have to and will put a lot of effort into documentation, be it tutorials, how-to guides, or explanations.
  • Full Commodore automation to automate and decentralize the cluster catalog compilation process.
  • Developer experience improvements for simplifying the development of Commodore Components even further.
  • Engineering of a new tool helping users to launch managed services on any Kubernetes cluster.
  • Cluster provisioning automation integration, to leverage third party tools for automatically bootstrapping Kubernetes clusters.

This is not all; check the more detailed roadmap on the Project Syn page for more. The GitHub project will grow with issues over the next few weeks.
If you think that this sounds interesting and you would like to contribute, we now have an initial Contribution Guide available and are very open to suggestions and pull requests. Just get in contact with us if you’re interested.

Our Product: VSHN Syn Support

Besides the Open Source project we were also working on defining what added value you can get from VSHN. We call this product “VSHN Syn Support.” If you’re interested in getting commercial support from VSHN for Project Syn on a Managed Kubernetes Cluster based on OpenShift 4 or Rancher, get in touch with us. More information about VSHN Syn Support can be found here.

Tobias Brunner

Tobias Brunner is working since over 20 years in IT and more than 15 years with Internet technology. New technology has to be tried and written about.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

How we manage 100+ Web Application Firewalls

26. Jun 2020

This is about how we manage large numbers of ModSecurity WAF deployments, including CRS and custom rules.

ModSecurity CRS Berne Meetup

At VSHN we’re not only 100% committed to Free and Open Source software, we also collaborate actively with the communities of the software we use, deploy and enhance. This June, we had the pleasure to join the OWASP ModSecurity Core Rule Set Berne Meetup group online who had asked us back in November at yet another Meetup to showcase the way we use ModSecurity and the Core Rule Set (CRS).

(more…)

peter.bittner

Continuous delivery for a living.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Project Syn Tech

Tutorial: Backing up Kubernetes Clusters with K8up

23. Jun 2020

One of the most common questions we got from companies moving to Kubernetes has always had to do with backups: how can we ensure that the information in our pods and services can be quickly and safely restored in case of problems?
This situation is so common that we VSHN decided to tackle it with our own Kubernetes operator for backups, which we called K8up.
Note: This tutorial is available in three versions, each in its own branch of the GitHub repository bundled with this text:

1. What is K8up?

K8up (pronounced “/keɪtæpp/” or simply “ketchup”) is a Kubernetes operator distributed via a Helm chart, compatible with OpenShift and plain Kubernetes. It allows cluster operators to:

  • Backup all PVCs marked as ReadWriteMany or with a specific annotation.
  • Perform individual, on-demand backups.
  • Schedule backups to be executed on a regular basis.
  • Schedule archivals (for example to AWS Glacier), usually executed in longer intervals.
  • Perform “Application Aware” backups, containing the output of any tool capable of writing to stdout.
  • Check the backup repository for its integrity.
  • Prune old backups from a repository.
  • Based on top of Restic, it can save backups in Amazon S3 buckets, and Minio (used we’ll see in this tutorial.)

K8up is written in Go and is an open source project hosted in GitHub.

2. Introduction

This tutorial will show you how to backup a small Minikube cluster running on your laptop. We are going to deploy MinioMariaDB and WordPress on this cluster, and create a blog post in our new website. Later we’re going to “deface” it, so that we can safely restore it later. Through this process, you are going to learn more about K8up and its capabilities.
Note: All the scripts and YAML files are available in GitHub: github.com/vshn/k8up-tutorial.

2.1 Requirements

This tutorial has been tested in both Linux (Ubuntu 18.04) and macOS (10.15 Catalina.) Please install the following software packages before starting:

  • Make sure PyYAML 5.1 or later is installed: pip install PyYAML==5.1
  • The kubectl command.
  • The Restic backup application.
  • The latest version of Minikube (1.9 at the time of this writing.)
  • Helm, required to install K8up in your cluster.
  • k9s to display the contents of our clusters on the terminal.
  • jq, a lightweight and flexible command-line JSON processor.

3. Tutorial

It consists of six steps to be executed in sequence:

  1. Setting up the cluster.
  2. Creating a blog.
  3. Backing up the blog.
  4. Restoring the contents of the backup.
  5. Scheduling regular backups.
  6. Cleaning up.

Let’s get started!

3.1 Setting up the cluster

Note: The operations of this step can be executed at once using the scripts/1_setup.sh script.

  1. Start your minikube instance with a configuration slightly more powerful than the default one:
    • minikube start --memory 4096 --disk-size 60g --cpus 4
      Note: On some laptops, running Minikube on battery power severely undermines its performance, and pods can take really long to start. Make sure to be plugged in to power before starting this tutorial.
  2. Copy all required secrets and passwords into the cluster:
    • kubectl apply -k secrets
  3. Install and run Minio in your cluster:
    • kubectl apply -k minio
  4. Install MariaDB in your cluster:
    • kubectl apply -k mariadb
  5. Install WordPress:
    • kubectl apply -k wordpress
  6. Install K8up in Minikube:
    • helm repo add appuio charts.appuio.ch
    • helm repo update
    • helm install appuio/k8up --generate-name --set k8up.backupImage.tag=v0.1.8-root

After finishing all these steps, check that everything is running; the easiest way is to launch k9s and leave it running in its own terminal window, and of course you can use the usual kubectl get pods.
Tip: In k9s you can easily delete a pod by going to the “Pods” view (type :, write pods at the prompt and hit Enter), selecting the pod to delete with the arrow keys, and hitting the CTRL+D key shortcut.

The asciinema movie below shows all of these steps in real time.

 

3.2 Viewing Minio and WordPress on a browser

Note: The operations of this step can be executed at once using the scripts/2_browser.sh script.

  1. Open WordPress in your default browser using the minikube service wordpress command. You should see the WordPress installation wizard appearing on your browser window.
  2. Open Minio in your default browser with the minikube service minio command.
    • You can login into minio with these credentials: access key minio, secret key minio123.

3.2.1 Setting up the new blog

Follow these instructions in the WordPress installation wizard to create your blog:

  1. Select your language from the list and click the Continue button.
  2. Fill the form to create new blog.
  3. Create a user admin.
  4. Copy the random password shown, or use your own password.
  5. Click the Install WordPress button.
  6. Log in to the WordPress console using the user and password.
    • Create one or many new blog posts, for example using pictures from Unsplash.
  7. Enter some text or generate some random text using a Lorem ipsum generator.
  8. Click on the “Document” tab.
  9. Add the image as “Featured image”.
  10. Click “Publish” and see the new blog post on the site.

3.3 Backing up the blog

Note: The operations of this step can be executed at once using the scripts/3_backup.sh script.
To trigger a backup, use the command kubectl apply -f k8up/backup.yaml. You can see the job in the “Jobs” section of k9s.
Running the logs command on a backup pod brings the following information:

$ kubectl logs backupjob-1564752600-6rcb4
No repository available, initialising...
created restic repository edaea22006 at s3:http://minio:9000/backups
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
Removing locks...
created new cache in /root/.cache/restic
successfully removed locks
Listing all pods with annotation appuio.ch/backupcommand in namespace default
Adding default/mariadb-9588f5d7d-xmbc7 to backuplist
Listing snapshots
snapshots command:
0 Snapshots
backing up via mariadb stdin...
Backup command: /bin/bash, -c, mysqldump -uroot -p"${MARIADB_ROOT_PASSWORD}" --all-databases
done: 0.00%
backup finished! new files: 1 changed files: 0 bytes added: 4184711
Listing snapshots
snapshots command:
1 Snapshots
sending webhook Listing snapshots
snapshots command:
1 Snapshots
backing up...
Starting backup for folder wordpress-pvc
done: 0.00%
backup finished! new files: 1932 changed files: 0 bytes added: 44716176
Listing snapshots
snapshots command:
2 Snapshots
sending webhook Listing snapshots
snapshots command:
2 Snapshots
Removing locks...
successfully removed locks
Listing snapshots
snapshots command:
2 Snapshots

If you look at the Minio browser window, there should be now a set of folders that appeared out of nowhere. That’s your backup in Restic format!

3.3.1 How does K8up work?

K8up runs Restic in the background to perform its job. It will automatically backup the following:

  1. All PVCs in the cluster with the ReadWriteMany attribute.
  2. All PVCs in the cluster with the k8up.syn.tools/backup: "true" annotation.

The PVC definition below shows how to add the required annotation for K8up to do its job.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wordpress-pvc
  labels:
    app: wordpress
  annotations:
    k8up.syn.tools/backup: "true"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Just like any other Kubernetes object, K8up uses YAML files to describe every single action: backups, restores, archival, etc. The most important part of the YAML files used by K8up is the backend object:

backend:
  repoPasswordSecretRef:
    name: backup-repo
    key: password
  s3:
    endpoint: http://minio:9000
    bucket: backups
    accessKeyIDSecretRef:
      name: minio-credentials
      key: username
    secretAccessKeySecretRef:
      name: minio-credentials
      key: password

This object specifies two major keys:

  • repoPasswordSecretRef contains the reference to the secret that contains the Restic password. This is used to open, read and write to the backup repository.
  • s3 specifies the location and credentials of the storage where the Restic backup is located. The only valid option at this moment is an AWS S3 compatible location, such as a Minio server in our case.

3.4 Restoring a backup

Note: The operations of this step can be executed at once using the scripts/4_restore.sh script.
Let’s pretend now that an attacker has gained access to your blog: we will remove all blog posts and images from the WordPress installation and empty the trash.

Oh noes! But don’t worry: thanks to K8up you can bring your old blog back in a few minutes.
There are many ways to restore Restic backups, for example locally (useful for debugging or inspection) and remotely (on PVCs or S3 buckets, for example.)

3.4.1 Restoring locally

To restore using Restic, set these variables (in a Unix-based system; for Windows, the commands are different):

export KUBECONFIG=""
export RESTIC_REPOSITORY=s3:$(minikube service minio --url)/backups/
export RESTIC_PASSWORD=p@ssw0rd
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123

Note: You can create these variables simply running source scripts/environment.sh.
With these variables in your environment, run the command restic snapshots to see the list of backups, and restic restore XXXXX --target ~/restore to trigger a restore, where XXXXX is one of the IDs appearing in the results of the snapshots command.

3.4.2 Restoring the WordPress PVC

K8up is able to restore data directly on specified PVCs. This requires some manual steps.

  • Using the steps in the previous section, “Restore Locally,” check the ID of the snapshot you would like to restore:
$ source scripts/environment.sh
$ restic snapshots
$ restic snapshots XXXXXXXX --json | jq -r '.[0].id'
  • Use that long ID in your restore YAML file k8up/restore/wordpress.yaml:
    • Make sure the restoreMethod:folder:claimName: value corresponds to the Paths value of the snapshot you want to restore.
    • Replace the snapshot key with the long ID you just found:
apiVersion: backup.appuio.ch/v1alpha1
kind: Restore
metadata:
  name: restore-wordpress
spec:
  snapshot: 00e168245753439689922c6dff985b117b00ca0e859cc69cc062ac48bf8df8a3
  restoreMethod:
    folder:
      claimName: wordpress-pvc
  backend:
  • Apply the changes:
    • kubectl apply -f k8up/restore/wordpress.yaml
    • Use the kubectl get pods commands to see when your restore job is done.

Tip: If you use the kubectl get pods --sort-by=.metadata.creationTimestamp command to order the pods in descending age order; at the bottom of the list you will see the restore job pod.

3.4.3 Restoring the MariaDB pod

In the case of the MariaDB pod, we have used a backupcommand annotation. This means that we have to “pipe” the contents of the backup into the mysql command of the pod, so that the information can be restored.
Follow these steps to restore the database:

  1. Retrieve the ID of the MariaDB snapshot:
    • restic snapshots --json --last --path /default-mariadb | jq -r '.[0].id'
  2. Save the contents of the backup locally:
    • restic dump SNAPSHOT_ID /default-mariadb > backup.sql
  3. Get the name of the MariaDB pod:
    • kubectl get pods | grep mariadb | awk '{print $1}'
  4. Copy the backup into the MariaDB pod:
    • kubectl cp backup.sql MARIADB_POD:/
  5. Get a shell to the MariaDB pod:
    • kubectl exec -it MARIADB_POD — /bin/bash
  6. Execute the mysql command in the MariaDB pod to restore the database:
    • mysql -uroot -p"${MARIADB_ROOT_PASSWORD}" < /backup.sql

Now refresh your WordPress browser window and you should see the previous state of the WordPress installation restored, working and looking as expected!

3.5 Scheduling regular backups

Note: The operations of this step can be executed at once using the scripts/5_schedule.sh script.
Instead of performing backups manually, you can also set a schedule for backups. This requires specifying the schedule in cron format.

backup:
  schedule: '*/2 * * * *'    # backup every 2 minutes
  keepJobs: 4
  promURL: http://minio:9000

Tip: Use crontab.guru to help you set up complex schedule formats in cron syntax.
The schedule can also specify archive and check tasks to be executed regularly.

archive:
  schedule: '0 0 1 * *'       # archive every week
  restoreMethod:
    s3:
      endpoint: http://minio:9000
      bucket: archive
      accessKeyIDSecretRef:
        name: minio-credentials
        key: username
      secretAccessKeySecretRef:
        name: minio-credentials
        key: password
check:
  schedule: '0 1 * * 1'      # monthly check
  promURL: http://minio:9000

Run the kubectl apply -f k8up/schedule.yaml command. This will setup an automatic schedule to backup the PVCs every 5 minutes (for minutes that are divisors of 5).
Wait for at most 2 minutes, and run the restic snapshots to see more backups piling up in the repository.
Tip: Running the watch restic snapshots command will give you a live console with your current snapshots on a terminal window, updated every 2 seconds.

3.6 Cleaning up the cluster

Note: The operations of this step can be executed at once using the scripts/6_stop.sh script.
When you are done with this tutorial, just execute the minikube stop command to shut the cluster down. You can also minikube delete it, if you would like to get rid of it completely.

4. Conclusion

We hope that this walkthrough has given you a good overview of K8up and its capabilities. But it can do much more than that! We haven’t talked about the archive, prune, and check commands, or about the backup of any data piped to stdout (called “Application Aware” backups.) You can check these features in the K8up documentation website where they are described in detail.
K8up is still a work in progress, but it is already being used in production in many clusters. It is also an open source project, and everybody is welcome to use it freely, and even better, to contribute to it!

Adrian Kosmaczewski

Adrian Kosmaczewski is in charge of Developer Relations at VSHN. He is a software developer since 1996, a trainer, and a published author. Adrian holds a Master in Information Technology from the University of Liverpool.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Press Project Syn Tech

First Pre-Release of Project Syn Tools

10. Mar 2020

We have been working hard since the initial announcement of Project Syn back in November 2019, and are proud to announce version 0.1.0, the first pre-release of a set of Project Syn tools.
Quick reminder about what Project Syn is about:

Project Syn is a pre-integrated set of tools to provision, update, backup, observe and react/alert production applications on Kubernetes and in the cloud. It supports DevOps through full self-service and automation using containers, Kubernetes and GitOps. And best of all: it is Open Source.

TL;DR: The code is on GitHub, under its own organization: https://github.com/projectsyn. The official documentation is in https://docs.syn.tools/ (The documentation is open source too!)

What does Project Syn do?

Short answer: it enables the management of many Kubernetes clusters, and provides a set of services to the users of those clusters. Project Syn is composed by many tools; some specially developed for the project, some already existing, all Open Source. It’s not only about tooling, it’s also about processes and best practices.
The actual story is a bit longer.

Features of version 0.1.0

To manage a big fleet of Kubernetes clusters, we need an inventory with the following information:

  • The cloud providers they are running on;
  • Locations;
  • Tenants each cluster belongs to;
  • Kubernetes versions deployed;
  • Kubernetes flavor / distribution used;
  • …and a lot more!

This is what the Project Syn tool Lieutenant (written in Go) gives us: an inventory application to register clusters, to assign them to a tenant and to store inventory data. It consists of a REST API (based on the OpenAPI 3 specification) and a Kubernetes Operator, to store data directly in the underlying Kubernetes cluster (in CRDs) and to act on events.
Knowing about clusters is just one part. Another important element is to continuously deploy and monitor system applications (like K8up, Prometheus, …) on Project Syn enabled Kubernetes clusters. This is all done with the GitOps pattern, managed by Argo CD, which is deployed to every cluster. Thanks to Argo CD we can make sure that the applications deployed to the cluster are exactly configured as specified in the corresponding Git repository, and that they are running just fine.
Each Project Syn enabled Kubernetes Cluster has its own so-called Catalog Git Repository. This contains a set of YAML files specifically crafted for each cluster, containing the system tools to operate the cluster, and to give access to well configured self-service tooling to the user of the cluster.
The generation of these YAML files is the responsibility of the Project Syn tool Commodore (written in Python). Commodore is based upon the Open Source tool Kapitan by leveraging inventory data from Lieutenant. After gathering all needed data about a cluster from the inventory, Commodore can fetch all defined components, parameterize them with configuration data from a hierarchical GIT data structure and generate the final YAML files, ready to be applied by Argo CD to the Kubernetes Cluster. The Lieutenant API also knows where the catalog Git repository is located, and Commodore is therefore able to automatically push the catalog to the matching Git repository.
Secrets are never stored in GitOps repositories. They are instead stored securely in Hashicorp Vault, and only retrieved during the “apply” phase, directly on the destination Kubernetes Cluster. This process is supported by the Kapitan secret management feature and by Commodore, who prepares the secret references during the catalog generation. Argo CD calls kapitan secrets --reveal  during the manifest apply phase, which then actually connects to Vault to retrieve the secrets and stores them in the Kubernetes Cluster, ready to be consumed by the application.
The management of all these Git repositories is the responsibility of the Lieutenant Operator (written in Go, based on Red Hat’s Operator SDK). It is able to manage remote Git repositories (GitLab, GitHub, Bitbucket, etc) and prepare them for Commodore and Argo CD, for example by configuring an SSH deploy key.
The Project Syn tool Steward (written in Go) has the responsibility of enabling Project Syn in a Kubernetes Cluster, communicating with the Lieutenant API, to perform the initial bootstrapping of Argo CD. This bootstrapping includes basic maintenance tasks: should Argo CD be removed from the cluster inadvertently, Steward will automatically reinstall it. An SSH deploy key is generated during bootstrapping and transmitted back to the API. With this procedure it is possible to bootstrap the whole GitOps workflow without any manual interaction.

Analogies with Puppet

For those familiar with Puppet, there are some similarities with the design of Project Syn:

  • Puppet Server: Commodore and Kapitan to generate the catalog, matching the facts from the cluster.
  • Puppet DB: Lieutenant acting as inventory / facts registry.
  • Hiera: Kapitan with its hierarchical configuration model.
  • Puppet Agent: Steward and Argo CD on the cluster. Steward to communicate with the API and Argo CD to apply the catalog.
  • Puppet Modules: Commodore Components, bringing modularity into Kubernetes application deployment.

Many of these concepts are documented in the Project Syn documentation pages, specifically the Syn Design Documents, documenting all the design decisions (even though they are still in “work-in-progress” stages).

What are the next steps for Project Syn?

This is really just the beginning! There are a lot of plans and ideas for the future evolution of Project Syn. We have crafted an initial roadmap, and we published it as part of the official Project Syn documentation.
This initial pre-release is just the tip of the iceberg. Under the surface there is a lot more brewing, to be released as soon as possible. To reiterate: It’s not only about tools, but also about concepts and processes, which also means a lot of documentation will emerge over the next months.
One of the focus of this initial pre-release was to lay the foundation for future development. It has a strong focus on the operations side. Future milestones will broaden the focus to include more and more self-service possibilities for the user, including tight integration of Crossplane for easy and fully automated cloud service provisioning.
We at VSHN are now starting to use Project Syn for an initial set of managed Kubernetes clusters, and will continue to develop the concept, tools and processes while we learn about more use cases and with the real-life experience we gather.

How can I contribute?

Project Syn is a young project and is making the first initial steps in the open world. Many things are just getting started, just like the documentation and the contribution guidelines. Testing and giving feedback through GitHub issues is certainly a great way to start contributing. And of course, if you are looking for a Managed Kubernetes or Managed OpenShift cluster, get in touch with us with the form at the bottom of this page!

Learn more

Second Beta Release of Project Syn Tools

Tobias Brunner

Tobias Brunner is working since over 20 years in IT and more than 15 years with Internet technology. New technology has to be tried and written about.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
General Tech

Our way to Managed OpenShift 4

23. Sep 2019

Red Hat OpenShift 4

This summer Red Hat released OpenShift 4. At first glance, the new major version is a continuous development of OpenShift 3 with relatively manageable changes for the user. But if you look under the hood, you will quickly see a completely revised OpenShift. The blogpost of Benjamin Affolter on the APPUiO blog examines the changes of OpenShift 4 and describes them in detail.

With the following article we would like to take a look behind the scenes of our Managed OpenShift offering and explain what we have to do to be able to offer our Managed Service with OpenShift 4.

Advantages of OpenShift 4

Red Hat promises the following improvements with version 4 of OpenShift:

  • New Installer
  • Fully automated operations, maintenance and configuration using Operators
  • Integration of Operator Hub
  • Current versions of Kubernetes

To fully understand the benefits and also the implications, we need to take a step back and take a look at OpenShift 3.

Managed OpenShift 3 – what’s included?

For better understanding, you can find a short overview of what our Managed OpenShift 3 service includes so far (not exhaustive):

  • Architecture engineering and setup of the OpenShift Cluster on almost any infrastructure (Cloud, On-Premise)
  • Monitoring of all cluster-relevant components to ensure operation
  • Regular backup of the cluster configuration incl. ensuring the integrity of the backup
  • Weekly maintenance of all systems, application of software patches and configuration improvements on all clusters
  • Automation of all work with Ansible (configuration, maintenance, updates, upgrades, installation, sanity checks and much more)
  • Integration into our central customer portal for an overview of the status of the cluster and other functions
  • Extensive dashboards in Grafana
  • Close cooperation with Red Hat Support for solving bugs in OpenShift, among others
  • Maintenance of various internal lab clusters to test changes to productive clusters
  • Provision of persistent storage using Gluster
  • Management and maintenance of the operating system Red Hat Enterprise Linux for the OpenShift masters and nodes
  • Training of system engineers to run OpenShift

All these listed points have been developed since the very first version of OpenShift 3 and are developed daily by our VSHNeers.

Status Quo VSHN Systems

From a technical point of view, our current system landscape looks something like this (brief overview):

  • Puppet for the local operating system management of all VMs (system configuration, maintenance of the defined state) and inventory of all systems and services.
  • Icinga2 for monitoring all operating system parameters within the VM, but also very extensive checks of all OpenShift cluster components. Icinga2 is configured and orchestrated by Puppet.
  • Ansible for installation and configuration of OpenShift, for regular maintenance and for much more
  • BURP for consistent data backups incl. cluster configuration, configured and orchestrated by Puppet
  • Gluster for persistent storage, managed by Ansible

Over the years, countless Ansible Playbooks have accumulated and all our knowledge and automation has gone into these Playbooks. We maintain our own fork from the official OpenShift Ansible Repository to be able to react quickly to any bugs. We regularly keep this fork up to date with upstream.
Puppet not only takes care of the local operating system configuration, but also controls many important components such as the monitoring and backup system. In addition, the PuppetDB provides us with an up-to-date inventory of all systems managed by us, including detailed version information of the installed components. This is also integrated in our customer portal and is used for automatic billing of our managed services.
The monitoring plugins we developed for Icinga2 cover almost every problem we have discovered with OpenShift and notify us if there is anything wrong with the cluster or one of its components.
Our system documentation and OpenShift operation guide include several dozen Wiki articles.

Managed OpenShift 4 – what is there to do for VSHN?

From a system engineering point of view, OpenShift 4 is a completely new product. For VSHN this means that we have to completely redevelop a large part of the above points.
A few examples:

  • The installation and configuration of OpenShift 4 is no longer based on Ansible, but on a separate installer (which uses Terraform in the background) and the configuration is done by In-Cluster Operators. Our Ansible Playbooks for OpenShift 3 can for the most part no longer be used for OpenShift 4.
  • The operating system is no longer Red Hat Enterprise Linux, but Red Hat CoreOS, which behaves completely different. Puppet cannot be used anymore and as described above we have to find other ways to inventory, orchestrate and bill the surrounding systems.
  • Our monitoring plugins for Icinga2 are no longer compatible with OpenShift 4 and the monitoring concept with Icinga2 no longer fits the platform’s revised architecture. For us this means a new development of our monitoring concept.
  • The backup system BURP can no longer be used in its current form, a new backup system has to be developed.

This is not an exhaustive list, there are many more details in our system landscape that need to be adapted.

The path to production

For us as a Managed Service Provider, stability and scalability are the most important points which are non-negotiable. This means that we have to take the necessary time to learn all the changes and peculiarities for a productive operation of OpenShift 4. The adaptation and development of the necessary tools and processes for the operation of dozens of clusters requires a lot of time and engineering effort. However, we started early and have already gained some experience with OpenShift 4. The experience gives us great confidence that OpenShift 4 can deliver on its promises of greatly simplified operation.
The current version OpenShift 4.1 also has some limitations. Here is a small selection of what we noticed:

  • No support for proxies
  • AWS and VMware are the only supported IaaS providers with OpenShift 4.1 (current version at the time of this article)
  • Installation on unsupported and non-cloud platforms is very fragile
  • Container storage only via CSI

Many IaaS providers are not yet ready for OpenShift 4, but we are in close contact with our IaaS & Cloud partners like cloudscale.ch, Exoscale, Swisscom and AWS, to ensure compatibility so that we can continue to offer a smooth operation with OpenShift 4.
OpenShift 4.1 reminds us partly of the early days of OpenShift 3, when it took some time until OpenShift 3 was ready for production.
But we are very confident that the open issues can be solved and we are looking forward to the 4th generation of Red Hat OpenShift!

More Info

Our friends from Adfinis SyGroup have described their first experiences with OpenShift 4 in their blog post “OpenShift 4 – Learnings from the first productive environment” which fits very well with our observations.
If you want to learn more about OpenShift and Kubernetes, we recommend reading our article “What is a Kubernetes Distribution and what are the differences between Kubernetes and OpenShift” or have a look at the impressions of Red Hat Forum Zurich 2019, where APPUiO was a sponsor and where we had a booth on-site.

APPUiO – Swiss Container Platform

With APPUiO.ch we have created a Swiss Container Platform based on Red Hat OpenShift on which we offer Managed Services as a PaaS solution (Platform-as-a-Service) on any infrastructure: public, dedicated, private and on-premises. Based on proven Open Source concepts like Docker and Kubernetes you develop, operate and scale an application according to your needs. With APPUiO, your applications can run on public clouds as well as in-house. The platform was originally developed in 2015 by the two IT specialists Puzzle ITC and VSHN AG for the professionalization of their internal IT. Today, APPUiO is used productively by many customers and is supported by a strong community.

How can we help?

With our experience in operating OpenShift clusters around the world, we offer managed OpenShift clusters on almost any public, private or on-premise cloud. We are happy to help with evaluation, integration and operation and support with our many years of Kubernetes experience. Contact us, subscribe to our newsletter and follow us on Twitter (@vshn_ch and @APPUiO) to keep up with the latest news and have a look at our Services.
We look forward to your feedback!

Tobias Brunner

Tobias Brunner is working since over 20 years in IT and more than 15 years with Internet technology. New technology has to be tried and written about.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us