Welcome to this first article of a series of three, about the subject of security in networking applications, TLS, certificates, keys, and more.

• Part 1: What is TLS?
• Part 2: Verifying and Connecting
• Part 3: Trust and Certificate Chains

We have noticed that many of our customers struggle to understand how TLS certificates work on a conceptual level. For example, how does “signing” work? What does it even mean? What are a “chain”, an “intermediate” or a “root”? Why do intermediates belong to a chain, but not the root? What is a CSR? How do CSR, Key, and certificate work together? And so on.

This series is an attempt to explain these critical concepts to our customers, starting this first part with the basic vocabulary and knowledge to get started.

HTTPS – TLS – SSL

Let’s talk about what we want to achieve first. When you visit a website via plain old HTTP, an attacker could intercept your request and grab any private information like usernames and password. Additionally, there is no way for you to verify that you indeed connected to the server you intended. An attacker could have modified the DNS response (which is also unencrypted) to send your browser to their server instead. See Man-in-the-middle attack on Wikipedia for more examples.

So in order to verify the identity of the server we connected to, and to make sure nobody except the server and our browser can read the data we exchange, websites these days use TLS (“Transport Layer Security”, or its predecessor SSL “Secure Sockets Layer”) to both authenticate the server and encrypt the traffic.

From a technical perspective, TLS sits between TCP and HTTP on the protocol stack, and while it’s mostly known for being the S (for Secure) part in HTTPS, it’s noteworthy that it can be used for other protocols as well. Some examples:

• Email protocols: IMAPS, POP3S, SMPTS, …​
• Database connections: MySQL, PostgreSQL, MongoDB, etcd, …​
• Data transfer: FTPS
• Telephony: SIPS
• Chat: XMPPS, IRCS, …​

The last thing to note here is that there are different versions of TLS (and SSL), and some of them are not considered secure anymore!

From oldest to newest, at the time of writing (March 2021):

• SSLv1: Insecure, not supported anymore
• SSLv2: Insecure, not supported anymore
• SSLv3: Insecure, not supported anymore
• TLS 1.0: Insecure, deprecated
• TLS 1.1: Insecure, deprecated
• TLS 1.2: OK
• TLS 1.3: OK

Keep this in mind when planning your environment! You wouldn’t want to protect your brand new microservice with outdated security protocols!

By the way, if you want to check which TLS versions a website supports, use SSL Labs’ SSL Server Test. It’s a great debugging tool and will show you a lot of information about the topics of this Blog post series!

Certificates

The next thing we should have a look at are Certificates, or more specifically X.509 v3 public key certificates.

A Certificate is a cryptographically signed piece of information, of which the most important part is the Subject, which identifies who this certificate belongs to, as well as the Issuer. Other attributes include details about when the certificate is about to expire as well as a lot of technical information about the key and signature algorithms used, and so on.

The Subject and Issuer of a certificate are characterized by a set of attributes:

• CN – Common Name
• C – Country
• ST – State
• L – Location (City)
• O – Organisation
• OU – Organisational Unit

Together, those attributes form a Distinguished Name (DN). Most attributes are optional, except for the Common Name. In the case of Server Certificates, the CN must match the address used to connect, for example www.vshn.ch.

The Certificate also contains a Public Key (embedded in the certificate) and its matching Private Key.

The last important feature of X.509 certificates is that they are signed by other certificates (or itself). Once a certificate is signed, its contents cannot be changed anymore.

Signing CAs

The Internet is a big place, so how do we know whom to trust? To solve this issue, the concept of Certificate Authorities came up.

A Certificate Authority (commonly referred to as CA) is a central trusted source of certificates. They have a CA Certificate (also known as the “root” certificate) that is well known and trusted. This certificate is only used to sign other certificates. By signing another certificate, the CA confirms “yes, I trust this certificate and the information in it is correct.”

A related concept is a Certificate revocation list, also known as “CRL”, which is a list of digital certificates that have been revoked by the CA before their scheduled expiration date, and should therefore no longer be trusted.

Now that we have explained what a certificate is, let’s talk about keys.

Keys

Each certificate has its own key. The key makes certificates actually useful; without them, they would not work.

Each key consists of two parts: a public key and a private key. While the public key is embedded as part of the certificate itself, the private key is stored in a different file altogether.

In order to sign another certificate, you need the Private key of the signing certificate (but not of the certificate you want to sign).

When you open a connection to a server, the server also needs the private key of the server certificate in order to authenticate itself and establish encrypted communication.

This whole concept is called Public-key cryptography.

Confidentiality

A very important thing to understand is that the key is the only confidential piece in the puzzle; needless to say, it is the most important piece! So please, keep this always in mind: do not ever exchange keys over unsecured channels, under any circumstances!

On the other hand, the certificate itself is not confidential. The CA certificate in particular must be provided to all clients and servers, in order to verify other certificates.

How can a client ask a CA to sign a certificate? Through a CSR (also known as Certificate Signing Request) which, as the name implies, performs exactly this task.

Coming Up

Speaking about verification and issuing new certificates, we are going to talk about these subjects in detail, respectively in the second and third parts of this series. Stay tuned!

I think it can be said that at VSHN we have a certain obsession with documentation. This goes beyond the requirements set by our customers, or by those imposed by certifications such as the ISO 27001 or ISAE-3402. We have found that as a team we can work better when we can refer to written documentation at all times.

In the past few years we have standardized our documentation toolkit: we have adopted Asciidoctor as the lingua franca for our documentation. We have put in production various documentation websites based on Antora: our Handbook, our Knowledge Base, and the Project Syn documentation are some examples.

That works great for text. But what about diagrams? Our documentation has lots of them, and although embedding PNG files works fine, it does not “scale” very well.

For that reason we have set up an instance of Kroki in production (in APPUiO, of course!), to enable the generation of SVG diagrams based on various formats: PlantUML for UML diagrams, Nwdiag for network diagrams, and Ditaa for simple box-and-arrow kind of diagrams, all produced and shown in a crisp and neat SVG format. We love Kroki! It is such an awesome idea, that it has even been integrated to GitLab as a supported diagramming tool. How cool is that? In general, the Asciidoctor ecosystem has very good integration of diagrams into workflows, and this extends to Antora as well.

I am not going to dive into the benefits of vector vs. raster images; I think the readers of this article are well aware of the bonuses the former has over the latter. SVG diagrams look great in both online and print, but they can be a bit complicated to draw by hand (duh!).

But here’s a confession: I’m not a big fan of editing text-based diagrams either. But that’s just a personal choice. And the truth is, many other non-technical VSHNeers find the whole “diagrams as text” thing a bit arcane.

So that means that we needed a more accessible tool for them. The ideal diagram drawing tool for VSHN must have the following requirements:

• Cross-platform: Most of us at VSHN use Linux, but a sizable number also use Macs and Windows laptops.
• Accessible to non-technical VSHNeers: Not everyone at VSHN is a DevOps engineer.
• Simple to use: If possible, using a point-and-click interface; for example, in our Confluence wiki we have enabled the Gliffy extension, which allows non-technical users to create stunning diagrams.

So we started our search. Leaving aside tools like Inkscape, that are too generic, like Asciiflow, that are too limited, or like Dia, that are too platform-specific, it took us a while to find a tool that would fit our requirements. But we found it!

Please welcome Diagrams.net (previously also known as Draw.io). It is a web application geared specifically to the generation of diagrams, very easy to use and ticking all the boxes that we needed.

And even better, there is a Visual Studio Code extension that allows to edit diagrams locally, directly on your laptop, in Linux, Mac and Windows.

How do we integrate this with Antora? Very simple. In the assets/images folder of our documentation components, create your diagrams using the *.drawio.svg extension. These are automatically associated by Visual Studio Code with the extension, and provides a live editor with all the bells and whistles you might expect.

And then, well, just image::diagram.drawio.svg[] in your documents, git commit and push.

This approach is definitely more accessible to non-technical VSHNeers, making it super easy to edit diagrams online, or locally in Visual Studio Code.

To finish, here’s a Pro Tip™®©: switch the Draw.io extension to the “Atlas” theme, since some lines in the extension lack contrast, and aren’t easily seen when using a dark theme. You can change the Draw.io extension theme with the selector at the bottom right of the screen.

PS: there is an open issue in Kroki to add support for Draw.io diagrams. Can’t wait to see it in action!

Docker Inc., the company behind Docker Hub, has recently announced an enforcement of image pull rate limits for Docker Hub users. This change affects all cloud native installations currently configured to consume container images stored in Docker Hub, one of the most popular image repositories available today. This measure has a direct impact in the deployment of many customer applications currently running on APPUiO Public and Private Clusters.

We are following the situation closely. Following recent announcements, Docker Hub is slowly starting to enforce a pull rate limit of 100 pulls per 6 hours for anonymous (unauthenticated) IP-Addresses, and 200 pulls per 6 hours for authenticated non-paying users. This means that a Kubernetes or OpenShift cluster can only pull 100 images in 6 hours without authentication for all of its users from Docker Hub. During maintenance periods, most of the images of a cluster are pulled at some point, and the reduction of pull rate limits can cause downtime when the images can’t be pulled due to the limit.

This situation should not affect today’s maintenance window. Even though the announced start date of the rate limit was Monday, October 2nd, 2020 the analysis of responses from Docker Hub indicates that the new limits are not yet applied, and will only be enforced during a 3-hour window from 18:00 to 21:00 (CET) tomorrow. Unfortunately at this point we don’t know when the new pull rate limits will be enforced fully.

Also noteworthy, these restrictions do not apply to the internal OpenShift registry included with APPUiO Public or Private, which is completely independent of Docker Hub, nor to Docker users with Pro or Team accounts. Authenticated users with Pro or Team accounts enjoy unlimited data transfer to and from Docker Hub.

VSHN is currently evaluating measures to prevent downtime, and reduce the impact of this situation for our customers. The most appropriate solution at this moment consists in switching to the aforementioned Pro or Team Docker account types. Even simpler, the use of an authenticated user account in a pull secret instead of an anonymous account will double the pull rate limit, and will reduce the risk of downtime considerably. Another possibility consists in migrating images to a different public registry, like Red Hat Quay. Another option is the use of a private registry, such as the OpenShift integrated registry, AWS ECR, GitLab Container Registry, Harbor, or other similar technologies. In particular, AWS has recently announced the future availability (“within weeks”) of a free public registry.

We remain at your service for any enquiry. Contact us if you need more information.

This is about how we manage large numbers of ModSecurity WAF deployments, including CRS and custom rules.

ModSecurity CRS Berne Meetup

At VSHN we’re not only 100% committed to Free and Open Source software, we also collaborate actively with the communities of the software we use, deploy and enhance. This June, we had the pleasure to join the OWASP ModSecurity Core Rule Set Berne Meetup group online who had asked us back in November at yet another Meetup to showcase the way we use ModSecurity and the Core Rule Set (CRS).

This post comes late after my EuroPython 2019 talk on “Modern Continuous Delivery” in Basel. But there is no need to worry: Advice on writing software that outlasts hypes on modern computing has no hurry to appear on stage.

What a great time!

Deployment automation, cloud platforms, containerization, short iterations to develop and release software—we’ve progressed a lot. And finally it’s official: Kubernetes and OpenShift are the established platforms to help us do scaling and zero downtime deployments with just a few hundred lines of YAML. It’s a great time.
Can we finally put all our eggs into one basket? Identify the cloud platform that fits our needs, and jump on it? That could well backfire: Vendor lock-in is the new waterfall, it slows you down. In future you’ll want to jump over to the next better platform, in a matter of minutes. Not months.
So, how do we do that? What’s the right approach?

The definition of Continuous Delivery

What does modern software development look like?

A modern software development project does:

1. Immutable infrastructure (“containerization”)
2. Container orchestration (e.g. Kubernetes, Docker Swarm)
3. Version control and automation (CI/CD, “pipelines”)
4. Cloud-native applications (resilient apps that scale)

Nothing new. You’ve heard this before.

Vendor lock-in is the new waterfall

Software development has become complex. So complex that there are numerous services popping up almost daily that help us getting things done. Most notably, these are application delivery platforms nowadays (Amazon AWS, Microsoft Azure, Google Cloud — just to name a few). When we use offerings across several such providers, which is becoming increasingly popular, we speak of multi-cloud dependencies.

While all offerings are choices they typically entail a lock-in, because we don’t have standards and tooling yet that allow us to effortlessly switch from one solution provider to another. And when there are reasons to make a switch, lock-ins make it inherently difficult to move fast. This is a problem.

What can we do about vendor lock-in?

Software development is not about “using <technology>” or “using <platform>”. As engineers we must think in terms of “problem to solve” (requirements) and applying proper development practices (engineering). If we rely on platforms to solve problems for us we’re doing it wrong. Engineers must learn to follow principles of good software design, to write and maintain “clean code”.

One of those principles you learn as an engineer is to maintain boundaries between systems. In a modern software development project you’ll see four of such “layers of responsibility” that define boundaries:

1. Application
2. Development
3. Deployment
4. Automation

Think in terms of technologies and tools, or services and environments you use in each to understand why we have those.

1. The application layer is like how you did software development 15 years ago. It’s just concerned with getting things running locally. Add the 12factor app methodology and you get an application that is prepared for potential target environments.
2. The development layer is concerned with supporting the development in the application layer. Developer tooling that gets the project running with a single command, a test infrastructure setup and QA tooling, which should also be easy to handle.
3. The deployment layer reuses that part of the development layer that made the developer deploy the application locally, for development. It also houses configuration files used only in productive target environments and deployment configuration that describes the entire productive setup.
4. The automation layer is only concerned about automating the steps you would otherwise perform manually to deploy your application.

Clean separation and interfaces

Now we have layers. What do we gain?
Note how the layers use an interface to talk to the next layer above. This allows us to address them separately. In other words,

• When you want to use a different CI service (e.g. switch from Bitbucket Pipelines to GitLab CI) you only refactor the implementation in the automation layer (i.e. the CI configuration).
• When you want to change your container orchestration (e.g. switch from OpenShift to Kubernetes) you only refactor your implementation of the deployment layer (e.g. use Kustomize instead of OpenShift templates and Ingress instead of Routes, etc.). You may also have to change some of the deployment tooling in the automation layer as a consequence (e.g. use kubectl instead of oc), but it’s all cleanly separated.

This is the power of separation of concerns.
Also note that for this to work well you need to make your interfaces simple and stable.

• For development use a task runner to turn your common tasks into single commands or simple one-liners (in JavaScript you may use gulp, in Ruby rake, in Python tox).
• Don’t design for target environments in your application layer! Design with features in mind and only combine them to environment configurations in your deployment layer.
• Stick to standard procedures and established tooling (instead of proprietary or self-invented solutions) for the technologies you use. This is typically more stable as an interface, will save collaborators from learning how to work with your setup and make it unnecessary to add extensive instructions to the README.

The more you invest in this flexibility and clean separation the easier your task will be when the day comes to make a change.

Working code samples to your rescue

What is better than starting to investigate late and working on refactoring under time pressure? Knowing your possibilities and turning to working code samples when you need them!
That’s where the Painless Continuous Delivery Cookiecutter comes into play. It’s a project template generator covering many of the most popular combinations of public service offerings you may be working with. At VSHN AG we’re working on extending it—for your independence. Try it out, let us know if you find it useful, and contribute if you feel like!
For the start you may want to take a look at one of the generated setups on GitLab, the live demo for the EuroPython talk. Enjoy!

EuroPython 2019: Modern Continuous Delivery for Python Developers

Slides of this talk are available from SpeakerDeck.
This article has originally been published on Peter’s personal website.

use create

deploy step by step our operators.
For this SDK supported workflow, we provide:

a) a description of the generated Operator structure (there is one for each specific workflow);

b) a link to our example(s) of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.

c) descriptions on how to add (e.g., 3rd party) resources, different from the Core Kubernetes resource types, to your Operator 

d) a description of the main pros and cons of using such Operator structure (there is one for each specific workflow) and logic.

a) Description of the generated Operator structure (there is one for each specific workflow)

As follows the structure of a generated GO operator:

Operator scope

A namespace-scoped operator (the default) watches and manages resources in a single namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide. Namespace-scoped operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions. However, there are use cases where a cluster-scoped operator may make sense. For example, the cert-manager operator is often deployed with cluster-scoped permissions and watches so that it can manage issuing certificates for an entire cluster.

b) Example of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.

We will discuss two examples of GO Operator: 
1) Example 1: this operator just replicates itself under a certain project: 
2) Example 2: this operator allows to manipulate the number of pods in a project.

c) How to add 3rd Party Resources (i.e., different from the Core Kubernetes resource types) to your Operator

The operators manager supports the Core Kubernetes resource types as found in the client-go scheme (“register.go“) package and will also register the schemes of all custom resource types defined in your project under pkg/apis

To add a 3rd party resource to an operator, you must add it to the managers scheme. By creating an AddToSchememethod or reusing one you can easily add a resource to your scheme. An example shows that you define a function and then use the runtime package to create a SchemeBuilder.
Register with the managers scheme
Call the AddToScheme() function for your 3rd party resource and pass it the managers scheme via mgr.GetScheme().
Example:

“After adding new import paths to your operator project, run dep ensure in the root of your project directory to fulfill these dependencies”.

d) A description of the main pros and cons of using this Operator structure (there is one for each specific workflow) and logic;

PROS (+) AND CONS (-):

(+) it is easy to use and create a simple operator

The main program for the operator cmd/manager/main.go initializes and runs the manager. The manager will automatically register the scheme for all custom resources defined under pkg/apis/... and run all controllers under pkg/controller/....

The manager can restrict the namespace that all controllers will watch for resources:

mgr, err := manager.New(cfg, manager.Options{Namespace: namespace})

By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty:

mgr, err := manager.New(cfg, manager.Options{Namespace: ""})

(+) enough clear way to add 3rd Party Resources (i.e., different from the Core Kubernetes resource types) to your Operator

(-) it is still unstable/changing the APIs and the future versions of it could be not supported (we are observing this evolution and update the blog post accordingly)

To make the operator working, we had to open some issues in Github, that was closed and classified as a bug):

https://github.com/operator-framework/operator-sdk/issues/651

https://github.com/operator-framework/operator-sdk/issues/927

https://github.com/operator-framework/operator-sdk/issues/1053

(-) The workflow to create the GO operators appear a bit more complex/unclear (steps in red) that the one generated with Ansible and Helm (which require fewer steps):

1. Create a new operator project using the SDK Command Line Interface (CLI)
2. Define new resource APIs by adding Custom Resource Definitions (CRD)
3. Define controllers to watch and reconcile resources
4. Write the reconciling logic for your controller using the SDK and controller-runtime APIs
5. Use the SDK CLI to build and generate the operator deployment manifests

Back to overview

Back to overview Section 3 – Examples of Supported Kubernetes Operator SDK workflows.

2022–03–09: There’s a newer version of this article!

(This blog post is the transcription of the presentation given at the CNC Switzerland meetup, May 9th 2019. Slides are available at the end of this page and impressions of the meetup can be found here.)
Serverless is one of those hot topics that, as many others in our industry, looks a bit like a good old idea recycled and brought back to fashion. Yet Serverless (or “Function as a Service”) looks like a natural evolution to a movement that started more than a decade ago, when Heroku and Google App Engine came under the spotlight.
As says Martin Fowler himself, Serverless and FaaS can be defined as follows:

FaaS is about running backend code without managing your own server systems or your own long-lived server applications.

At VSHN we have been interested in the subject for years now, and even worked in a FaaS project for a customer two years ago.
But now that Kubernetes and Knative are emerging as the true “Golden Standards” of hosted cloud operations, we see the emergence of FaaS solutions running directly on top of them. And developers have been fast creating lots of solutions that somehow appear to occupy the same space. How do they compare? Which one to choose?
To answer these questions, we are going to give a brief description of the following frameworks, outlining some of their characteristics and relative strengths:

1. OpenFaaS
2. Fn Project
3. Fission
4. OpenWhisk
5. Kubeless
6. TriggerMesh

The order of the frameworks is not trivial; they are roughly ordered following their “level of abstraction” on top of Docker and Kubernetes. For each of these projects, we are going to provide the following information:

1. Project details;
2. Three demos, all of them recorded with asciinema using minikube;
3. Available triggers;
4. Supported programming languages.

1. OpenFaaS

OpenFaaS is the project with the most stars (14’000) on Github of all those in this article. It is mostly written in Go, featuring around 100 contributors, and the latest available version at the time of this writing is 0.13.0 (April 4th, 2019).
It is an independent project, funded through donations (Patreon, OpenCollective, and PayPal.)
From a developer experience perspective, it is a project quite complex to setup and use. It is based on Docker, which means that functions are actually packaged as containers, to be pushed to a repository, and to be built using a local Docker installation in the developer workstation. OpenFaaS manages the Dockerfile for the developers automatically, though.
OpenFaas has a “template store” with several available programming languages. It also provides developers with a command-line utility, called faas-cli, itself talking to a REST API documented using Swagger. Finally, there is a Serverless Framework plugin for those who need it.
The following asciicast shows a very simple interaction with OpenFaas. First we create a Python function, which we customize a bit, and then we deploy and call it from the command line; both directly and also using curl:

OpenFaaS functions can be called through the following triggers:

• HTTP
• Event Connectors: Kafka, Amazon SNS, Minio, CloudEvents, IFTTT, Redis, RabbitMQ…

Finally, developers can use the following programming languages with OpenFaaS:

• C#, F# (.NET Core)
• Go
• JavaScript (Node.js)
• Java (JVM)
• PHP 7
• Python / Python 3
• Ruby

2. Fn Project

The Fn Project has been started and is currently funded by Oracle, who uses a fork to power its own Oracle Functions product.
Just like OpenFaaS, it is hosted in Github and written in Go. The project has around 4000 stars and 80 contributors, and the latest version at the time of this writing is 0.3.703 (May 6th, 2019).
From a technical point of view, Fn can use any Docker container as a function, and it can run anywhere: in public, private, and hybrid clouds.
Fn has two major concepts: – Functions (defined in YAML) – Applications: groups of functions, which can be deployed all at once.
For developers, it offers a command-line tool called fn and a Serverless Framework plugin.
Fn functions can be triggered with HTTP calls, and can be developed using the following languages:

• Go
• JavaScript (Node.js)
• Java (JVM)
• Python
• Ruby
• C# (community supported)

The Fn marketing material further states that it “supports all languages”.

3. Fission

Fission is an open source, Kubernetes-native Serverless Framework. It allows functions to be deployed and executed instantly, mapping them to HTTP requests.
Its Github project is mostly written in Go, features 4300 stars and around 80 contributors at the time of this writing. Its latest available version is 1.2.0 (May 3rd, 2019). It was started and is currently maintained by Platform9.
Fission does not need Dockerfiles or Docker registries; it is based on the notion of environments. Functions are injected into those environments, which are a pool of containers with language runtimes, where functions are loaded and launched on demand.
Fission keeps in memory a set of images containing the runtimes where the functions will be run, injecting them and running them immediately when invoked. In this sense, it is similar to how AWS Lambda works.
For developers, it features a command line tool (fission) and a Serverless Framework plugin. They do not need to have a local Docker environment to build their functions into.
The following asciicast shows the basic operations required to create, deploy and call a function:

Currently, Fission supports following types of triggers:

• HTTP
• Time
• Message Queue
• Kubernetes Watch

Only the following programming languages can be used to create functions in Fission; the project is quite young and the list will probably grow in the future:

• Go
• Python
• JavaScript (Node.js)
• Java (JVM)

In our tests, using minikube as a support, Fission appears easy to use, but at the same time very fragile (in spite of what its version number might suggest.) Removing and re-creating environments and functions led to many problems, and the project is too young to have more than 5 (unanswered) questions on Stack Overflow. In short, a promising but yet rather immature product.

4. OpenWhisk

OpenWhisk is the behemoth in the room. This open source project was created by IBM and is currently managed by the Apache Foundation. This project is the most “corporate” ones of those described in this blog post. It is written in Scala, it features around 4000 stars in Github and has around 150 contributors. The latest available version at the time of this writing is 0.9.0 (October 31st, 2018.)
This framework has the following features:

• Very “corporate” in design and functionality;
• Secure by default;
• Forked by Adobe and other big corporations;

For developers, it features a command-line tool (wsk) and a Serverless Framework plugin.
OpenWhisk functions can be triggered by the following mechanisms:

• Message Queues
• Databases
• Document Stores
• Website or Web Apps
• Service APIs
• IoT Frameworks…

OpenWhisk function can be created using the following programming languages:

• C#, F# (.NET Core)
• JavaScript (Node.js)
• Swift
• Java, Scala (JVM)
• Go
• Python
• PHP
• Ruby
• Ballerina
• Through Docker Actions: Rust, Haskell…

The installation on minikube was the most complex and difficult of all the frameworks considered in this document. The tools have changed a lot in the last two years and resources online might be outdated. But in spite of those factors, this framework stands out by the quantity, breadth, and depth of the documentation, as well as by the number of integration and supported languages.

5. Kubeless

Kubeless is a promising framework created and maintained by Bitnami. It is an open source project on Github written in Go, with around 4600 stars and 80 contributors. At the time of this writing, its latest version is 1.0.3 (March 14th, 2019.)
In our tests it was the one offering the best developer experience. Very simple to install and use, it offers a command-line tool (kubeless) that is very similar to the AWS Lambda CLI. This is no coincidence, as the whole aim of the project is to provide an experience very close to that of AWS Lambda, Azure Functions or Google Cloud Functions.
For DevOps teams, Kubeless provides Prometheus monitoring of functions calls and latency, and a Serverless Framework plugin.
The following asciicast shows the basic interaction to create, deploy, and test a function using Kubeless:

Kubeless functions can be triggered through the following mechanisms:

• HTTP
• Cronjob
• PubSub mechanisms
  • Kafka
  • NATS messaging
• Data stream events
  • AWS Kinesis

These functions can be developed using the following languages and runtimes:

• Go
• Python
• JavaScript (Node.js)
• Java (JVM)
• Ruby
• C#, F# (.NET Core)
• Ballerina
• Custom runtimes possible

All frameworks herewith considered, Kubeless offered the most flawless experience of all. The documentation was solid and easy to follow, and there are plenty of online resources to guide developers into building applications using this platform.

6. TriggerMesh

TriggerMesh is the newest entry in the world of FaaS, and it will represent a major shift in the way serverless applications are deployed and executed. Founded ex-Kubeless engineers, it builds upon Kubernetes & Knative, providing features yet unseen in the serverless arena.
TriggerMesh functions can be triggered through cross-cloud triggers from AWS to functions on Knative:

• Code Commit
• Cognito
• DynamoDB
• Kinesis
• S3
• SNS
• SQS
• …

TriggerMesh has announced the following programming languages as options:

• Go
• JavaScript (Node.js)
• Python
• Ruby

Together with VSHN, TriggerMesh released the TriggerMesh Operator for OpenShift 4.0. OKD 4.0 was recently announced by Red Hat to bring additional automation to Kubernetes applications. The operator allows OpenShift users to install the TriggerMesh management platform and benefit from its integration with Knative, to power serverless workloads across multiple clouds.. TriggerMesh also allows CI/CD of serverless functions, as well as access to multi-cloud event sources, like Azure and AWS.

Comparison

The following chart summarizes some of the ideas of this article, and has been adapted from the “Examining the FaaS on K8S Market” article on the Cisco Blog.

Popularity

The following tweet provides interesting information about the current state of the FaaS-on-Kubernetes market:

32% of those using or planning to use a serverless platform have a Kubernetes-based installable platform in their plans, with Kubeless often being cited alongside other options. https://t.co/RYEwShkrLb pic.twitter.com/lqWWyrQu8B

— Lawrence Hecht (@LawrenceHecht) October 25, 2018

Conclusion

I hope this summary will be useful to you! Having tested all of these options, here at VSHN we will be focusing our efforts in the TriggerMesh platform, which will provide a much more solid developer experience and an unprecedented level of flexibility. We believe that this is the next generation of serverless platforms and we cannot wait to bring its power to our customers.
TriggerMesh will be launching their TriggerMesh Cloud service in the near future which will allow users to host serverless functions and consume events from many cloud sources.  To join the TriggerMesh Early Adopters program and get access for free serverless hosting for a limited time only please visit cloud.triggermesh.io.

Sources

The author used the following articles, documents, and books for inspiration and guidance:

Articles

• Examining the FaaS on K8S Market
• Knative or OpenFaas?
• IBM Embraces Knative to Drive Serverless Standardization
• OpenFaaS on OpenShift
• Serverless: who’s on call now?
• Serverless Computing / Functions as a Service mit Appuio und Snafu
• Getting started with OpenFaaS on minikube
• The evolution of serverless and FaaS: Knative brings change
• Knative, a FaaS anti-pattern?
• 15 Frameworks for Creating Serverless Apps

Projects

• FaaS on Kubernetes
• Fission Python Guestbook

Presentations

• Function-as-a-service on Kubernetes
• Best practices for serverless architectures

Books

• • Serveless Ops
  • Kubernetes for Serverless Applications
  • Guide to Serverless Technologies

People

• Michael Hausenblas

Slides

The slides of the presentation are available in (and can be downloaded from) SpeakerDeck.

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
IN A PREVIOUS BLOG POST WE TALKED ABOUT:
 Section 1 – Operators, Operator Framework, and Operators SDK, and in particular:

• in a general setting about Operators, Operator Framework, and Operators SDK.
• the Operators SDK emerging popularity on GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.
  

IN THIS BLOG POST WE WILL TALK ABOUT:
Section 2 – Supported Kubernetes Operator SDK workflows

• Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  
• We also discuss the pros and cons of using the various operators’ workflows.

Section 2 – Supported Operator SDK workflows

As discussed before, on GitHub, the Operator SDK is a very active project, with over 10 releases produced in less than a year. This means that the Operator SDK is a toolkit that is continuing evolving over the time (e.g., its code,  structure, and logic is changing). In particular, as reported in the main Github page of the operator SDK, the libraries and tools are labeled with “Project Status: pre-alpha”, and thus are “expected breaking changes to the API in the upcoming releases”.
The project started in April 2018 and we started monitoring it intensively from September 2018. We found out that the SDK provides three different workflows to develop operators based on Go, Ansible, or Helm.
These versions of the Operators SDK emerged between 2018 and 2019. Specifically, the first version of the operator was based on Go, and only from December 2018 it was provided a version based on Ansible.
Finally, in the beginning of 2019 (January), the operator workflow based on Helm was also released.
Thus, the SDK provides a workflow to develop operators based on Go, Ansible, or Helm.
The following workflow is for a new Go operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Define new resource APIs by adding Custom Resource Definitions (CRD)
      3. Define Controllers to watch and reconcile resources
      4. Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
      5. Use the SDK CLI to build and generate the operator deployment manifests

The following workflow is for a new Ansible operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Write the reconciling logic for your object using Ansible playbooks and roles
      3. Use the SDK CLI to build and generate the operator deployment manifests
      4. Optionally add additional CRD’s using the SDK CLI and repeat steps 2 and 3

The following workflow is for a new Helm operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Create a new (or add your existing) Helm chart for use by the operator’s reconciling logic
      3. Use the SDK CLI to build and generate the operator deployment manifests
      4. Optionally add additional CRD’s using the SDK CLI and repeat steps 2 and 3

Guidelines:
Command Line Interface: To learn more about the SDK CLI, see the SDK CLI Reference, or run operator-sdk [command] -h.
For a guide on Reconcilers, Clients, and interacting  with resource Events, see the Client API doc.
As it is possible to see from the following figure, there is not much difference among the various operator workflows.
However, the workflow that has reached more maturity and gives more control over the operator behavior is the one based on Go: 

Next article

Section 3 – Examples of Supported Operator SDK workflows

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

Introduction to Kubernetes Operators, Operator Framework, and Operators SDK

This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK: 

• Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
• Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.

Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK

a) Operators are Kubernetes applications
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your applications that run on Kubernetes. You can think of Operators as the “runtime that manages this type of application on Kubernetes“.
Thus, an Operator is a method of packaging, deploying and managing a Kubernetes application. Conceptually, an Operator takes human operational knowledge and encodes it into software that is more easily packaged and shared with consumers. We can think of an Operator as an extension of the software vendor’s engineering team that watches over your Kubernetes environment and uses its current state to make decisions in milliseconds. Operators follow a [maturity model] from basic to having specific logic for an application.
We’ve seen in the last years that Operators’ capabilities differ in sophistication depending on how much intelligence has been added into the implementation logic of the Operator itself. We’ve also learned that the creation of an Operator typically starts by automating an application’s installation and self-service provisioning capabilities, and then evolves to take on more complex automation, this depending on the specific use case. As result, advanced operators are, nowadays, designed to handle upgrades seamlessly, react to failures automatically, and not take shortcuts, like skipping a software backup process to save time.
b) Operator Framework
Operators are Kubernetes native applications that facilitate the management of complex stateful applications on top of Kubernetes, however writing such operators can be very difficult because of challenges such as using (i) low level APIs and (ii) a lack of modularity which leads to duplication, inconsistencies, and unexpected behaviors.
To address such issue, several tools are now being launched (e.g., the Operator Framework, Kooper, Metacontroller, etc.) as results of years of work and experience of the Red Hat, Kubernetes, and CoreOS open source communities in building Operators. Specifically, Red Hat and the Kubernetes open source community shared the Operator Framework — an open source toolkit designed to manage  operators in a more effective, automated, and scalable way.
The Operator Framework is an open source toolkit composed by several low-level APIs. We believe that the new Operator Framework represents the next big step for Kubernetes by using a baseline of leading practices to help lower the application development barrier on Kubernetes. The project delivers a software development kit (SDK) and the ability to manage app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise operator capabilities on any Kubernetes cluster.
The Operator Framework includes:

• • Operator SDK: Enables developers to build Operators based on their expertise without requiring knowledge of Kubernetes API complexities.
  • Operator Lifecycle Management: Oversees installation, updates, and management of the lifecycle of all of the Operators (and their associated services) running across a Kubernetes cluster. Once built, Operators need to be deployed on a Kubernetes cluster. The Operator Lifecycle Manager is the backplane that facilitates management of operators on a Kubernetes cluster. With it, administrators can control what Operators are available in what namespaces and who can interact with running Operators. They can also manage the overall lifecycle of Operators and their resources, such as triggering updates to both an Operator and its resources.
  • Operator Metering (joining in the coming months): Enables usage reporting for Operators that provide specialized services. In a future version, the Operator Framework will also include the ability to meter application usage – a Kubernetes first, which provides extensions for central IT teams to budget and for software vendors providing commercial software. Operator Metering is designed to tie into the cluster’s CPU and memory reporting, as well as calculate IaaS cost and customized metrics like licensing.

Simple, stateless applications can leverage the Lifecycle Management features of the Operator Framework without writing any code by using a generic Operator (for example, the Helm Operator). However, complex and stateful applications are where an Operator can shine. The cloud-like capabilities that are encoded into the Operator code can provide an advanced user experience, automating such features as updates, backups and scaling.
In the next subsection we discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.
c) Operators SDK popularity
The Operator-SDK is a toolkit,  recently built on top of the Operator Framework, that provides the tools to build, test and package Operators. Initially, the SDK facilitated the marriage of an application’s business logic (for example, how to scale, upgrade, or backup) with the Kubernetes API to execute those operations. However, over time, the SDK is evolving to allow engineers to make applications smarter and have the user experience of cloud services. As consequence, leading practices and code patterns that are shared across Operators are included in the SDK to help prevent reinventing the wheel.
From a developer perspective, the entry point is the Operator SDK, originating from CoreOS, which is offered as part of the Operator Framework that is, according to its self-description, “an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way”. The SDK specifically targets Go developers and applications, and even if support for other programming languages (e.g., Java, C, etc.) is currently lacking,  future plans for their integration are already in place.
In GitHub, the Operator SDK is becoming a very active project, which already gained a high visibility/popularity with:

• • over 200 developers that forked its contributions on GitHub, most of them playing/extending some simple examples of operators;
  • around 1000 commits done since the first release (5 Apr 2018).
  • over 10 releases produced in less than a year.
  • an increasing number of GitHub issues opened by other contributors in recent months
    • [some of them were opened also by us, see https://github.com/operator-framework/operator-sdk/issues/651 and https://github.com/operator-framework/operator-sdk/issues/927]
  • over 1200 stars it is one of the most popular/trending Go projects on Github https://github.com/trending/go?since=monthly

However, even if the project is becoming with the time more popular, its project Status is still “pre-alpha“, which means that “are expected breaking changes to the API in the upcoming releases“.
Thus, the Operator SDK toolkit requires still a bit more of maturity to be used in wider practical working scenarios. As researchers, we believe that this software development kit (SDK) will be widely adopted in future, as it will support the developers during the management of app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise operator capabilities on any Kubernetes cluster (see the following Figure, it highlights the overall view of envisioned Operator SDK support).

As follow we talk about the Operators SDK General Workflow.
d) Operators SDK General Workflow
The Operator-SDK is a toolkit that provides the tools to build, test and packageOperators, as shown in the following Figure.

Specifically, the following specific workflow is provided by the toolkit for supporting the writing, building, testing and packaging of a new Go operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Define new resource APIs by adding Custom Resource Definitions (CRD)
      3. Define Controllers to watch and reconcile resources
      4. Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
      5. Use the SDK CLI to build and generate the operator deployment manifests

In this context, the Operator SDK uses for its workflow the controller-runtime library, which makes the writing of operators easier by providing:

• • • High level APIs and abstractions to write the operational logic more intuitively.
    • Tools for scaffolding and code generation to bootstrap a new project fast.
    • Extensions to cover common operator use cases.

A simple example to create and deploy a simple operator with the SDK toolkit is provided  in the official operator SDK GitHub repository:
https://github.com/operator-framework/operator-sdk
The resulting automatically generated GO operator will present the following reference Structure:

In the next blog post we will talk about the Operators SDK current status, e.g., available versions and workflows.

Next article

Section 2 – Supported Kubernetes Operator SDK workflows

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

How to leverage Kubernetes operators using the Operator SDK framework

Kubernetes has become an omnipresent platform to host cloud-native applications. As a rather low-level platform, it is often made developer-friendly by wrapping it into higher-level platforms, such as OpenShift (OKD), and by turning it into a managed service platform, such as APPUiO, which can be deployed to any cloud infrastructure. Application engineers interact with Kubernetes mostly by authoring appropriate deployment descriptors and by pushing their code which triggers deployments. Due to ongoing feature additions, not so much is known about useful combinations of annotations on Kubernetes deployments (and other declaratively described objects), Kubernetes operators (a kind of hooks) and custom resources definitions.
In this blog post series, we share some of the experience we have gained while researching how to trigger actions upon certain updates to the descriptors, as a precursor to dynamic and autonomous feedback loops which can self-manage application deployments.
In particular, we provide access to the adapted original examples of operators generated with the Operator SDK toolkit, which deal with Kubernetes resources by combining annotations on Kubernetes deployments and Kubernetes operators concepts. The link to our operators examples are available on Github: https://github.com/appuio/operator-sdk-examples. In further blog posts we will describe some, discussing also how they could be extended for more advanced decision making. In particular, adapting the (Go) operators to work on different environments require to modify some important go files (e.g., pkg/controller/memcached/memcached_controller.go as shown in the following Figure).

IN FURTHER BLOG POSTS WE WILL TALK ABOUT:

Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK

• Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
• Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.

Section 2 – Supported Kubernetes Operator SDK workflows

• Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  
• We also discuss pros and cons of using the various operators workflows.
  

Section 3 – Examples of Supported Kubernetes Operator SDK workflows

• Here we provide examples about the three available alternative workflows to generate Operators provided by the Operator SDK APIs.
• We specifically focus on Go operators, as they are in our opinion the more stable available APIs.

Section 4 – Example(s) of Operator(s) Monitoring the Service with the usage of Prometheus (coming soon)

• Here we provide an example of an operator that communicates with Prometheus (currently used to monitor Kubernetes Clusters) for more advanced decision making (e.g., advanced monitoring of the service).

About the authors

These blog posts have been written by Dr. Josef Spillner and Dr. Sebastiano Panichella from ZHAW (Zurich University of Applied Sciences) School of Engineering. Thank you very much Josef and Sebastiano for sharing your know how with our readers!