This post comes late after my EuroPython 2019 talk on “Modern Continuous Delivery” in Basel. But there is no need to worry: Advice on writing software that outlasts hypes on modern computing has no hurry to appear on stage.

What a great time!

Deployment automation, cloud platforms, containerization, short iterations to develop and release software—we’ve progressed a lot. And finally it’s official: Kubernetes and OpenShift are the established platforms to help us do scaling and zero downtime deployments with just a few hundred lines of YAML. It’s a great time.
Can we finally put all our eggs into one basket? Identify the cloud platform that fits our needs, and jump on it? That could well backfire: Vendor lock-in is the new waterfall, it slows you down. In future you’ll want to jump over to the next better platform, in a matter of minutes. Not months.
So, how do we do that? What’s the right approach?

The definition of Continuous Delivery

What does modern software development look like?

A modern software development project does:

1. Immutable infrastructure (“containerization”)
2. Container orchestration (e.g. Kubernetes, Docker Swarm)
3. Version control and automation (CI/CD, “pipelines”)
4. Cloud-native applications (resilient apps that scale)

Nothing new. You’ve heard this before.

Vendor lock-in is the new waterfall

Software development has become complex. So complex that there are numerous services popping up almost daily that help us getting things done. Most notably, these are application delivery platforms nowadays (Amazon AWS, Microsoft Azure, Google Cloud — just to name a few). When we use offerings across several such providers, which is becoming increasingly popular, we speak of multi-cloud dependencies.

While all offerings are choices they typically entail a lock-in, because we don’t have standards and tooling yet that allow us to effortlessly switch from one solution provider to another. And when there are reasons to make a switch, lock-ins make it inherently difficult to move fast. This is a problem.

What can we do about vendor lock-in?

Software development is not about “using <technology>” or “using <platform>”. As engineers we must think in terms of “problem to solve” (requirements) and applying proper development practices (engineering). If we rely on platforms to solve problems for us we’re doing it wrong. Engineers must learn to follow principles of good software design, to write and maintain “clean code”.

One of those principles you learn as an engineer is to maintain boundaries between systems. In a modern software development project you’ll see four of such “layers of responsibility” that define boundaries:

1. Application
2. Development
3. Deployment
4. Automation

Think in terms of technologies and tools, or services and environments you use in each to understand why we have those.

1. The application layer is like how you did software development 15 years ago. It’s just concerned with getting things running locally. Add the 12factor app methodology and you get an application that is prepared for potential target environments.
2. The development layer is concerned with supporting the development in the application layer. Developer tooling that gets the project running with a single command, a test infrastructure setup and QA tooling, which should also be easy to handle.
3. The deployment layer reuses that part of the development layer that made the developer deploy the application locally, for development. It also houses configuration files used only in productive target environments and deployment configuration that describes the entire productive setup.
4. The automation layer is only concerned about automating the steps you would otherwise perform manually to deploy your application.

Clean separation and interfaces

Now we have layers. What do we gain?
Note how the layers use an interface to talk to the next layer above. This allows us to address them separately. In other words,

• When you want to use a different CI service (e.g. switch from Bitbucket Pipelines to GitLab CI) you only refactor the implementation in the automation layer (i.e. the CI configuration).
• When you want to change your container orchestration (e.g. switch from OpenShift to Kubernetes) you only refactor your implementation of the deployment layer (e.g. use Kustomize instead of OpenShift templates and Ingress instead of Routes, etc.). You may also have to change some of the deployment tooling in the automation layer as a consequence (e.g. use kubectl instead of oc), but it’s all cleanly separated.

This is the power of separation of concerns.
Also note that for this to work well you need to make your interfaces simple and stable.

• For development use a task runner to turn your common tasks into single commands or simple one-liners (in JavaScript you may use gulp, in Ruby rake, in Python tox).
• Don’t design for target environments in your application layer! Design with features in mind and only combine them to environment configurations in your deployment layer.
• Stick to standard procedures and established tooling (instead of proprietary or self-invented solutions) for the technologies you use. This is typically more stable as an interface, will save collaborators from learning how to work with your setup and make it unnecessary to add extensive instructions to the README.

The more you invest in this flexibility and clean separation the easier your task will be when the day comes to make a change.

Working code samples to your rescue

What is better than starting to investigate late and working on refactoring under time pressure? Knowing your possibilities and turning to working code samples when you need them!
That’s where the Painless Continuous Delivery Cookiecutter comes into play. It’s a project template generator covering many of the most popular combinations of public service offerings you may be working with. At VSHN AG we’re working on extending it—for your independence. Try it out, let us know if you find it useful, and contribute if you feel like!
For the start you may want to take a look at one of the generated setups on GitLab, the live demo for the EuroPython talk. Enjoy!

EuroPython 2019: Modern Continuous Delivery for Python Developers

Slides of this talk are available from SpeakerDeck.
This article has originally been published on Peter’s personal website.

use create

deploy step by step our operators.
For this SDK supported workflow, we provide:

a) a description of the generated Operator structure (there is one for each specific workflow);

b) a link to our example(s) of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.

c) descriptions on how to add (e.g., 3rd party) resources, different from the Core Kubernetes resource types, to your Operator 

d) a description of the main pros and cons of using such Operator structure (there is one for each specific workflow) and logic.

a) Description of the generated Operator structure (there is one for each specific workflow)

As follows the structure of a generated GO operator:

Operator scope

A namespace-scoped operator (the default) watches and manages resources in a single namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide. Namespace-scoped operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions. However, there are use cases where a cluster-scoped operator may make sense. For example, the cert-manager operator is often deployed with cluster-scoped permissions and watches so that it can manage issuing certificates for an entire cluster.

b) Example of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.

We will discuss two examples of GO Operator: 
1) Example 1: this operator just replicates itself under a certain project: 
2) Example 2: this operator allows to manipulate the number of pods in a project.

c) How to add 3rd Party Resources (i.e., different from the Core Kubernetes resource types) to your Operator

The operators manager supports the Core Kubernetes resource types as found in the client-go scheme (“register.go“) package and will also register the schemes of all custom resource types defined in your project under pkg/apis

To add a 3rd party resource to an operator, you must add it to the managers scheme. By creating an AddToSchememethod or reusing one you can easily add a resource to your scheme. An example shows that you define a function and then use the runtime package to create a SchemeBuilder.
Register with the managers scheme
Call the AddToScheme() function for your 3rd party resource and pass it the managers scheme via mgr.GetScheme().
Example:

“After adding new import paths to your operator project, run dep ensure in the root of your project directory to fulfill these dependencies”.

d) A description of the main pros and cons of using this Operator structure (there is one for each specific workflow) and logic;

PROS (+) AND CONS (-):

(+) it is easy to use and create a simple operator

The main program for the operator cmd/manager/main.go initializes and runs the manager. The manager will automatically register the scheme for all custom resources defined under pkg/apis/... and run all controllers under pkg/controller/....

The manager can restrict the namespace that all controllers will watch for resources:

mgr, err := manager.New(cfg, manager.Options{Namespace: namespace})

By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty:

mgr, err := manager.New(cfg, manager.Options{Namespace: ""})

(+) enough clear way to add 3rd Party Resources (i.e., different from the Core Kubernetes resource types) to your Operator

(-) it is still unstable/changing the APIs and the future versions of it could be not supported (we are observing this evolution and update the blog post accordingly)

To make the operator working, we had to open some issues in Github, that was closed and classified as a bug):

https://github.com/operator-framework/operator-sdk/issues/651

https://github.com/operator-framework/operator-sdk/issues/927

https://github.com/operator-framework/operator-sdk/issues/1053

(-) The workflow to create the GO operators appear a bit more complex/unclear (steps in red) that the one generated with Ansible and Helm (which require fewer steps):

1. Create a new operator project using the SDK Command Line Interface (CLI)
2. Define new resource APIs by adding Custom Resource Definitions (CRD)
3. Define controllers to watch and reconcile resources
4. Write the reconciling logic for your controller using the SDK and controller-runtime APIs
5. Use the SDK CLI to build and generate the operator deployment manifests

Back to overview

Back to overview Section 3 – Examples of Supported Kubernetes Operator SDK workflows.

2022–03–09: There’s a newer version of this article!

(This blog post is the transcription of the presentation given at the CNC Switzerland meetup, May 9th 2019. Slides are available at the end of this page and impressions of the meetup can be found here.)
Serverless is one of those hot topics that, as many others in our industry, looks a bit like a good old idea recycled and brought back to fashion. Yet Serverless (or “Function as a Service”) looks like a natural evolution to a movement that started more than a decade ago, when Heroku and Google App Engine came under the spotlight.
As says Martin Fowler himself, Serverless and FaaS can be defined as follows:

FaaS is about running backend code without managing your own server systems or your own long-lived server applications.

At VSHN we have been interested in the subject for years now, and even worked in a FaaS project for a customer two years ago.
But now that Kubernetes and Knative are emerging as the true “Golden Standards” of hosted cloud operations, we see the emergence of FaaS solutions running directly on top of them. And developers have been fast creating lots of solutions that somehow appear to occupy the same space. How do they compare? Which one to choose?
To answer these questions, we are going to give a brief description of the following frameworks, outlining some of their characteristics and relative strengths:

1. OpenFaaS
2. Fn Project
3. Fission
4. OpenWhisk
5. Kubeless
6. TriggerMesh

The order of the frameworks is not trivial; they are roughly ordered following their “level of abstraction” on top of Docker and Kubernetes. For each of these projects, we are going to provide the following information:

1. Project details;
2. Three demos, all of them recorded with asciinema using minikube;
3. Available triggers;
4. Supported programming languages.

1. OpenFaaS

OpenFaaS is the project with the most stars (14’000) on Github of all those in this article. It is mostly written in Go, featuring around 100 contributors, and the latest available version at the time of this writing is 0.13.0 (April 4th, 2019).
It is an independent project, funded through donations (Patreon, OpenCollective, and PayPal.)
From a developer experience perspective, it is a project quite complex to setup and use. It is based on Docker, which means that functions are actually packaged as containers, to be pushed to a repository, and to be built using a local Docker installation in the developer workstation. OpenFaaS manages the Dockerfile for the developers automatically, though.
OpenFaas has a “template store” with several available programming languages. It also provides developers with a command-line utility, called faas-cli, itself talking to a REST API documented using Swagger. Finally, there is a Serverless Framework plugin for those who need it.
The following asciicast shows a very simple interaction with OpenFaas. First we create a Python function, which we customize a bit, and then we deploy and call it from the command line; both directly and also using curl:

OpenFaaS functions can be called through the following triggers:

• HTTP
• Event Connectors: Kafka, Amazon SNS, Minio, CloudEvents, IFTTT, Redis, RabbitMQ…

Finally, developers can use the following programming languages with OpenFaaS:

• C#, F# (.NET Core)
• Go
• JavaScript (Node.js)
• Java (JVM)
• PHP 7
• Python / Python 3
• Ruby

2. Fn Project

The Fn Project has been started and is currently funded by Oracle, who uses a fork to power its own Oracle Functions product.
Just like OpenFaaS, it is hosted in Github and written in Go. The project has around 4000 stars and 80 contributors, and the latest version at the time of this writing is 0.3.703 (May 6th, 2019).
From a technical point of view, Fn can use any Docker container as a function, and it can run anywhere: in public, private, and hybrid clouds.
Fn has two major concepts: – Functions (defined in YAML) – Applications: groups of functions, which can be deployed all at once.
For developers, it offers a command-line tool called fn and a Serverless Framework plugin.
Fn functions can be triggered with HTTP calls, and can be developed using the following languages:

• Go
• JavaScript (Node.js)
• Java (JVM)
• Python
• Ruby
• C# (community supported)

The Fn marketing material further states that it “supports all languages”.

3. Fission

Fission is an open source, Kubernetes-native Serverless Framework. It allows functions to be deployed and executed instantly, mapping them to HTTP requests.
Its Github project is mostly written in Go, features 4300 stars and around 80 contributors at the time of this writing. Its latest available version is 1.2.0 (May 3rd, 2019). It was started and is currently maintained by Platform9.
Fission does not need Dockerfiles or Docker registries; it is based on the notion of environments. Functions are injected into those environments, which are a pool of containers with language runtimes, where functions are loaded and launched on demand.
Fission keeps in memory a set of images containing the runtimes where the functions will be run, injecting them and running them immediately when invoked. In this sense, it is similar to how AWS Lambda works.
For developers, it features a command line tool (fission) and a Serverless Framework plugin. They do not need to have a local Docker environment to build their functions into.
The following asciicast shows the basic operations required to create, deploy and call a function:

Currently, Fission supports following types of triggers:

• HTTP
• Time
• Message Queue
• Kubernetes Watch

Only the following programming languages can be used to create functions in Fission; the project is quite young and the list will probably grow in the future:

• Go
• Python
• JavaScript (Node.js)
• Java (JVM)

In our tests, using minikube as a support, Fission appears easy to use, but at the same time very fragile (in spite of what its version number might suggest.) Removing and re-creating environments and functions led to many problems, and the project is too young to have more than 5 (unanswered) questions on Stack Overflow. In short, a promising but yet rather immature product.

4. OpenWhisk

OpenWhisk is the behemoth in the room. This open source project was created by IBM and is currently managed by the Apache Foundation. This project is the most “corporate” ones of those described in this blog post. It is written in Scala, it features around 4000 stars in Github and has around 150 contributors. The latest available version at the time of this writing is 0.9.0 (October 31st, 2018.)
This framework has the following features:

• Very “corporate” in design and functionality;
• Secure by default;
• Forked by Adobe and other big corporations;

For developers, it features a command-line tool (wsk) and a Serverless Framework plugin.
OpenWhisk functions can be triggered by the following mechanisms:

• Message Queues
• Databases
• Document Stores
• Website or Web Apps
• Service APIs
• IoT Frameworks…

OpenWhisk function can be created using the following programming languages:

• C#, F# (.NET Core)
• JavaScript (Node.js)
• Swift
• Java, Scala (JVM)
• Go
• Python
• PHP
• Ruby
• Ballerina
• Through Docker Actions: Rust, Haskell…

The installation on minikube was the most complex and difficult of all the frameworks considered in this document. The tools have changed a lot in the last two years and resources online might be outdated. But in spite of those factors, this framework stands out by the quantity, breadth, and depth of the documentation, as well as by the number of integration and supported languages.

5. Kubeless

Kubeless is a promising framework created and maintained by Bitnami. It is an open source project on Github written in Go, with around 4600 stars and 80 contributors. At the time of this writing, its latest version is 1.0.3 (March 14th, 2019.)
In our tests it was the one offering the best developer experience. Very simple to install and use, it offers a command-line tool (kubeless) that is very similar to the AWS Lambda CLI. This is no coincidence, as the whole aim of the project is to provide an experience very close to that of AWS Lambda, Azure Functions or Google Cloud Functions.
For DevOps teams, Kubeless provides Prometheus monitoring of functions calls and latency, and a Serverless Framework plugin.
The following asciicast shows the basic interaction to create, deploy, and test a function using Kubeless:

Kubeless functions can be triggered through the following mechanisms:

• HTTP
• Cronjob
• PubSub mechanisms
  • Kafka
  • NATS messaging
• Data stream events
  • AWS Kinesis

These functions can be developed using the following languages and runtimes:

• Go
• Python
• JavaScript (Node.js)
• Java (JVM)
• Ruby
• C#, F# (.NET Core)
• Ballerina
• Custom runtimes possible

All frameworks herewith considered, Kubeless offered the most flawless experience of all. The documentation was solid and easy to follow, and there are plenty of online resources to guide developers into building applications using this platform.

6. TriggerMesh

TriggerMesh is the newest entry in the world of FaaS, and it will represent a major shift in the way serverless applications are deployed and executed. Founded ex-Kubeless engineers, it builds upon Kubernetes & Knative, providing features yet unseen in the serverless arena.
TriggerMesh functions can be triggered through cross-cloud triggers from AWS to functions on Knative:

• Code Commit
• Cognito
• DynamoDB
• Kinesis
• S3
• SNS
• SQS
• …

TriggerMesh has announced the following programming languages as options:

• Go
• JavaScript (Node.js)
• Python
• Ruby

Together with VSHN, TriggerMesh released the TriggerMesh Operator for OpenShift 4.0. OKD 4.0 was recently announced by Red Hat to bring additional automation to Kubernetes applications. The operator allows OpenShift users to install the TriggerMesh management platform and benefit from its integration with Knative, to power serverless workloads across multiple clouds.. TriggerMesh also allows CI/CD of serverless functions, as well as access to multi-cloud event sources, like Azure and AWS.

Comparison

The following chart summarizes some of the ideas of this article, and has been adapted from the “Examining the FaaS on K8S Market” article on the Cisco Blog.

Popularity

The following tweet provides interesting information about the current state of the FaaS-on-Kubernetes market:

32% of those using or planning to use a serverless platform have a Kubernetes-based installable platform in their plans, with Kubeless often being cited alongside other options. https://t.co/RYEwShkrLb pic.twitter.com/lqWWyrQu8B

— Lawrence Hecht (@LawrenceHecht) October 25, 2018

Conclusion

I hope this summary will be useful to you! Having tested all of these options, here at VSHN we will be focusing our efforts in the TriggerMesh platform, which will provide a much more solid developer experience and an unprecedented level of flexibility. We believe that this is the next generation of serverless platforms and we cannot wait to bring its power to our customers.
TriggerMesh will be launching their TriggerMesh Cloud service in the near future which will allow users to host serverless functions and consume events from many cloud sources.  To join the TriggerMesh Early Adopters program and get access for free serverless hosting for a limited time only please visit cloud.triggermesh.io.

Sources

The author used the following articles, documents, and books for inspiration and guidance:

Articles

• Examining the FaaS on K8S Market
• Knative or OpenFaas?
• IBM Embraces Knative to Drive Serverless Standardization
• OpenFaaS on OpenShift
• Serverless: who’s on call now?
• Serverless Computing / Functions as a Service mit Appuio und Snafu
• Getting started with OpenFaaS on minikube
• The evolution of serverless and FaaS: Knative brings change
• Knative, a FaaS anti-pattern?
• 15 Frameworks for Creating Serverless Apps

Projects

• FaaS on Kubernetes
• Fission Python Guestbook

Presentations

• Function-as-a-service on Kubernetes
• Best practices for serverless architectures

Books

• • Serveless Ops
  • Kubernetes for Serverless Applications
  • Guide to Serverless Technologies

People

• Michael Hausenblas

Slides

The slides of the presentation are available in (and can be downloaded from) SpeakerDeck.

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
IN A PREVIOUS BLOG POST WE TALKED ABOUT:
 Section 1 – Operators, Operator Framework, and Operators SDK, and in particular:

• in a general setting about Operators, Operator Framework, and Operators SDK.
• the Operators SDK emerging popularity on GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.
  

IN THIS BLOG POST WE WILL TALK ABOUT:
Section 2 – Supported Kubernetes Operator SDK workflows

• Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  
• We also discuss the pros and cons of using the various operators’ workflows.

Section 2 – Supported Operator SDK workflows

As discussed before, on GitHub, the Operator SDK is a very active project, with over 10 releases produced in less than a year. This means that the Operator SDK is a toolkit that is continuing evolving over the time (e.g., its code,  structure, and logic is changing). In particular, as reported in the main Github page of the operator SDK, the libraries and tools are labeled with “Project Status: pre-alpha”, and thus are “expected breaking changes to the API in the upcoming releases”.
The project started in April 2018 and we started monitoring it intensively from September 2018. We found out that the SDK provides three different workflows to develop operators based on Go, Ansible, or Helm.
These versions of the Operators SDK emerged between 2018 and 2019. Specifically, the first version of the operator was based on Go, and only from December 2018 it was provided a version based on Ansible.
Finally, in the beginning of 2019 (January), the operator workflow based on Helm was also released.
Thus, the SDK provides a workflow to develop operators based on Go, Ansible, or Helm.
The following workflow is for a new Go operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Define new resource APIs by adding Custom Resource Definitions (CRD)
      3. Define Controllers to watch and reconcile resources
      4. Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
      5. Use the SDK CLI to build and generate the operator deployment manifests

The following workflow is for a new Ansible operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Write the reconciling logic for your object using Ansible playbooks and roles
      3. Use the SDK CLI to build and generate the operator deployment manifests
      4. Optionally add additional CRD’s using the SDK CLI and repeat steps 2 and 3

The following workflow is for a new Helm operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Create a new (or add your existing) Helm chart for use by the operator’s reconciling logic
      3. Use the SDK CLI to build and generate the operator deployment manifests
      4. Optionally add additional CRD’s using the SDK CLI and repeat steps 2 and 3

Guidelines:
Command Line Interface: To learn more about the SDK CLI, see the SDK CLI Reference, or run operator-sdk [command] -h.
For a guide on Reconcilers, Clients, and interacting  with resource Events, see the Client API doc.
As it is possible to see from the following figure, there is not much difference among the various operator workflows.
However, the workflow that has reached more maturity and gives more control over the operator behavior is the one based on Go: 

Next article

Section 3 – Examples of Supported Operator SDK workflows

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

Introduction to Kubernetes Operators, Operator Framework, and Operators SDK

This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK: 

• Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
• Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.

Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK

a) Operators are Kubernetes applications
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your applications that run on Kubernetes. You can think of Operators as the “runtime that manages this type of application on Kubernetes“.
Thus, an Operator is a method of packaging, deploying and managing a Kubernetes application. Conceptually, an Operator takes human operational knowledge and encodes it into software that is more easily packaged and shared with consumers. We can think of an Operator as an extension of the software vendor’s engineering team that watches over your Kubernetes environment and uses its current state to make decisions in milliseconds. Operators follow a [maturity model] from basic to having specific logic for an application.
We’ve seen in the last years that Operators’ capabilities differ in sophistication depending on how much intelligence has been added into the implementation logic of the Operator itself. We’ve also learned that the creation of an Operator typically starts by automating an application’s installation and self-service provisioning capabilities, and then evolves to take on more complex automation, this depending on the specific use case. As result, advanced operators are, nowadays, designed to handle upgrades seamlessly, react to failures automatically, and not take shortcuts, like skipping a software backup process to save time.
b) Operator Framework
Operators are Kubernetes native applications that facilitate the management of complex stateful applications on top of Kubernetes, however writing such operators can be very difficult because of challenges such as using (i) low level APIs and (ii) a lack of modularity which leads to duplication, inconsistencies, and unexpected behaviors.
To address such issue, several tools are now being launched (e.g., the Operator Framework, Kooper, Metacontroller, etc.) as results of years of work and experience of the Red Hat, Kubernetes, and CoreOS open source communities in building Operators. Specifically, Red Hat and the Kubernetes open source community shared the Operator Framework — an open source toolkit designed to manage  operators in a more effective, automated, and scalable way.
The Operator Framework is an open source toolkit composed by several low-level APIs. We believe that the new Operator Framework represents the next big step for Kubernetes by using a baseline of leading practices to help lower the application development barrier on Kubernetes. The project delivers a software development kit (SDK) and the ability to manage app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise operator capabilities on any Kubernetes cluster.
The Operator Framework includes:

• • Operator SDK: Enables developers to build Operators based on their expertise without requiring knowledge of Kubernetes API complexities.
  • Operator Lifecycle Management: Oversees installation, updates, and management of the lifecycle of all of the Operators (and their associated services) running across a Kubernetes cluster. Once built, Operators need to be deployed on a Kubernetes cluster. The Operator Lifecycle Manager is the backplane that facilitates management of operators on a Kubernetes cluster. With it, administrators can control what Operators are available in what namespaces and who can interact with running Operators. They can also manage the overall lifecycle of Operators and their resources, such as triggering updates to both an Operator and its resources.
  • Operator Metering (joining in the coming months): Enables usage reporting for Operators that provide specialized services. In a future version, the Operator Framework will also include the ability to meter application usage – a Kubernetes first, which provides extensions for central IT teams to budget and for software vendors providing commercial software. Operator Metering is designed to tie into the cluster’s CPU and memory reporting, as well as calculate IaaS cost and customized metrics like licensing.

Simple, stateless applications can leverage the Lifecycle Management features of the Operator Framework without writing any code by using a generic Operator (for example, the Helm Operator). However, complex and stateful applications are where an Operator can shine. The cloud-like capabilities that are encoded into the Operator code can provide an advanced user experience, automating such features as updates, backups and scaling.
In the next subsection we discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.
c) Operators SDK popularity
The Operator-SDK is a toolkit,  recently built on top of the Operator Framework, that provides the tools to build, test and package Operators. Initially, the SDK facilitated the marriage of an application’s business logic (for example, how to scale, upgrade, or backup) with the Kubernetes API to execute those operations. However, over time, the SDK is evolving to allow engineers to make applications smarter and have the user experience of cloud services. As consequence, leading practices and code patterns that are shared across Operators are included in the SDK to help prevent reinventing the wheel.
From a developer perspective, the entry point is the Operator SDK, originating from CoreOS, which is offered as part of the Operator Framework that is, according to its self-description, “an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way”. The SDK specifically targets Go developers and applications, and even if support for other programming languages (e.g., Java, C, etc.) is currently lacking,  future plans for their integration are already in place.
In GitHub, the Operator SDK is becoming a very active project, which already gained a high visibility/popularity with:

• • over 200 developers that forked its contributions on GitHub, most of them playing/extending some simple examples of operators;
  • around 1000 commits done since the first release (5 Apr 2018).
  • over 10 releases produced in less than a year.
  • an increasing number of GitHub issues opened by other contributors in recent months
    • [some of them were opened also by us, see https://github.com/operator-framework/operator-sdk/issues/651 and https://github.com/operator-framework/operator-sdk/issues/927]
  • over 1200 stars it is one of the most popular/trending Go projects on Github https://github.com/trending/go?since=monthly

However, even if the project is becoming with the time more popular, its project Status is still “pre-alpha“, which means that “are expected breaking changes to the API in the upcoming releases“.
Thus, the Operator SDK toolkit requires still a bit more of maturity to be used in wider practical working scenarios. As researchers, we believe that this software development kit (SDK) will be widely adopted in future, as it will support the developers during the management of app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise operator capabilities on any Kubernetes cluster (see the following Figure, it highlights the overall view of envisioned Operator SDK support).

As follow we talk about the Operators SDK General Workflow.
d) Operators SDK General Workflow
The Operator-SDK is a toolkit that provides the tools to build, test and packageOperators, as shown in the following Figure.

Specifically, the following specific workflow is provided by the toolkit for supporting the writing, building, testing and packaging of a new Go operator:

1. 1. 1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Define new resource APIs by adding Custom Resource Definitions (CRD)
      3. Define Controllers to watch and reconcile resources
      4. Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
      5. Use the SDK CLI to build and generate the operator deployment manifests

In this context, the Operator SDK uses for its workflow the controller-runtime library, which makes the writing of operators easier by providing:

• • • High level APIs and abstractions to write the operational logic more intuitively.
    • Tools for scaffolding and code generation to bootstrap a new project fast.
    • Extensions to cover common operator use cases.

A simple example to create and deploy a simple operator with the SDK toolkit is provided  in the official operator SDK GitHub repository:
https://github.com/operator-framework/operator-sdk
The resulting automatically generated GO operator will present the following reference Structure:

In the next blog post we will talk about the Operators SDK current status, e.g., available versions and workflows.

Next article

Section 2 – Supported Kubernetes Operator SDK workflows

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

How to leverage Kubernetes operators using the Operator SDK framework

Kubernetes has become an omnipresent platform to host cloud-native applications. As a rather low-level platform, it is often made developer-friendly by wrapping it into higher-level platforms, such as OpenShift (OKD), and by turning it into a managed service platform, such as APPUiO, which can be deployed to any cloud infrastructure. Application engineers interact with Kubernetes mostly by authoring appropriate deployment descriptors and by pushing their code which triggers deployments. Due to ongoing feature additions, not so much is known about useful combinations of annotations on Kubernetes deployments (and other declaratively described objects), Kubernetes operators (a kind of hooks) and custom resources definitions.
In this blog post series, we share some of the experience we have gained while researching how to trigger actions upon certain updates to the descriptors, as a precursor to dynamic and autonomous feedback loops which can self-manage application deployments.
In particular, we provide access to the adapted original examples of operators generated with the Operator SDK toolkit, which deal with Kubernetes resources by combining annotations on Kubernetes deployments and Kubernetes operators concepts. The link to our operators examples are available on Github: https://github.com/appuio/operator-sdk-examples. In further blog posts we will describe some, discussing also how they could be extended for more advanced decision making. In particular, adapting the (Go) operators to work on different environments require to modify some important go files (e.g., pkg/controller/memcached/memcached_controller.go as shown in the following Figure).

IN FURTHER BLOG POSTS WE WILL TALK ABOUT:

Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK

• Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
• Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.

Section 2 – Supported Kubernetes Operator SDK workflows

• Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  
• We also discuss pros and cons of using the various operators workflows.
  

Section 3 – Examples of Supported Kubernetes Operator SDK workflows

• Here we provide examples about the three available alternative workflows to generate Operators provided by the Operator SDK APIs.
• We specifically focus on Go operators, as they are in our opinion the more stable available APIs.

Section 4 – Example(s) of Operator(s) Monitoring the Service with the usage of Prometheus (coming soon)

• Here we provide an example of an operator that communicates with Prometheus (currently used to monitor Kubernetes Clusters) for more advanced decision making (e.g., advanced monitoring of the service).

About the authors

These blog posts have been written by Dr. Josef Spillner and Dr. Sebastiano Panichella from ZHAW (Zurich University of Applied Sciences) School of Engineering. Thank you very much Josef and Sebastiano for sharing your know how with our readers!

Docker Swarm with encrypted node-to-node traffic

VSHNeer Elia set up a Docker Swarm Cluster with full traffic encryption inside the cluster (crosspost to his private blog):
I have set up a Docker Swarm cluster on the new Hetzner Cloud. First things first – the Hetzner Cloud is really amazing: Super simple, super cheap and performs as expected. It is not a bloated cloud provider that has 100x services and features that you can use for your servers, this keeps the costs and complexity down – I am really a big fan of it.
To the topic: Because the feature-set is simple, the Hetzner Cloud does not provide private networking (yet!). With only public IP addresses, we need to secure the overlay traffic between our docker containers!
 

The Problem

Per default, Docker Swarm encrypts the traffic between the managers, so we won’t have any issues there. However, this default setting is not set for container-to-container traffic. Any traffic that uses the overlay network is not encrypted by default because most of the time people do have private network setups with a floating IP as access point to the cluster. Docker assumes that the private network is secure and thus can spare some resources for other tasks (Which for example is not the case at DigitalOcean, so I recommend using overlay encryption anyway!).
Now, let’s assume we have the following stack:

version: '3'
services:
  db:
    networks:
      - internal
    image: mysql:5.7
    environment:
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: securepw
  wordpress:
    networks:
      - traefik_public
      - internal
    depends_on:
      - db
    image: wordpress:latest
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: securepw
    deploy:
      labels:
        - traefik.frontend.rule=Host:blog.example.com
        - traefik.docker.network=traefik_public
        - traefik.port=80
networks:
  traefik_public:
    external: true
  internal: 

This is a WordPress Stack that creates the WP site and a MySQL database. Those two networks are defined:

• internal
• traefik_public

The interal  overlay network is used for the communication between the WP container and the database. This network is not reachable by the outside. traefik_public  is the network used for the reverse proxy. It is only attached to the WP container as this is the only public facing side of this setup.
The problem here is: Without a secured private network, traffic running through the network internal will go out to another worker (Docker Node), and this fully plain visible. Any password/authentication/<SENSITIVE_DATA> is sent plain text between the docker containers, should they be on two different nodes.
Most of the docker images are not made for public access in their simple utilization and that’s why most keep it as simple as possible, no complicated encryption. You can of course build your own image to enable application side encryption.

The Solution

Docker has a solution for this issue. You can simply enable encryption of the overlay network. Sadly I really didn’t see much discussion about this hence why I thought a blog post about this particular issue might be useful.
The encryption of the network needs to happen during it’s creation, you cannot encrypt a network once it has been setup already. To enable encryption we need to add a flag to the network definition:

networks:
   traefik_public:
     external: true
   internal:
     driver_opts:
       encrypted: ""

The network traefik_public  is of course also encrypted as you don’t want the reverse proxying in plain text.
The option encrypted  will create a IPSec tunnel between all the workers where tasks are scheduled for a stack. This will fully encrypt all the traffic of the overlay network internal and thus allow sensitive data being shared between the database and WordPress.
You can read the official documentation about this here.
 

Final thoughts

The information in regards of encryption is very “hidden” and mostly ignored in my opinion. People want to simply deploy applications with docker without thinking about the infrastructure under it and thus running into the problem like plain text traffic on overlay networks.
I hope to make people more aware of encryption with this blog post.
If you have any questions, please let me know below!
 
 

Amazee.io set out to make hosting of Drupal websites easier and more flexible. We always strive to stay at the cutting edge of technology. So it does not come to a surprise that we started adopting a container based approach to hosting. We’ll talk about what containers are and why we believe it’s important to go one step further and open source our technology we use to host websites.
 

How Does Hosting Without Containers Look Like?

First and foremost, it’s less flexible. If your project has special needs (i.e. caching with redis or a decoupled frontend with node.js)that your provider does not support, you might end up in a tricky situation.
You might have to either add another provider that supports your technology or you find another provider which hosts your project on one platform.
Adding another provider to the mix ads more complexity and less streamlined support structures if you ever run into issues. amazee.io’s new system, Lagoon, gives developers the flexibility to create their own service architecture, supporting everything from a simple Drupal website to a high traffic website with a decoupled frontend and server side rendering. We do this with through the magic of containers.
 

So, What are containers?

You can think about hosting companies as shipping companies and the servers we have are ships that hold the code from individual projects.
Before the standard shipping container existed it was quite an endeavour to load and unload a ships cargo because everything had custom dimensions – so stacking them in an efficient manner was really hard.As soon as a standard was established – in this case the shipping container – everything got much easier because they were stackable and had all the same form factor.
This allegory shows that the age of custom servers is over. Sure you can run a lot of things on a server but it will not scale past a certain degree. Containerizing your applications makes it much easier to move between platforms because you follow a certain standard.
You can see why hosting companies solve this by only hosting one kind of thing. Drupal sites, for example. But the beauty of containers is that we can host all sorts of projects and still make everything efficient and optimized for our ships. We do this with the help of container technology.
 

Open source

We’re entering the third decade of open source as the Open Source Initiative celebrated its 20th anniversary beginning of February 2018. From the very beginning, amazee.io was committed to  basing our work open source software and technologies as much as possible.  This enabled us to continue the great work started byother projects and bring innovative solutions a step further our solutions a step further.
Every version of our hosting system is more flexible than the last, with more  functionality for our clients. We always saw it a bit troubling that we host open source CMS like Drupal or WordPress on proprietary solutions where you don’t know what happens behind the curtains. ast year we decided to go one step further and we have open sourced our Docker in Production hosting System. It’s there for everyone to look at it, understand it, tinker with it. We welcome feedback, contributions, and high fives at all times.
You can find more about our Project on our Website or if you want to talk about hosting we’re always there for you on our Slack

Update: demo projects published

This article is now outdated! We have prepared a new set of demo projects with full source code, public CI/CD pipelines and Kubernetes integration. Please visit https://gitlab.com/appuio and let us know how you like them! Issues and Merge Requests welcome!

Overview

“Deploy early and often” is one of the core principles of agile development. One solution to this problem are automated build and deploy pipelines.
This project uses such a pipeline, based on GitLab CI and OpenShift. A sample PHP application is built and deployed to APPUiO.
Three environments in form of OpenShift projects are used: test, qual and production. Those are the most commonly used setups but can be configured and extended. The idea is to build and deploy every commit from the master branch to the test environment automatically. If a git tag is created, the previously built docker image is tagged with the git tag name and automatically deployed to qual. The deployment to production is a manual step and can be invoked on GitLab.

Pipelines

There are two pipelines setup in .gitlab-ci.yml: One is run on every commit to the master branch and consists of the stages lint, build and deploy-test. The lint stage starts a docker container in which the PHP code is verified for correct syntax. The build stage starts a new OpenShift build (S2I), waits for its completion and shows the log output of the build. The last stage automatically deploys the built image to the test environment.
The second pipeline is started upon creation of a git tag. It consists of the stages release, deploy-qual and deploy-production. The first stage tags the docker image that was built for the respective git commit with the name of the created git tag. The deploy-qual stage deploys the tagged docker image automatically to the qualenvironment. The stage deploy-production needs to be started manually and deploys to the production environment.

APPUiO

The application and the build is run on APPUiO which allows us to use the concepts and solutions OpenShift provides to work with builds and deployments. The directory os contains the yaml-files which describe all the used OpenShift resources. These files live together with the code in the same git repository, which guarantees that the same version of the code is always built and deployed using same version of the OpenShift resources.

Build

The build consists of an OpenShift BuildConfig which uses a PHP source-to-image builder and builds a docker image containing the application. The image is tagged with the git commit hash and stored on the OpenShift docker registry.

Deployment

A DeploymentConfig is used to run the application. With the help of rolling deployments, zero downtime deployments are possible and the end user won’t experience any service disruption during updates. The liveness and readiness probes enable OpenShift to see if the container is running properly (liveness) and able to accept incoming traffic (readiness). Additionally an OpenShift service and route are set up to route traffic from outside the cluster to the application. The URL is specific for each environment and can be opened from within GitLab.

Demo

To see the pipeline in action you need three things:

1. Your own fork of the repository on GitLab with GitLab CI enabled
2. A project on APPUiO for each stage: test, qual and production
3. A ServiceAccount with edit membership in all projects

Step by step

1. Set up Kubernetes/OpenShift integration in GitLab: https://docs.gitlab.com/ce/user/project/clusters/index.html#adding-an-existing-kubernetes-cluster
2. Configure .gitlab-ci.yaml with your APPUiO projects
3. Commit & push the changes
4. Watch the automated pipeline build the application and deploy it to the test project
5. Create and push a git tag
6. Watch the automated pipeline create image tags and deploy it to the qual project
7. Manually run a deployment to the production environemnt by visiting the pipeline on GitLab and click “play” on the deploy:production job

Outlook

To further extend this setup, feature branches could be built and deployed to OpenShift. With this in place every merge request is built and a live version of the application is available to test the introduced changes. This ensures that only buildable changes are merged into the master branch and changes can be easily reviewed.
To learn more about automated builds and deployments with GitLab and APPUiO, read the docs and create a project on APPUiO Public.