Tech

Go operator

18. Jun 2019

use create

deploy step by step our operators.
For this SDK supported workflow, we provide:

a) a description of the generated Operator structure (there is one for each specific workflow);

b) a link to our example(s) of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.

c) descriptions on how to add (e.g., 3rd party) resources, different from the Core Kubernetes resource types, to your Operator 

d) a description of the main pros and cons of using such Operator structure (there is one for each specific workflow) and logic.

a) Description of the generated Operator structure (there is one for each specific workflow)

As follows the structure of a generated GO operator:

File/Folders
Purpose
cmd
Contains manager/main.go which is the main program of the operator. This instantiates a new manager which registers all custom resource definitions under pkg/apis/... and starts all controllers under pkg/controllers/... .
pkg/apis
Contains the directory tree that defines the APIs of the Custom Resource Definitions (CRD). Users are expected to edit the pkg/apis/<group>/<version>/<kind>_types.go files to define the API for each resource type and import these packages in their controllers to watch for these resource types.
pkg/controller
This pkg contains the controller implementations. Users are expected to edit the pkg/controller/<kind>/<kind>_controller.go to define the controller’s reconcile logic for handling a resource type of the specified kind.
build
Contains the Dockerfile and build scripts used to build the operator.
deploy
Contains various YAML manifests for registering CRDs, setting up RBAC, and deploying the operator as a Deployment.
Gopkg.toml Gopkg.lock
The Go Dep manifests that describe the external dependencies of this operator.
vendor
The golang vendor folder that contains the local copies of the external dependencies that satisfy the imports of this project. Go Dep manages the vendor directly.

Operator scope

A namespace-scoped operator (the default) watches and manages resources in a single namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide. Namespace-scoped operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions. However, there are use cases where a cluster-scoped operator may make sense. For example, the cert-manager operator is often deployed with cluster-scoped permissions and watches so that it can manage issuing certificates for an entire cluster.

b) Example of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.

We will discuss two examples of GO Operator: 
1) Example 1: this operator just replicates itself under a certain project: 
2) Example 2: this operator allows to manipulate the number of pods in a project.

c) How to add 3rd Party Resources (i.e., different from the Core Kubernetes resource types) to your Operator

The operators manager supports the Core Kubernetes resource types as found in the client-go scheme (“register.go“) package and will also register the schemes of all custom resource types defined in your project under pkg/apis

import (
  "github.com/example-inc/memcached-operator-long/pkg/apis"
  ...
)
// Setup Scheme for all resources
if err := apis.AddToScheme(mgr.GetScheme()); err != nil {
  log.Error(err, "")
  os.Exit(1)
}

To add a 3rd party resource to an operator, you must add it to the managers scheme. By creating an AddToSchememethod or reusing one you can easily add a resource to your scheme. An example shows that you define a function and then use the runtime package to create a SchemeBuilder.
Register with the managers scheme
Call the AddToScheme() function for your 3rd party resource and pass it the managers scheme via mgr.GetScheme().
Example:

import (
    ....
    routev1 "github.com/openshift/api/route/v1"
)
func main() {
    ....
    if err := routev1.AddToScheme(mgr.GetScheme()); err != nil {
      log.Error(err, "")
      os.Exit(1)
    }
    ....
}

After adding new import paths to your operator project, run dep ensure in the root of your project directory to fulfill these dependencies”.

$ dep ensure

d) A description of the main pros and cons of using this Operator structure (there is one for each specific workflow) and logic;

PROS (+) AND CONS (-):

(+) it is easy to use and create a simple operator

The main program for the operator cmd/manager/main.go initializes and runs the manager. The manager will automatically register the scheme for all custom resources defined under pkg/apis/... and run all controllers under pkg/controller/....

The manager can restrict the namespace that all controllers will watch for resources:

mgr, err := manager.New(cfg, manager.Options{Namespace: namespace})

By default this will be the namespace that the operator is running in. To watch all namespaces leave the namespace option empty:

mgr, err := manager.New(cfg, manager.Options{Namespace: ""})

(+) enough clear way to add 3rd Party Resources (i.e., different from the Core Kubernetes resource types) to your Operator

(-) it is still unstable/changing the APIs and the future versions of it could be not supported (we are observing this evolution and update the blog post accordingly)

To make the operator working, we had to open some issues in Github, that was closed and classified as a bug):

https://github.com/operator-framework/operator-sdk/issues/651

https://github.com/operator-framework/operator-sdk/issues/927

https://github.com/operator-framework/operator-sdk/issues/1053

(-) The workflow to create the GO operators appear a bit more complex/unclear (steps in red) that the one generated with Ansible and Helm (which require fewer steps):

  1. Create a new operator project using the SDK Command Line Interface (CLI)
  2. Define new resource APIs by adding Custom Resource Definitions (CRD)
  3. Define controllers to watch and reconcile resources
  4. Write the reconciling logic for your controller using the SDK and controller-runtime APIs
  5. Use the SDK CLI to build and generate the operator deployment manifests

Back to overview

Back to overview Section 3 – Examples of Supported Kubernetes Operator SDK workflows.

simon.beck

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Examples of Supported Kubernetes Operator SDK workflows

3. Apr 2019
This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
IN PREVIOUS BLOG POSTS WE TALKED ABOUT:
Section 1 – Operators, Operator Framework, and Operators SDK: 

  • Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
  • Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.

Section 2 – Supported Kubernetes Operator SDK workflows

  • Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  • We also discuss pros and cons of using the various operators workflows.

IN THIS BLOG POST WE WILL TALK ABOUT:
Section 3 – Examples of Supported Kubernetes Operator SDK workflows

  • Here we provide examples of the three available alternative workflows to generate Operators provided by the Operator SDK APIs.
  • We specifically focus on Go operators, as they are in our opinion the more stable available APIs.

Section 3 – Examples of Supported Kubernetes Operator SDK workflows

We will refer to:
1) Operator: Go operator 
2) Operator: Ansible operator [coming soon]
3) Operator: Helm operator [coming soon]
 
For each of these SDK supported workflows, we provide:
a) a description of the generated Operator structure (there is one for each specific workflow);
b) a link to our example(s) of operator(s) based on such Operator structure (there is one for each specific workflow) and logic.
c) descriptions on how to add (e.g., 3rd Party) Resources, different from the Core Kubernetes resource types, to your Operator 
d) a description of the main pros and cons of using suchOperator structures (there is one for each specific workflow) and logic.

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

simon.beck

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Supported Kubernetes Operator SDK workflows

This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
IN A PREVIOUS BLOG POST WE TALKED ABOUT:
 Section 1 – Operators, Operator Framework, and Operators SDK, and in particular:

  • in a general setting about Operators, Operator Framework, and Operators SDK.
  • the Operators SDK emerging popularity on GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.

IN THIS BLOG POST WE WILL TALK ABOUT:
Section 2 – Supported Kubernetes Operator SDK workflows

  • Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  • We also discuss the pros and cons of using the various operators’ workflows.

Section 2 – Supported Operator SDK workflows

As discussed before, on GitHub, the Operator SDK is a very active project, with over 10 releases produced in less than a year. This means that the Operator SDK is a toolkit that is continuing evolving over the time (e.g., its code,  structure, and logic is changing). In particular, as reported in the main Github page of the operator SDK, the libraries and tools are labeled with “Project Status: pre-alpha”, and thus are “expected breaking changes to the API in the upcoming releases”.
The project started in April 2018 and we started monitoring it intensively from September 2018. We found out that the SDK provides three different workflows to develop operators based on GoAnsible, or Helm.
These versions of the Operators SDK emerged between 2018 and 2019. Specifically, the first version of the operator was based on Go, and only from December 2018 it was provided a version based on Ansible.
Finally, in the beginning of 2019 (January), the operator workflow based on Helm was also released.
Thus, the SDK provides a workflow to develop operators based on GoAnsible, or Helm.
The following workflow is for a new Go operator:

      1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Define new resource APIs by adding Custom Resource Definitions (CRD)
      3. Define Controllers to watch and reconcile resources
      4. Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
      5. Use the SDK CLI to build and generate the operator deployment manifests

The following workflow is for a new Ansible operator:

      1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Write the reconciling logic for your object using Ansible playbooks and roles
      3. Use the SDK CLI to build and generate the operator deployment manifests
      4. Optionally add additional CRD’s using the SDK CLI and repeat steps 2 and 3

The following workflow is for a new Helm operator:

      1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Create a new (or add your existing) Helm chart for use by the operator’s reconciling logic
      3. Use the SDK CLI to build and generate the operator deployment manifests
      4. Optionally add additional CRD’s using the SDK CLI and repeat steps 2 and 3

Guidelines:
Command Line Interface: To learn more about the SDK CLI, see the SDK CLI Reference, or run operator-sdk [command] -h.
For a guide on Reconcilers, Clients, and interacting  with resource Events, see the Client API doc.
As it is possible to see from the following figure, there is not much difference among the various operator workflows.
However, the workflow that has reached more maturity and gives more control over the operator behavior is the one based on Go: 

Next article

Section 3 – Examples of Supported Operator SDK workflows

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

simon.beck

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

Introduction to Kubernetes Operators, Operator Framework, and Operators SDK

1. Mar 2019

Introduction to Kubernetes Operators, Operator Framework, and Operators SDK

This blog post is part of the series How to leverage Kubernetes operators using the Operator SDK framework.
Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK: 

  • Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
  • Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.


Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK

a) Operators are Kubernetes applications
A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. To be able to make the most of Kubernetes, you need a set of cohesive APIs to extend in order to service and manage your applications that run on Kubernetes. You can think of Operators as the “runtime that manages this type of application on Kubernetes“.
Thus, an Operator is a method of packaging, deploying and managing a Kubernetes application. Conceptually, an Operator takes human operational knowledge and encodes it into software that is more easily packaged and shared with consumers. We can think of an Operator as an extension of the software vendor’s engineering team that watches over your Kubernetes environment and uses its current state to make decisions in milliseconds. Operators follow a [maturity model] from basic to having specific logic for an application.
We’ve seen in the last years that Operators’ capabilities differ in sophistication depending on how much intelligence has been added into the implementation logic of the Operator itself. We’ve also learned that the creation of an Operator typically starts by automating an application’s installation and self-service provisioning capabilities, and then evolves to take on more complex automation, this depending on the specific use case. As result, advanced operators are, nowadays, designed to handle upgrades seamlessly, react to failures automatically, and not take shortcuts, like skipping a software backup process to save time.
b) Operator Framework
Operators are Kubernetes native applications that facilitate the management of complex stateful applications on top of Kubernetes, however writing such operators can be very difficult because of challenges such as using (i) low level APIs and (ii) a lack of modularity which leads to duplication, inconsistencies, and unexpected behaviors.
To address such issue, several tools are now being launched (e.g., the Operator FrameworkKooperMetacontroller, etc.) as results of years of work and experience of the Red Hat, Kubernetes, and CoreOS open source communities in building Operators. Specifically, Red Hat and the Kubernetes open source community shared the Operator Framework — an open source toolkit designed to manage  operators in a more effective, automated, and scalable way.
The Operator Framework is an open source toolkit composed by several low-level APIs. We believe that the new Operator Framework represents the next big step for Kubernetes by using a baseline of leading practices to help lower the application development barrier on Kubernetes. The project delivers a software development kit (SDK) and the ability to manage app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise operator capabilities on any Kubernetes cluster.
The Operator Framework includes:

    • Operator SDK: Enables developers to build Operators based on their expertise without requiring knowledge of Kubernetes API complexities.
    • Operator Lifecycle Management: Oversees installation, updates, and management of the lifecycle of all of the Operators (and their associated services) running across a Kubernetes cluster. Once built, Operators need to be deployed on a Kubernetes cluster. The Operator Lifecycle Manager is the backplane that facilitates management of operators on a Kubernetes cluster. With it, administrators can control what Operators are available in what namespaces and who can interact with running Operators. They can also manage the overall lifecycle of Operators and their resources, such as triggering updates to both an Operator and its resources.
    • Operator Metering (joining in the coming months): Enables usage reporting for Operators that provide specialized services. In a future version, the Operator Framework will also include the ability to meter application usage – a Kubernetes first, which provides extensions for central IT teams to budget and for software vendors providing commercial software. Operator Metering is designed to tie into the cluster’s CPU and memory reporting, as well as calculate IaaS cost and customized metrics like licensing.

Simple, stateless applications can leverage the Lifecycle Management features of the Operator Framework without writing any code by using a generic Operator (for example, the Helm Operator). However, complex and stateful applications are where an Operator can shine. The cloud-like capabilities that are encoded into the Operator code can provide an advanced user experience, automating such features as updates, backups and scaling.
In the next subsection we discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.
c) Operators SDK popularity
The Operator-SDK is a toolkit,  recently built on top of the Operator Framework, that provides the tools to build, test and package Operators. Initially, the SDK facilitated the marriage of an application’s business logic (for example, how to scale, upgrade, or backup) with the Kubernetes API to execute those operations. However, over time, the SDK is evolving to allow engineers to make applications smarter and have the user experience of cloud services. As consequence, leading practices and code patterns that are shared across Operators are included in the SDK to help prevent reinventing the wheel.
From a developer perspective, the entry point is the Operator SDK, originating from CoreOS, which is offered as part of the Operator Framework that is, according to its self-description, “an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way”. The SDK specifically targets Go developers and applications, and even if support for other programming languages (e.g., Java, C, etc.) is currently lacking,  future plans for their integration are already in place.
In GitHub, the Operator SDK is becoming a very active project, which already gained a high visibility/popularity with:

However, even if the project is becoming with the time more popular, its project Status is still “pre-alpha, which means that “are expected breaking changes to the API in the upcoming releases“.
Thus, the Operator SDK toolkit requires still a bit more of maturity to be used in wider practical working scenarios. As researchers, we believe that this software development kit (SDK) will be widely adopted in future, as it will support the developers during the management of app installs and updates by using the lifecycle management mechanism, while enabling administrators to exercise operator capabilities on any Kubernetes cluster (see the following Figure, it highlights the overall view of envisioned Operator SDK support).

As follow we talk about the Operators SDK General Workflow.
d) Operators SDK General Workflow
The Operator-SDK is a toolkit that provides the tools to build, test and packageOperators, as shown in the following Figure.

Specifically, the following specific workflow is provided by the toolkit for supporting the writingbuilding, testing and packaging of a new Go operator:

      1. Create a new operator project using the SDK Command Line Interface (CLI)
      2. Define new resource APIs by adding Custom Resource Definitions (CRD)
      3. Define Controllers to watch and reconcile resources
      4. Write the reconciling logic for your Controller using the SDK and controller-runtime APIs
      5. Use the SDK CLI to build and generate the operator deployment manifests

In this context, the Operator SDK uses for its workflow the controller-runtime library, which makes the writing of operators easier by providing:

      • High level APIs and abstractions to write the operational logic more intuitively.
      • Tools for scaffolding and code generation to bootstrap a new project fast.
      • Extensions to cover common operator use cases.

A simple example to create and deploy a simple operator with the SDK toolkit is provided  in the official operator SDK GitHub repository:
https://github.com/operator-framework/operator-sdk
The resulting automatically generated GO operator will present the following reference Structure:

File/Folders
Purpose
cmd
Contains manager/main.go which is the main program of the operator. This instantiates a new manager which registers all custom resource definitions under pkg/apis/... and starts all controllers under pkg/controllers/... .
pkg/apis
Contains the directory tree that defines the APIs of the Custom Resource Definitions (CRD).
pkg/controller
This pkg contains the controller implementations.
build
Contains the Dockerfile and build scripts used to build the operator.
deploy
Contains various YAML manifests for registering CRDs, setting up RBAC, and deploying the operator as a Deployment.
Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within an enterprise ]
Gopkg.toml Gopkg.lock
The Go Dep manifests that describe the external dependencies of this operator.
vendor
The golang vendor folder that contains the local copies of the external dependencies that satisfy the imports of this project. Go Dep manages the vendor directly.

In the next blog post we will talk about the Operators SDK current status, e.g., available versions and workflows.

Next article

Section 2 – Supported Kubernetes Operator SDK workflows

Back to overview

Back to overview How to leverage Kubernetes operators using the Operator SDK framework.

simon.beck

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Tech

How to leverage Kubernetes operators using the Operator SDK framework

How to leverage Kubernetes operators using the Operator SDK framework

Kubernetes has become an omnipresent platform to host cloud-native applications. As a rather low-level platform, it is often made developer-friendly by wrapping it into higher-level platforms, such as OpenShift (OKD), and by turning it into a managed service platform, such as APPUiO, which can be deployed to any cloud infrastructure. Application engineers interact with Kubernetes mostly by authoring appropriate deployment descriptors and by pushing their code which triggers deployments. Due to ongoing feature additions, not so much is known about useful combinations of annotations on Kubernetes deployments (and other declaratively described objects), Kubernetes operators (a kind of hooks) and custom resources definitions.
In this blog post series, we share some of the experience we have gained while researching how to trigger actions upon certain updates to the descriptors, as a precursor to dynamic and autonomous feedback loops which can self-manage application deployments.
In particular, we provide access to the adapted original examples of operators generated with the Operator SDK toolkit, which deal with Kubernetes resources by combining annotations on Kubernetes deployments and Kubernetes operators concepts. The link to our operators examples are available on Github: https://github.com/appuio/operator-sdk-examples. In further blog posts we will describe some, discussing also how they could be extended for more advanced decision making. In particular, adapting the (Go) operators to work on different environments require to modify some important go files (e.g., pkg/controller/memcached/memcached_controller.go as shown in the following Figure).

IN FURTHER BLOG POSTS WE WILL TALK ABOUT:

Section 1 – Kubernetes Operators, Operator Framework, and Operators SDK
  • Here we discuss in a general setting about Operators, Operator Framework, and Operators SDK.
  • Then we will discuss about the Operators SDK emerging popularity in GitHub, and in general about the “Operator SDK workflow” adopted for generating and handling operators.
Section 2 – Supported Kubernetes Operator SDK workflows
  • Here we discuss about the three available alternative workflows to generate Operators provided by the last versions of Operator SDK APIs.
  • We also discuss pros and cons of using the various operators workflows.
Section 3 – Examples of Supported Kubernetes Operator SDK workflows
  • Here we provide examples about the three available alternative workflows to generate Operators provided by the Operator SDK APIs.
  • We specifically focus on Go operators, as they are in our opinion the more stable available APIs.
Section 4 – Example(s) of Operator(s) Monitoring the Service with the usage of Prometheus (coming soon)
  • Here we provide an example of an operator that communicates with Prometheus (currently used to monitor Kubernetes Clusters) for more advanced decision making (e.g., advanced monitoring of the service).

About the authors

These blog posts have been written by Dr. Josef Spillner and Dr. Sebastiano Panichella from ZHAW (Zurich University of Applied Sciences) School of Engineering. Thank you very much Josef and Sebastiano for sharing your know how with our readers!

simon.beck

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us