(This post was originally published on the Crossplane blog.)

Crossplane has recently celebrated its fifth birthday, but at VSHN, we’ve been using it in production for almost three years now. In particular, it has become a crucial component of one of our most popular products. We’ve invested a lot of time and effort on Crossplane, to the point that we’ve developed (and open-sourced) our own custom modules for various technologies and cloud providers, such as Exoscale, cloudscale.ch, or MinIO.

In this blog post, we will provide an introduction to a relatively new feature of Crossplane called Composition Functions, and show how the VSHN team uses it in a very specific product: the VSHN Application Catalog, also known as VSHN AppCat.

Crossplane Compositions

To understand Composition Functions, we need to understand what standard Crossplane Compositions are in the first place. Compositions, available in Crossplane since version 0.10.0, can be understood as templates that can be applied to Kubernetes clusters to modify their configuration. What sets them apart from other template technologies (such as Kustomize, OpenShift Template objects, or Helm charts) is their capacity to perform complex transformations, patch fields on Kubernetes manifests, following more advanced rules and with better reusability and maintainability. Crossplane compositions are usually referred to as “Patch and Transform” compositions, or “PnT” for short.

As powerful as standard Crossplane Compositions are, they have some limitations, which can be summarized in a very geeky yet technically appropriate phrase: they are not “Turing-complete”.

• Compositions don’t support conditions, meaning that the transformations they provide are applied on an “all or nothing” basis.
• They also don’t support loops, which means that you cannot apply transformations iteratively.
• Finally, advanced operations are not supported either, like checking for statuses in other systems, or performing dynamic data lookups at runtime.

To address these limitations, Crossplane 1.11 introduced a new Alpha feature called “Composition Functions”. Note that as of writing, Composition Functions are in Beta in 1.14.

Composition Functions

Composition functions complement and in some cases replace Crossplane “PnT” Compositions entirely. Most importantly, DevOps engineers can create Composition Functions using any programming language; this is because they run as standard OCI containers, following a specific set of interface requirements. The result of applying a Composition Function is a new composite resource applied to a Kubernetes cluster.

Let’s look at an elementary “Hello World” example of a Composition Function.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: example-bucket-function
spec:
  compositeTypeRef:
    apiVersion: example.crossplane.io/v1
    kind: XBucket
  mode: Pipeline
  pipeline:
  - step: handle-xbucket-xr
    functionRef:
      name: function-xr-xbucket

The example above shows a standard Crossplane composition with a new field: “pipeline” specifying an array of functions, referred to via their name.

As stated previously, the function itself can be written in any programming language, like Go.

func (f *Function) RunFunction(_ context.Context, req *fnv1beta1.RunFunctionRequest) (*fnv1beta1.RunFunctionResponse, error) {
    rsp := response.To(req, response.DefaultTTL)
    response.Normal(rsp, "Hello world!")
    return rsp, nil
}

The example above, borrowed from the official documentation, does just one thing: it reads a request object, modifies a value, and returns it to the caller. Needless to say, this example is for illustration purposes only, lacking error checking, logging, security, and more, and should not be used in production. Developers use the Crossplane CLI to create, test, build, and push functions.

Here are a few things to keep in mind when working with Composition Functions:

• They run in order, as specified in the “pipeline” array of the Composition object, from top to bottom.
• The output of the previous Composition Function is used as input for the following one.
• They can be combined with standard “PnT” compositions by using the function-patch-and-transform function, allowing you to reuse your previous investment in standard Crossplane compositions.
• In the Alpha release, if you combined “PnT” compositions with Composition Functions, “PnT” compositions ran first, and the output of the last one is fed to the first function; since the latest release, this is no longer the case, and “PnT” compositions can now run at any step of the pipeline.
• Composition Functions must be called using RunFunctionRequest objects, and return RunFunctionResponse objects.
• In the Alpha release, these two objects were represented by a now deprecated “FunctionIO” structure in YAML format.
• RunFunctionRequest and RunFunctionResponse objects contain a full and coherent “desired state” for your resources. This means that if an object is not explicitly specified in a request payload, it will be deleted. Developers must pass the full desired state of their resources at every invocation.

Practical Example: VSHN AppCat

Let’s look at a real-world use case for Crossplane Composition Functions: the VSHN Application Catalog, also known as AppCat. The AppCat is an application marketplace allowing DevOps engineers to self-provision different kinds of middleware products, such as databases, message queues, or object storage buckets, in various cloud providers. These products are managed by VSHN, which frees application developers from a non-negligible burden of maintenance and oversight.

Standard Crossplane “PnT” Compositions proved limited very early in the development of VSHN AppCat, so we started using Composition Functions as soon as they became available. They have allowed us to do the following:

• Composition Functions enable complex tasks, involving the verification of current deployment values and taking decisions before deploying services.
• They can drive the deployment of services involving Helm charts, modifying values on-the-go as required by our customers, their selected cloud provider, and other parameters.
• Conditionals allow us to script complex scenarios, involving various environmental decisions, and to reuse that knowledge.
• Thanks to Composition Functions, the VSHN team can generalize many activities, like backup handling, automated maintenance, etc.

All things considered, it is difficult to overstate the many benefits that Composition Functions have brought to our workflow and to our VSHN AppCat product.

Learnings of the Alpha Version

We’ve learned a lot while experimenting with the Alpha version of Composition Functions, and we’ve documented our findings for everyone to learn from our mistakes.

• Running Composition Functions in Red Hat OpenShift used to be impossible in Alpha because OpenShift uses crun, but this issue has now been solved in the Beta release.
• In particular, when using the Alpha version of Composition Functions, we experienced slow execution speeds with crun but this is no longer the case.
• We learned the hard way that missing resources on function requests were actually deleted!

Our experience with Composition Functions led us to build our own function runner. This feature uses another capability of Crossplane, which allows functions to specify their runner in the Composition definition:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
[...]
  functions:
  - name: my-function
    type: Container
    container:
      image: my-function
      runner:
        endpoint: grpc-server:9547

Functions run directly on the gRPC server, which, for security reasons, must run as a sidecar to the Crossplane pod. Just like everything we do at VSHN, the Composition Function gRPC Server runner (as well as its associated webhook and all of its code) is open-source, and you can find it on our GitHub. As of the composition function beta, we replaced the custom GRPC logic with the go-sdk. To improve the developer experience, we have created a proxy and enabled the gRPC server to run locally. The proxy runs in kind and redirects to the local gRPC server. This enables us to debug the code and to test changes more efficiently.

Moving to Beta

We recently finished migrating our infrastructure to the most recent Beta version of Composition Functions, released in Crossplane 1.14, and we have been able to do that without incidents. This release included various bits and pieces such as Function Pipelines, an ad-hoc gRPC server to execute functions in memory, and a Function CRD to deploy them directly to clusters.

We are also migrating all of our standard “PnT” Crossplane Compositions to pure Composition Functions as we speak, thanks to the functions-go-sdk project, which has proven very helpful, even if we are missing typed objects. Managing the same objects with the “PnT” and Composition Functions increases complexity dramatically. As it can be difficult to determine where an actual change happens.

Conclusion

In this blog post, we have seen how Crossplane Composition Functions compare to standard “PnT” Crossplane compositions. We have provided a short example, highlighting their major characteristics and caveats, and we have outlined a real-world use case for them, specifically VSHN’s Application Catalog (or AppCat) product.

Crossplane Composition Functions provide an unprecedented level of flexibility and power to DevOps engineers. They enable the creation of complex transformations, with all the advantages of an Infrastructure as Code approach, and the flexibility of using the preferred programming language of each team.

Check out my talk at Control Plane Day with Crossplane, where I walk you through Composition Functions in Production in 15 minutes.

Yesterday evening, on Monday, July 24th, 2023, at around 21:15 CEST / 12:15 PDT, our security team received a notification about a critical security vulnerability called “Zenbleed” potentially affecting the cloud providers where VSHN’s customers systems run on.

This blog post provides details about Zenbleed and the steps taken to mitigate its risks.

What is Zenbleed?

Zenbleed, also known as CVE-2023-20593, is a speculative execution bug discovered by Google, related to but somewhat different from side channel bugs like Meltdown or Spectre. It is a vulnerability affecting AMD processors based on the Zen2 microarchitecture, ranging from AMD’s EPYC datacenter processors to the Ryzen 3000 CPUs used in desktop & laptop computers. This flaw can be exploited to steal sensitive data stored in the CPU, including encryption keys and login credentials.

VSHN’s Response

VSHN immediately set up a task force to discuss this issue, including the team of one of our main cloud providers (cloudscale.ch) in a call to determine choices of action; among possible options, were contemplated ideas like isolating VSHN customers on dedicated nodes, or patching the affected systems directly.

At around 22:00 CEST, the cloud provider decided after a fruitful discussion with the task force that the best approach was to implement a microcode update. Since Zenbleed is caused by a bug in CPU hardware, the only possible direct fix (apart from the replacement of the CPU) consists of updating the CPU microcode. Such updates can be applied by updating the BIOS on affected systems, or applying an operating system kernel update, like the recently released new Linux kernel version that addresses this vulnerability.

Zenbleed isn’t limited to just one cloud provider, and may affect customers operating their own infrastructure as well. We acknowledged that addressing this vulnerability is primarily a responsibility of the cloud providers, as VSHN doesn’t own any infrastructure that could directly be affected.

The VSHN task force handed the monitoring over to VSHN Canada to test the update as it rolled out to production systems, who stayed in close contact to ensure there were no QoS degradations after the microcode update.

Aftermath

cloudscale.ch successfully finished its work at 01:34 CEST / 16:34 PDT. All VSHN systems running on that provider have been patched accordingly, and the tests carried show that this specific vulnerability has been fixed as required. VSHN Canada confirmed that all systems were running without any problems.

We will continue to monitor this situation and to inform our customers accordingly. All impacted customers will be contacted by VSHN. Please do not hesitate to contact us for more information.

Kubernetes is a powerful platform for deploying and managing containerized applications at scale, and it has become increasingly popular in Switzerland in recent years.

One way to approach it is outsourcing. This can be a strategic and cost-effective option for organizations that do not have the in-house DevOps expertise, know-how, or resources to manage their infrastructure and application operations efficiently.

Not every tech company is in the business of building platforms and operating Kubernetes clusters. Thus by partnering with an experienced partner, companies can tap into a wealth of knowledge and expertise to help them run their applications.

Some companies adopt Kubernetes and look to leverage its capabilities themselves. It’s essential to consider time, effort, and possible implications while utilizing the latest developments and continually adding value to the core business.

In all cases, it will be helpful to align with fundamentals. For this reason, I have compiled a quick guide to Kubernetes in 2023 and best practices in Switzerland.

1. Understand the basics: Before diving into Kubernetes, have a solid understanding of the reasoning and concepts. This could include cloud infrastructure, networking, containers, how they liaise with each other, and how they can be managed with Kubernetes.
2. Plan your deployment carefully: When deploying applications with Kubernetes, you must plan thoroughly and consider your workloads’ specific needs and requirements. This includes but is not limited to resource requirements, network connectivity, scalability, latency, and security considerations.
3. Use appropriate resource limits: One of the critical benefits of Kubernetes is its ability to manage resources dynamically based on the needs of your applications. To take advantage of this, try to set appropriate resource limits for your application. This will help ensure that your application has the resources they need to run effectively while preventing them from consuming too many resources and impacting other applications.
4. Monitor your application: It’s essential to monitor your applications and the underlying Kubernetes cluster to ensure they are running smoothly and to identify any issues that may arise. You want to analyze the alerts and react accordingly. You can use several tools and practices to monitor your applications, including log analysis, monitoring with tools like Prometheus and Grafana, and alerting systems.
5. Use appropriate networking configurations: Networking is critical to any Kubernetes deployment, and choosing the proper network configuration is substantial. What about load balancing, service discovery, and network segmentation?
6. Secure your application: Security is a top concern for many companies and organizations in Switzerland. You cannot proceed without ensuring that your Kubernetes deployment is secure. At this stage, your team is implementing network segmentation, using secure container runtime environments, and implementing advanced authentication and authorization systems.
7. Consider using a managed Kubernetes service: For companies without the resources or needing DevOps expertise to manage their clusters, managed Kubernetes services can be a business-saving solution. With managed services, you can get a production-ready cluster, i.e., a fully-managed Kubernetes environment, allowing teams and software engineers to focus on developing new features and deploying their applications rather than managing the underlying infrastructure.
8. Stay up-to-date with the latest developments: The Kubernetes ecosystem is constantly evolving, and it’s better to stay up-to-date with the latest developments and best practices. This may involve subscribing to newsletters like VSHN, VSHN.timer, or Digests, attending conferences and CNCF meetups, and following key players in the Kubernetes community.

By following best practices, IT leaders, stakeholders, and decision-makers can ensure that they use Kubernetes constructively and get the most out of the platform technology.

We have recently received a request from a customer, asking us to restrict the container registries that could be used to deploy images from in their OpenShift 4 cluster.

We could have added such configuration directly at node level, as explained in Red Hat’s documentation; it’s indeed possible to whitelist registries on repository and tag level, but that would have forced us to keep all those whitelists updated with those our Project Syn components regularly use.

We have instead chosen to use Kyverno for this task: it allows us to enforce the limitations on a per-namespace level, with much more flexibility and maintanability.

This is a ClusterPolicy object for Kyverno, adapted from the solution we provided to our customer, showing how we can restrict the limitation to some namespaces, so that containers can be pulled only from some specific registries.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-registries
  annotations:
    policies.kyverno.io/title: Restrict Image Registries
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Restrict image pulling only to whitelisted registries
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: validate-registries
    match:
      all:
      - resources:
          kinds:
          - Pod
          namespaces:
          - "namespace-wildcard-*"
    validate:
      message: "Image registry not whitelisted"
      pattern:
        spec:
          containers:
          - image: "registry.example.com/* | hub.docker.com/some-username/*"

As you know (and if you didn’t, now you do) GitLab has deprecated the certificate-based integration with Kubernetes in version 14.5, and it is expected that version 15 will disable it completely.

The official replacement to the (now legacy) certificate-based integration mechanism is the GitLab Agent, to be installed in your Kubernetes cluster, and providing a tighter integration between our two beloved platforms.

Well, hold on; things aren’t that easy. For example in our case, if we wanted to offer the GitLab Agent to our APPUiO Cloud users, we would run into two issues:

• On one side, installing the GitLab Agent is more complicated and expensive, because it would run as another pod in the same namespace, consuming resources.
• And on the other, we cannot install it cluster-wide, because of the admin permissions we have configured in this multi-tenant shared platform. Users would have to manage each and every GitLab Agent on their own, possibly having multiple agents deployed in several namespaces and GitLab repositories.

So, what’s a DevOps engineer to do? Well, here’s a simple, very simple solution; so simple that you might have never thought of it before: create your own KUBECONFIG variable in GitLab with a dedicated service account!

Here’s how it works using the OpenShift oc tool in your own system:

Create a service account in your OpenShift project:

oc create serviceaccount gitlab-ci

Add an elevated role to this service account so it can manage your deployments:

oc policy add-role-to-user admin -z gitlab-ci --rolebinding-name gitlab-ci

Create a local KUBECONFIG variable and login to your OpenShift cluster using the gitlab-ci service account:

TOKEN=$(oc sa get-token gitlab-ci)
export KUBECONFIG=gitlab-ci.kubeconfig
oc login --server=$OPENSHIFT_API_URL --token=$TOKEN
unset KUBECONFIG

You should now have a file named gitlab-ci.kubeconfig in your current working directory; copy its contents and create a variable named KUBECONFIG in the GitLab settings with the value of the file (that’s under “Settings” > “CI/CD” > “Variables” > “Expand” > “Add variable”). Remember to set the “environment” scope for the variable and to disable the old Kubernetes integration, as the KUBECONFIG variable might collide with this new variable.

Tada! There are a few advantages to this approach:

• It is certainly faster and simpler to setup for simple push-based deployments than by using the GitLab Agent.
• It is also easier to get rid of deprecated features without having to change the pipeline or migrating to the GitLab Agent.

But there are a few drawbacks as well:

• You don’t get all bells and whistles. It is reasonable to think that at some point the GitLab Agent will offer advanced features, such as access to pod logs from GitLab, monitoring alerts directly from the GitLab user interface, and many other things the old Kubernetes certificate-based integration could do. This approach does not provide anything like this.
• The cluster’s API endpoint has to be publicly accessible, not behind any firewall or VPN; conveniently enough, the GitLab Agent solves exactly this problem.

If you are interested in GitLab and Kubernetes, join us next week in our GitLab Switzerland Meetup for more information about the GitLab Agent for Kubernetes, and afterwards for a nice apéro with like-minded GitLab enthusiasts!

Oh, and a final tip; if you find yourself having to log in to OpenShift integrated registry but you don’t have yq installed in the job’s image, you can use sed to extract the token from the $KUBECONFIG file:

sed -n 's/^\s*token:\s*\(.*\)/\1/ p' "${KUBECONFIG}" | docker login -u gitlab-ci --password-stdin "${OPENSHIFT_REGISTRY}"

Back in 2019 we published a review of the most relevant serverless frameworks available for Kubernetes. That article became one of the most visited in our blog in the past two years, so we decided to return to this subject and provide an update for our readers.

TL;DR: Serverless in Kubernetes in 2022 means, to a large extent, Knative.

What’s in a Word

The “Serverless” word is polarizing.

Robert Scoble, one of the first tech influencers, uttered it for the first time fourteen years ago, as James Potter reported recently. In those times, “Serverless” just meant “not having servers and being an AWS EC2 customer”. Because, yes, companies used to have their own physical servers back then. Amazing, isn’t it?

Fast forward to 2022, and the CNCF Serverless Landscape has grown to such an extent that it can be very hard to figure out what “serverless” truly means.

Even though for some it just represents the eternal return of 1970s style batch computing, in the past five years the “Serverless” word has taken a different and very specific meaning.

The winning definition caters for the complete abstraction of the infrastructure required to run individual pieces of software, at scale, on a cloud infrastructure provider.

Or, in less buzzword-y terms, just upload your code, and let the platform figure out how to run it for you: the famous “FaaS”, also known as Function as a Service.

The IaaS Front

The three major Infrastructure as a Service providers in the world offer their own, more-or-less-proprietary version of the Serverless paradigm: AWS Lambda, Azure Functions, and Google Cloud Run (which complemented the previous Google Cloud Functions service). These are three different approaches to the subject of FaaS, each with its advantages and caveats.

Some companies, like A Cloud Guru, have successfully embraced the Serverless architecture from the very start (in this case, based on AWS Lambda), creating cost-effective solutions with incredible scalability.

But one of the aforementioned caveats, and not a small one for that matter, is platform lock-in. Portability has always been a major concern for enterprise computing: if building apps on AWS Lambda is an interesting prospect, could we move them to a different provider later on?

Well, we now have an answer to that question, thanks to our good old friend: the container.

The Debate Is Over

Almost exactly three years ago, Anthony Skipper wrote:

We will say it again… packaging code into containers should be considered a FaaS anti-pattern!

Containers or not? This was still a big debate at the time of our original article in 2019.

Some frameworks like Fission and IaaS services such as AWS Lambda and Google Cloud Functions did not require developers to package their apps as containers; just upload your code and watch it run. On the other hand, OpenFaaS and Knative-based offerings did require containers. Who would win this fight?

The world of Serverless solutions in 2022 has decided that wrapping functions in containers is the way to go. Even AWS Lambda started offering this option in December 2020. This is a huge move, allowing enterprises to run their code in whichever infrastructure they would like to.

In retrospect, the market has chosen wisely. Containers are now a common standard, allowing the same code to run unchanged, from a Raspberry Pi to your laptop to an IBM Mainframe. It is a natural choice, and it turned out that it was a matter of time before this happened.

Even better, with increased industry experience, container images got smaller and smaller, thanks to Alpine-based, scratch-based, and distroless-based images. Being lightweight allows containers to start and stop almost instantly, and makes scaling applications faster and easier than ever.

And this choice turned out to benefit one specific framework among all: Knative.

The Age of Knative

In the Kubernetes galaxy, Knative has slowly by steadily imposed its mark as the core infrastructure of Kubernetes serverless workloads.

In 2019, our article compared six different mechanisms to run serverless payloads on Kubernetes:

1. OpenFaaS
2. Fn Project
3. Fission
4. OpenWhisk
5. Kubeless
6. TriggerMesh

Of those, Fn Project and Kubeless have been simply abandoned. Other frameworks suffered the same fate: Riff has disappeared, just like Gestalt, including its parent company Galatic Fog. IronFunctions moved away from Kubernetes into its own PaaS product. Funktion has been sandboxed and abandoned; Leveros is abandoned too; BlueNimble does not show much activity.

On the other hand, new players have appeared in the serverless market: Rainbond, for example; or Nuclio, targeting the scientific computation market.

But many new contenders are based on Knative: apart from TriggerMesh, which we mentioned in 2019 already, we have now Kyma, Knix, and Red Hat’s OpenShift 4 serverless, all powered by Knative.

The interest in Knative is steadily growing these days: CERN uses it. IBM is talking about it. The Serverless Framework has a provider for it. Even Google Cloud Run is based on it! Which shouldn’t be surprising, knowing that Knative was originally created by Google, just like Kubernetes.

And now Knative has just been accepted as a CNCF incubating project!

Even though Knative is not exactly a FaaS per se, it deserves the top spot in our review of 2022 FaaS-on-K8s technologies, being the platform upon which other serverless services are built, receiving huge support from the major names of the cloud-native industry.

Getting Started with Knative

Want to see Knative in action? Getting started with Knative on your laptop now is easier than ever.

1. Install kind.
2. Run the following command on your terminal:

$ curl -sL install.konk.dev | bash

To work with Knative objects on your cluster, install the kn command-line tool. Once you have launched your new Knative-powered Kind cluster, create a file called knative-service.yaml with the contents below:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
spec:
  template:
    metadata:
      name: hello-world
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go
          ports:
            - containerPort: 8080
          env:
            - name: TARGET
              value: "World"

And then just apply it: kubectl apply -f knative-service.yaml.

The kn service list command should now display your “helloworld” service, which should become available after a few seconds. Once it’s ready, you can execute it simply with curl:

$ curl http://hello.default.127.0.0.1.sslip.io

(If you prefer to use Minikube, you can follow this tutorial instead.)

Thanks to Knative, developers can roll out new versions of their services (called “revisions” in Knative terminology) while the old ones are still running, and even distribute traffic among them. This can be very useful in A/B testing sessions, for example. Knative services can be triggered by a large array of events, with great flexibility.

A full introduction to Knative is outside of the scope of this review, so here are some resources we recommend to learn everything about Knative serving and eventing:

• The excellent and funny “Knative in Action” (2021) book by Jacques Chester, available for free courtesy of VMWare.
• A free, full introduction to Knative (July 2021) by Sebastian Goasguen, the founder of TriggerMesh; a video and its related source code are provided as well.
• And to top it off, the “Knative Cookbook” (April 2020) by Burr Suter and Kamesh Sampath, also available for free, courtesy of Red Hat.

Interested in Knative and Red Hat OpenShift Serverless? Get in touch with us and let us help you in your FaaS journey!

As part of our ongoing improvement process, we evaluate the requirements and new features of each version of Red Hat OpenShift, not only for our customers, but also for our internal use. Version 4.9 of Red Hat’s flagship container platform was announced October 18th, 2021 and it included some very important changes, some of which are potentially breaking ones.

In this article, targeted towards DevOps engineers and maintenance crews, I will provide a short overview of the most important points to take care before upgrading to this new version.

Kubernetes 1.22

The most important change in OpenShift 4.9 is the update to Kubernetes 1.22. This release of Kubernetes has completely removed APIs marked as v1beta1. The complete list is available in the Red Hat documentation website, but suffice to say that common objects such as Role or RoleBinding (rbac.authorization.k8s.io/v1beta1) and even Ingress (networking.k8s.io/v1beta1) are no more.

This is a major change, and it needs to be taken care of for all users of your clusters before an upgrade takes place, and all manifest files using those resources should be updated accordingly. Red Hat’s Knowledge Base includes a special article explaining all the steps required for upgrading to OpenShift 4.9.

And of course, if you need any help regarding the upgrade process towards OpenShift 4.9, just contact us, we will be glad to help you and your teams.

Mutual TLS Authentication

Mutual TLS is a strong way to secure an application running in the cloud, allowing a server application to authenticate any client connecting to it. It comes with some complexity to setup and manage, as it requires a certificate authority. But thanks to its inclusion as a feature in OpenShift 4.9, its usage is much simpler now.

Please check the release notes section related to Mutual TLS for more information.

Registry Multiple Logins

In previous versions of OpenShift, you could only list one repository from a given registry per project. OpenShift 4.9 includes multiple logins to the same registry, which allows pods to pull images from specific repositories in the same registry, each with different credentials. You can even define a registry with a specific namespace. The documentation contains examples of manifests to use this feature.

Changes to lastTriggeredImageID Field Update

And finally, here’s a change that can cause unforeseen headaches to your teams: OpenShift 4.9 does not update anymore the buildConfig.spec.triggers[].imageChange.lastTriggeredImageID field when the ImageStreamTag changes and references a new image. This subtle change in behavior is easy to overlook, and if your team depends on this feature, beware for trouble.

Learn More about OpenShift 4.9

If you’re interested in knowing what else changed in OpenShift 4.9, here are some selected resources published by Red Hat to help you:

• Red Hat OpenShift 4.9 Is Now Generally Available
• OpenShift Container Platform 4.9 release notes
• Time Is On Your Side: A Change to the OpenShift 4 Lifecycle
• How Customer Feedback Shaped OpenShift 4.9
• Ask an OpenShift Admin Office Hour – What’s new in OpenShift 4.9?
• Kuryr in OpenShift 4.9
• Introducing New Virtualization Features in OpenShift 4.9

Let’s take a somewhat tangential approach to the subject of the Microservices architecture. Most discussions about it are centered around technological aspects: which language to choose, how to create the most RESTful services, which service mesh is the most performant, etc.

However at VSHN we have long ago figured out that the most important factor of success for software projects is people. And the thesis of this article is that the choice of Microservices as an architectural pattern has more to do with the organizational structure of your organization, rather than the technological constraints and features of the final deliverable.

Or, put differently: your team has already chosen an architecture for your software, even if they are not fully aware of it. And lo and behold, Microservices might or might not be it.

Definition

First of all we must define the Microservices architecture. What is it?

“Microservices” is an architectural pattern in which the functionality of the whole system is decomposed into completely independent components, communicating with each other over the network.

The counterpart of the Microservices architecture is what is commonly referred to as the “Monolith”, which has been the most common approach for web applications and services in the past 25 years. In a Monolith, all functions of the application, from data management to the UI, are all contained within the same binary or package.

On the opposite side of the street we find the Microservices architecture, where each service is responsible of its own implementation and data storage.

By definition, Microservices are fine grained, and have a single purpose. They have strong cohesion, that is, the operations they encapsulate have a high degree of relatedness. They are an example of the “Single Responsibility Principle”. They are also deployed separately, with visibly bounded contexts, and they require DevOps approaches to their deployment and management, such as automation and CI/CD pipelines.

Very importantly, the Microservices architecture is a “share nothing architecture”, in which individual services never share common code through libraries, instead restricting all interaction and communication through the network connecting them.

And last but not least, Microservices (as the name implies) should be as small as possible, have low requirements of memory, and should be able to start and stop in seconds.

Given all of these characteristics, Microservices are, by far, the most complex architectural pattern ever created. It is difficult to plan, it can dramatically increase the complexity of projects, and for some teams, experience has shown that it was impossible to cope with.

History

This idea of “components sending messages to one another” is absolutely not new. Back in 1966, one of the greatest computer scientists of all time, Alan Kay, coined the term “Object Oriented Programming”. The industry co-opted and deformed his original definition, which was the following:

OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.

Alan Kay, source.

This text is from 2003; the following is from 1998:

The big idea is “messaging” — that is what the kernal of Smalltalk/Squeak is all about (…) The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.

Alan Kay, source.

Alan Kay designed in the 1970s the Smalltalk programming language, based on these concepts. And after reading the texts above, it becomes clear that to a large extent, the Microservices architecture is an implementation of Alan Kay’s ideas of messaging, decoupling and abstraction, taken to the extreme.

To achieve the current state of Microservices, though, many other breakthroughs were required. During the 2000s, the emergence of XML, the SOAP protocol, and its related web services made the term “Service Oriented Architecture” a very common staple in architectural discussions. The rise of Agile methodologies made the industry pivot towards the REST approach instead of SOAP. During the last decade, the rise of DevOps and the rise of containerization through Docker and Kubernetes finally enabled teams to deploy thousands of containers as a microservice architecture, thanks to the whole catalog of Cloud Native technologies.

Pros and Cons

If Microservice architectures are so complex, why use them? It turns out, they can bring many benefits:

• Since each component can be completely isolated from the others, once teams agree on their interfaces they can be developed, documented, and tested thoroughly, 100% independently from others. This allows teams to move forward in parallel, implementing features that have 0% chance of collision with one another.
• Teams are also encouraged to choose the programming language that fits best the particular task that their microservice must fulfill.
• Since they are by definition “micro” services, they are designed to be quickly launched and dismissed, so that they only intervene when required, making the whole system more efficient and responsive.
• The size of microservices also allows for higher availability, since it is possible to have many of them behind a load balancer; should any instance of a microservice fail, it can be quickly dismissed and a new one instantiated in its place, without loss of availability.
• Systems can be updated progressively, with each team fixing bugs and adding functionality without disturbing the others. As long as interfaces are respected (and eventually versioned) there is no reason for the system to suffer from updates.

But there are many reasons not to choose the Microservices architectural approach; among the most important:

• Performance: a system built with Microservices must take into account the latency between those services; and in this respect, Monolithic applications are faster; a function call is always faster than a network call.
• Team maturity: Microservices demand a certain degree of experience and expertise from teams; those new to Microservices have a higher chance of project failures.
• Cost: creating a Microservices system will be costlier, if anything, because of the overhead created by each individual service.
• Feasibility: sometimes it is simply not possible to use Microservices, for example when dealing with legacy systems.
• Team structure: this is a decisive factor we will talk about extensively later.
• Complexity: it is not uncommon to end up with systems composed of thousands of concurrent services, and this creates challenges that we will also discuss later.

I would like to talk now about the last two points, which are in our experience the biggest issues in Microservice implementations: team structure and the perception and management of complexity.

Conway’s Law

One of the decisive factors that constrain teams’ ability to implement microservices is often invisible and overlooked: their own structure. Again, this phenomenon is not new. In 1968, Melvin A. Conway wrote an article for the Datamation magazine called “How Do Committees Invent?” in which the following idea stands out:

The basic thesis of this article is that organizations which design systems (…) are constrained to produce designs which are copies of the communication structures of these organizations.

Melvin Conway, source.

There is extensive evidence, both anecdotal and empirical, of this fact through research.

The corollary of this principle is the following: the choice of a Monolithic versus a Microservices architecture is already ingrained in the very hierarchical chart representing any organization.

One of the services we offer at VSHN tackles, precisely, this very issue. In our “DevOps enablement workshop” we evaluate the degree of agility of organizations, and the extent and improvements that DevOps could bring. Based on that information, we reverse engineer their structure through Conway’s Law in order to find a starting point for their digital transformation.

Complex vs. Complicated

Another important point is the distinction of “Complex” versus “Complicated”, as these two words can sometimes be confused with one another in everyday language, and to make things more difficult, the word “Simple” can be used as an antonym of both.

“Complex” is borrowed from the Latin complexus, meaning “made of intertwined elements”. Complexus is itself derived from plectere (“bend, intertwine”). This word has been used since the XVI century to qualify that which is made of heterogenous elements. It shares the same root (plectere) with the medical term “plexus” meaning “interlacing” and used since the 16th century as a medical term for “network of nerves or blood vessels”.

(Source: Dictionnaire historique de la langue française by Alan Rey)

On the other hand, “Complicated” has a similar origin but a different construction: it comes from the Latin complicare, literally meaning “to fold by rolling up”. Figuratively speaking this was taken as close to the notion of embarrassment or awkwardness. The word is composed of the word plicare which means “to fold”. Watches commonly known as “Complications” (such as the Patek Philippe Calibre 89, the Franck Muller Aeternitas Mega and the “Référence 57260” de Vacheron Constantin) are, well, complicated machines by definition.

In short, “Complex” and “Complicated” stem from slightly different roots: the Latin root plectere (“to intertwine”) in the former, and plicare (“to fold”) for the latter. Complex conveys the idea of a network of intertwined objects, whose state and behavior are continuously altered by the interaction with their peers in said network. The word complicated implies an intrinsic apparent “obscurity” through folding unto itself, inviting to an “unfolding” discovery process.

Or, put in another way: individual Microservices should not be complicated, but a Microservice architecture is complex by definition. Monoliths, on the other hand, tend to become very complicated as time passes. And of course, neither are simple.

History shows that software developers have a passionate relationship with complication; complicated systems are great to brag about on Hacker News, while maintainers also cry about them in private.

A “best practice” in this context has one and only one basic job: to help engineers translate the complicated into the complex. Most software-related disasters are caused by a simple fact: because of deadlines, organization, tooling, or just plain ignorance, software developers have a tendency to build complicated, instead of complex, systems.

This is another point we take care of in our DevOps Workshop, through the evaluation of the current assets (not only source code, but also current databases schemas, security requirements, network topologies, deployment procedures and rhythms, etc.)

Migrate or Rewrite? Equilibrium

The complication of Monoliths by itself is not problematic; it makes for tightly bound systems, which tend to be very fast, because as we said, a function call is faster than a network call. After all, we have been building very successful monoliths in the past. But experience shows that they tend to present problems regarding availability and scalability. Microservices represent a diametrally oposed approach, based on complexity rather than complication, but one that solves those issues.

There is a tension, then, between complexity and complication on one side, and organizational forces on the other. Put in other words, there is a tension between monolithic and microservices systems on one side, and more or less hierarchical structures on the other. Achieving equilibrium between these forces is, then, the engineering challenge faced by software architects these days.

Many teams face today the task, either requested internally (from their management) or externally (through customer demand or vendor requirements) of migrating their monoliths into Microservice-based architectures. Architects can thankfully apply a few techniques to find an equilibrium:

1. Start your migration path knowing that very often one does not need to migrate the whole application to Microservices. Some parts can and should remain monolithic, and in particular, proven older systems, even if written in COBOL or older technologies, can still deliver value, and can play a very important role in the success of the transition.
2. Identify components correctly, so that when isolated they will be neither only functionality-driven, nor only data-driven, nor only request-driven, but rather driven by these three factors (functionality, data, and request) at the same time. Pay attention to the organizational chart, and use that as a basis for the decomposition in microservices.
3. Remember that network bandwidth is not infinite. Some interfaces should be “chunky”, while others should be “chatty”. Plan for latency issues from the start.
4. Reduce inter-service communication as much as possible, which can be done in many ways:
   1. Consolidating services together
   2. Consolidating data domains (combining database schemas or using common caches)
   3. Using technologies such as GraphQL to reduce network bandwidth
   4. Using messaging queues, such as RabbitMQ.
5. Adopt Microservice-friendly technologies, such as Docker containers, Kubernetes, Knative, or Red Hat’s OpenShift and Quarkus.
6. Implement an automated testing strategy for each and every microservice.
7. Standardize technology stacks around containers & Kubernetes, and create common ground for a true microservice ecosystem within organizations.
8. Automate all of your work as much as possible, knowing that the effort for automation (DevOps, CI/CD) can be amortized over many services, and becomes thus a net investment in the long run.

As mentioned previously, we regularly help organizations in their digital transformation towards microservices, Kubernetes, OpenShift, DevOps, CI/CD, GitLab, and DevOps in general, to support your teams with the tooling they will need in the future. Borrowing Henny Portman’s Team Topologies concept, VSHN can support both as an “Enabling Team” (DevOps workshop, consulting) and a “Platform Team” (Kubernetes/OpenShift) to build microservices on, ensuring stability and peace of mind.

Conclusion

Going beyond the hype, Microservice architectures bring great benefits, but can become huge challenges to software teams.

The best way to tackle those challenges consists in reverse engineering Conway’s Law, and start by the analysis of the human organization of your teams first. Make them independent, agile, and free to choose the best tools for their job. Encourage them to negotiate with one another the interfaces of their respective components.

Let us create and run complex, not complicated, systems. We cannot get away from complexity; that is our job as engineers. But we can get rid of the complicated part.

As a closing thought, I will quote my former colleague and lifetime friend Adam Jones, an independent IT consultant from Geneva: in order to achieve success with the Microservice architecture, you must embed structure in your activities, and remove it from your hierarchy. It is not about making the structure go away; but instead, moving it to where it does the most good.