Let’s take a somewhat tangential approach to the subject of the Microservices architecture. Most discussions about it are centered around technological aspects: which language to choose, how to create the most RESTful services, which service mesh is the most performant, etc.

However at VSHN we have long ago figured out that the most important factor of success for software projects is people. And the thesis of this article is that the choice of Microservices as an architectural pattern has more to do with the organizational structure of your organization, rather than the technological constraints and features of the final deliverable.

Or, put differently: your team has already chosen an architecture for your software, even if they are not fully aware of it. And lo and behold, Microservices might or might not be it.

Definition

First of all we must define the Microservices architecture. What is it?

“Microservices” is an architectural pattern in which the functionality of the whole system is decomposed into completely independent components, communicating with each other over the network.

The counterpart of the Microservices architecture is what is commonly referred to as the “Monolith”, which has been the most common approach for web applications and services in the past 25 years. In a Monolith, all functions of the application, from data management to the UI, are all contained within the same binary or package.

On the opposite side of the street we find the Microservices architecture, where each service is responsible of its own implementation and data storage.

By definition, Microservices are fine grained, and have a single purpose. They have strong cohesion, that is, the operations they encapsulate have a high degree of relatedness. They are an example of the “Single Responsibility Principle”. They are also deployed separately, with visibly bounded contexts, and they require DevOps approaches to their deployment and management, such as automation and CI/CD pipelines.

Very importantly, the Microservices architecture is a “share nothing architecture”, in which individual services never share common code through libraries, instead restricting all interaction and communication through the network connecting them.

And last but not least, Microservices (as the name implies) should be as small as possible, have low requirements of memory, and should be able to start and stop in seconds.

Given all of these characteristics, Microservices are, by far, the most complex architectural pattern ever created. It is difficult to plan, it can dramatically increase the complexity of projects, and for some teams, experience has shown that it was impossible to cope with.

History

This idea of “components sending messages to one another” is absolutely not new. Back in 1966, one of the greatest computer scientists of all time, Alan Kay, coined the term “Object Oriented Programming”. The industry co-opted and deformed his original definition, which was the following:

OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.

Alan Kay, source.

This text is from 2003; the following is from 1998:

The big idea is “messaging” — that is what the kernal of Smalltalk/Squeak is all about (…) The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.

Alan Kay, source.

Alan Kay designed in the 1970s the Smalltalk programming language, based on these concepts. And after reading the texts above, it becomes clear that to a large extent, the Microservices architecture is an implementation of Alan Kay’s ideas of messaging, decoupling and abstraction, taken to the extreme.

To achieve the current state of Microservices, though, many other breakthroughs were required. During the 2000s, the emergence of XML, the SOAP protocol, and its related web services made the term “Service Oriented Architecture” a very common staple in architectural discussions. The rise of Agile methodologies made the industry pivot towards the REST approach instead of SOAP. During the last decade, the rise of DevOps and the rise of containerization through Docker and Kubernetes finally enabled teams to deploy thousands of containers as a microservice architecture, thanks to the whole catalog of Cloud Native technologies.

Pros and Cons

If Microservice architectures are so complex, why use them? It turns out, they can bring many benefits:

• Since each component can be completely isolated from the others, once teams agree on their interfaces they can be developed, documented, and tested thoroughly, 100% independently from others. This allows teams to move forward in parallel, implementing features that have 0% chance of collision with one another.
• Teams are also encouraged to choose the programming language that fits best the particular task that their microservice must fulfill.
• Since they are by definition “micro” services, they are designed to be quickly launched and dismissed, so that they only intervene when required, making the whole system more efficient and responsive.
• The size of microservices also allows for higher availability, since it is possible to have many of them behind a load balancer; should any instance of a microservice fail, it can be quickly dismissed and a new one instantiated in its place, without loss of availability.
• Systems can be updated progressively, with each team fixing bugs and adding functionality without disturbing the others. As long as interfaces are respected (and eventually versioned) there is no reason for the system to suffer from updates.

But there are many reasons not to choose the Microservices architectural approach; among the most important:

• Performance: a system built with Microservices must take into account the latency between those services; and in this respect, Monolithic applications are faster; a function call is always faster than a network call.
• Team maturity: Microservices demand a certain degree of experience and expertise from teams; those new to Microservices have a higher chance of project failures.
• Cost: creating a Microservices system will be costlier, if anything, because of the overhead created by each individual service.
• Feasibility: sometimes it is simply not possible to use Microservices, for example when dealing with legacy systems.
• Team structure: this is a decisive factor we will talk about extensively later.
• Complexity: it is not uncommon to end up with systems composed of thousands of concurrent services, and this creates challenges that we will also discuss later.

I would like to talk now about the last two points, which are in our experience the biggest issues in Microservice implementations: team structure and the perception and management of complexity.

Conway’s Law

One of the decisive factors that constrain teams’ ability to implement microservices is often invisible and overlooked: their own structure. Again, this phenomenon is not new. In 1968, Melvin A. Conway wrote an article for the Datamation magazine called “How Do Committees Invent?” in which the following idea stands out:

The basic thesis of this article is that organizations which design systems (…) are constrained to produce designs which are copies of the communication structures of these organizations.

Melvin Conway, source.

There is extensive evidence, both anecdotal and empirical, of this fact through research.

The corollary of this principle is the following: the choice of a Monolithic versus a Microservices architecture is already ingrained in the very hierarchical chart representing any organization.

One of the services we offer at VSHN tackles, precisely, this very issue. In our “DevOps enablement workshop” we evaluate the degree of agility of organizations, and the extent and improvements that DevOps could bring. Based on that information, we reverse engineer their structure through Conway’s Law in order to find a starting point for their digital transformation.

Complex vs. Complicated

Another important point is the distinction of “Complex” versus “Complicated”, as these two words can sometimes be confused with one another in everyday language, and to make things more difficult, the word “Simple” can be used as an antonym of both.

“Complex” is borrowed from the Latin complexus, meaning “made of intertwined elements”. Complexus is itself derived from plectere (“bend, intertwine”). This word has been used since the XVI century to qualify that which is made of heterogenous elements. It shares the same root (plectere) with the medical term “plexus” meaning “interlacing” and used since the 16th century as a medical term for “network of nerves or blood vessels”.

(Source: Dictionnaire historique de la langue française by Alan Rey)

On the other hand, “Complicated” has a similar origin but a different construction: it comes from the Latin complicare, literally meaning “to fold by rolling up”. Figuratively speaking this was taken as close to the notion of embarrassment or awkwardness. The word is composed of the word plicare which means “to fold”. Watches commonly known as “Complications” (such as the Patek Philippe Calibre 89, the Franck Muller Aeternitas Mega and the “Référence 57260” de Vacheron Constantin) are, well, complicated machines by definition.

In short, “Complex” and “Complicated” stem from slightly different roots: the Latin root plectere (“to intertwine”) in the former, and plicare (“to fold”) for the latter. Complex conveys the idea of a network of intertwined objects, whose state and behavior are continuously altered by the interaction with their peers in said network. The word complicated implies an intrinsic apparent “obscurity” through folding unto itself, inviting to an “unfolding” discovery process.

Or, put in another way: individual Microservices should not be complicated, but a Microservice architecture is complex by definition. Monoliths, on the other hand, tend to become very complicated as time passes. And of course, neither are simple.

History shows that software developers have a passionate relationship with complication; complicated systems are great to brag about on Hacker News, while maintainers also cry about them in private.

A “best practice” in this context has one and only one basic job: to help engineers translate the complicated into the complex. Most software-related disasters are caused by a simple fact: because of deadlines, organization, tooling, or just plain ignorance, software developers have a tendency to build complicated, instead of complex, systems.

This is another point we take care of in our DevOps Workshop, through the evaluation of the current assets (not only source code, but also current databases schemas, security requirements, network topologies, deployment procedures and rhythms, etc.)

Migrate or Rewrite? Equilibrium

The complication of Monoliths by itself is not problematic; it makes for tightly bound systems, which tend to be very fast, because as we said, a function call is faster than a network call. After all, we have been building very successful monoliths in the past. But experience shows that they tend to present problems regarding availability and scalability. Microservices represent a diametrally oposed approach, based on complexity rather than complication, but one that solves those issues.

There is a tension, then, between complexity and complication on one side, and organizational forces on the other. Put in other words, there is a tension between monolithic and microservices systems on one side, and more or less hierarchical structures on the other. Achieving equilibrium between these forces is, then, the engineering challenge faced by software architects these days.

Many teams face today the task, either requested internally (from their management) or externally (through customer demand or vendor requirements) of migrating their monoliths into Microservice-based architectures. Architects can thankfully apply a few techniques to find an equilibrium:

1. Start your migration path knowing that very often one does not need to migrate the whole application to Microservices. Some parts can and should remain monolithic, and in particular, proven older systems, even if written in COBOL or older technologies, can still deliver value, and can play a very important role in the success of the transition.
2. Identify components correctly, so that when isolated they will be neither only functionality-driven, nor only data-driven, nor only request-driven, but rather driven by these three factors (functionality, data, and request) at the same time. Pay attention to the organizational chart, and use that as a basis for the decomposition in microservices.
3. Remember that network bandwidth is not infinite. Some interfaces should be “chunky”, while others should be “chatty”. Plan for latency issues from the start.
4. Reduce inter-service communication as much as possible, which can be done in many ways:
   1. Consolidating services together
   2. Consolidating data domains (combining database schemas or using common caches)
   3. Using technologies such as GraphQL to reduce network bandwidth
   4. Using messaging queues, such as RabbitMQ.
5. Adopt Microservice-friendly technologies, such as Docker containers, Kubernetes, Knative, or Red Hat’s OpenShift and Quarkus.
6. Implement an automated testing strategy for each and every microservice.
7. Standardize technology stacks around containers & Kubernetes, and create common ground for a true microservice ecosystem within organizations.
8. Automate all of your work as much as possible, knowing that the effort for automation (DevOps, CI/CD) can be amortized over many services, and becomes thus a net investment in the long run.

As mentioned previously, we regularly help organizations in their digital transformation towards microservices, Kubernetes, OpenShift, DevOps, CI/CD, GitLab, and DevOps in general, to support your teams with the tooling they will need in the future. Borrowing Henny Portman’s Team Topologies concept, VSHN can support both as an “Enabling Team” (DevOps workshop, consulting) and a “Platform Team” (Kubernetes/OpenShift) to build microservices on, ensuring stability and peace of mind.

Conclusion

Going beyond the hype, Microservice architectures bring great benefits, but can become huge challenges to software teams.

The best way to tackle those challenges consists in reverse engineering Conway’s Law, and start by the analysis of the human organization of your teams first. Make them independent, agile, and free to choose the best tools for their job. Encourage them to negotiate with one another the interfaces of their respective components.

Let us create and run complex, not complicated, systems. We cannot get away from complexity; that is our job as engineers. But we can get rid of the complicated part.

As a closing thought, I will quote my former colleague and lifetime friend Adam Jones, an independent IT consultant from Geneva: in order to achieve success with the Microservice architecture, you must embed structure in your activities, and remove it from your hierarchy. It is not about making the structure go away; but instead, moving it to where it does the most good.

On the evening of August 25th, 1991, Linus Torvalds wrote a short message on the comp.os.minix Usenet newsgroup:

I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones.

Fast-forward 30 years, and to make a long story short, well: Linux is now both big and professional. It’s nearly everywhere, hidden behind the shiny graphical user interface of your smartphone or tablet (well, except for iOS users), in your gaming box, in your car, and even on Mars.

I discovered Linux as a young boy still at school, in 1998. It was in a book store where I was attracted by a shiny package (I had no idea what Linux was) called “SuSE Linux 6.0” (nowadays available for download from archive.org), and since then I couldn’t stop working with it.

Over time, I got more and more into Linux, and it became an important hobby, with all my PCs running Linux, and also on my home server under my desk at home. Many years later, in 2011, I could finally start working full-time as a Linux administrator. My desktop computer has been only powered by Linux since around 2004, and I’ve been a big fan of KDE ever since.

Today, the Linux Kernel is more present, yet less visible than ever. We interact with it on containers and in VMs on the cloud, and it gets more and more abstracted away, deep down in serverless architectures, making the Kernel even more invisible than ever before. Albeit out of sight of most users, Linux has become much more solid, mature, and pervasive, and does its great job behind the scenes without interruption.

Linux empowers VSHN at all levels; not only it is the dominating technology we use every day, empowering Kubernetes, containers, and cloud VMs, but it is also the operating system that the majority of VSHNeers (around 66%, according to an internal poll) use for their daily job, the remaining third using macOS or Windows.

Some numbers: of those two thirds of VSHNeers that use Linux every day in their laptops, 61% chose Ubuntu (or one of its various derivatives); 17% use Arch, 11% Fedora, and others use Debian, Mint, and other distributions. Some even contemplate switching to Qubes OS soon! As for desktop environments, around 35% use GNOME, 25% use KDE, 20% use i3, and 6% use Cinnamon.

Each one of us VSHNeers has a unique feeling about Linux; here are some thoughts about what Linux means to us:

Before using Linux, I was primarily focused on how to use and work with computer systems. With the switch to Linux I started to understand how they actually work.

What I really appreciate about Linux is that it’s (still relatively) lightweight, powerful, transparent and adaptable. I do heavyweight gaming and video livestreaming on the same OS that runs my file servers and backup systems (not all on the same machine, don’t worry). Even my car and my television run Linux! This universality combined with the permissive licenses means that whenever one industry improves Linux (the kernel), every other industry profits.

I originally just wanted to play Minecraft with my friends. Suddenly I had to learn how to host this on a Linux server, which then turned into a fascination on how things work on the backstage. It’s the urge to see how our modern society works!

Linux is the operating system of our software-driven world.

On to the next 30 years of Linux!

One of the most difficult subjects in the world of Kubernetes is storage. In our day-to-day operations we’ve often had to choose the best storage solution for our customers, but in a changing landscape of requirements and technical offerings, such choice becomes a major task.

Faced to many options, we decided to benchmark storage solutions in real life conditions, to generate the data required for a proper decision. In this article we’re going to share with you our methodology, our results, and our final choice.

The chosen storage providers for this evaluation were:

• Ceph on Rook and OpenShift Data Foundation (previously known as OpenShift Container Storage)
• Longhorn
• Gluster (benchmarked on APPUiO Exoscale OpenShift 3.11)

All of these benchmarks (except Gluster) were run on an OpenShift 4.7 cluster on Exoscale VMs.

We benchmarked Ceph both with unencrypted and encrypted storage for the OSDs (object-storage daemons). We included Gluster in our evaluation for reference and comparison only, as that’s the solution we offered for storage on OpenShift Container Platform 3.x. We never intended to use Gluster as the storage engine for our new Kubernetes storage cluster product.

Methodology

We first created a custom Python script driving kubestr, which in turn orchestrates Fio. This script performed ten (10) iterations for each benchmark, each of which included the following operations in an isolated Fio run:

• Read iops
• Read bandwidth
• Write iops, with different frequencies of calls to fsync:
  • no fsync calls during each benchmark iteration
  • an fsync call after each operation (“fsync=1”)
  • an fsync call after every 32 operations (“fsync=32”)
  • an fsync call after every 128 operations (“fsync=128”)
• Write bandwidth, with different frequencies of calls to fsync:
  • no fsync calls during each benchmark iteration
  • an fsync call after each operation (“fsync=1”)
  • an fsync call after every 32 operations (“fsync=32”)
  • an fsync call after every 128 operations (“fsync=128”)

This is the Fio configuration used for benchmarking:

[global]
randrepeat=0
verify=0
ioengine=libaio
direct=1
gtod_reduce=1
[job]
name=JOB_NAME     (1)
bs=BLOCKSIZE      (2)
iodepth=64
size=2G
readwrite=OP      (3)
time_based
ramp_time=5s
runtime=30s
fsync=X           (4)

1. We generate a descriptive fio job name based on the benchmark we’re executing. The job name is generated by taking the operation (“read” or “write”) and the measurement (“bw” or “iops”) and concatenating them as “OP_MEASUREMENT”, for example “read_iops”.
2. The blocksize for each operation executed by fio. We use a blocksize of 4K (4kB) for IOPS benchmarks and 128K (128kB) for bandwidth benchmarks.
3. The IO pattern which fio uses for the benchmark. randread for read benchmarks. randwrite for write benchmarks.
4. The number of operations to batch between fsync calls. This parameter doesn’t have an influence on read benchmarks.

If writing to a file, issue an fsync(2) (or its equivalent) of the dirty data for every number of blocks given. For example, if you give 32 as a parameter, fio will sync the file after every 32 writes issued. If fio is using non-buffered I/O, we may not sync the file. The exception is the sg I/O engine, which synchronizes the disk cache anyway. Defaults to 0, which means fio does not periodically issue and wait for a sync to complete. Also see end_fsync and fsync_on_close.

Fio documentation

Results

The following graph, taken from the full dataset for our benchmark (available for download and study) shows the type of comparison performed across all considered solutions.

The table below gives an overview over the data gathered during our evaluation (each column shows mean ± standard deviation):

Unencrypted Rook/OCS numbers are from OCS, encrypted Rook/OCS numbers from vanilla Rook.

Conclusion

After careful evaluation of the results shown above, we chose Rook to implement our APPUiO Managed Storage Cluster product. Rook allows us to have a single product for all the Kubernetes distributions we’re offering at VSHN.

We have released the scripts on GitHub for everyone to verify our results, and published even more data in our Products documentation. Feel free to check the data, run these tests in your own infrastructure, and of course, your pull requests are more than welcome.

Welcome to the third and last installment of our “Networking Security” blog post series. If you didn’t read the previous ones yet, go ahead and read the first two parts:

• Part 1: What is TLS?
• Part 2: Verifying and Connecting
• Part 3: Trust and Certificate Chains

In this article we’ll talk about trust, and how it’s implemented thanks to certificate chains.

Chain of Trust

As explained in the previous posts, a server certificate can be signed by a CA certificate in order to confirm their identity. We trust that the CA has verified the contents of the server certificate before signing it, and hence we trust that the contents of the server certificate are valid.

Now, since a certificate that is used to sign other certificates, can in turn be signed by another certificate, we can build a whole “chain of trust”:

• At the top, there is the Root Certificate (also called the Anchor). It is signed by itself (self-signed).
• In between, there may be one or more Intermediate Certificates. They are signed by either other intermediates or the root certificate.
• On the bottom is our server (or client) certificate, called the Leaf Certificate.

In order to verify trust of a leaf certificate, we follow the “certificate chain” upwards until we reach a root certificate we already know and trust.

Root Certificates

In order to know which root certificates to trust, we must have a “trust store” containing all the root certificates. This store is usually provided by the operating systems, but some clients (most notably web browsers) have their own “trust store”.

In order for a CA to have its root certificate accepted into the trust store of an operating system or browser, they must meet certain quality criterias. They must prove that they take CSR validation seriously and that they have measures put in place to keep their private key confidential.

Having their private key stolen by an attacker is the absolute worst-case scenario for any CA. It would mean that the attacker could create and sign ANY certificate using the CA’s cert! And since the CA’s certificate is trusted by all operating systems, clients connecting to the attacker would be none the wiser!

The only way to react to a compromised key is to rotate it, that is, to create a new one using a new private key. This however would mean that the certificate also has to be replaced in all trust stores. Since this can take a long time for some operating systems or clients, CA’s usually only use Intermediate Certificates to sign your certificates, and store the root key somewhere safe (and by safe I mean printed out on a piece of paper and stored in an actual safe.)

Intermediate Certificates

As explained, CA’s usually use an Intermediate Certificate to sign your CSR’s. An Intermediate Certificate, in turn, can be signed by either the Root Certificate or another Intermediate Certificate. This means that, in theory, there could be any number of intermediate certificates between your server (leaf) certificate and the Root. In practice, however, there is usually only one or two intermediates.

CA Bundles

In order to verify a certificate, a client must possess the certificate that was used to sign it. So in order to verify the whole chain, it must have each intermediate certificate as well as the root certificate. The root certificate we already have in our trust store. But what about the intermediates?

When a CA has signed your CSR and sends you back the certificate, they will also send you all the intermediate certificates between your server cert and the Root. This is called the “CA bundle” or “CA chain”.

When we install our Server certificate in our web server, we also provide it the CA bundle. And when a client connects to the server, the server will send along the bundle with the certificate. This is safe because as long as one of the certificates in the chain is signed by a trusted root certificate, we know that we can trust the whole chain.

Note that the root certificate itself must NOT be part of the chain. It MUST come from the client’s trust store! Unfortunately some CA’s have the bad habit to include their root in the CA bundle, so make sure to remove it before configuring your webserver.

If you check your website against SSL Labs’ SSL Server Test, it will warn you if either intermediates are missing or the root (“anchor”) is included.

Closing Words

Congratulations, you’ve made it! You should now have some basic understanding about TLS/SSL certificates, what they do, and how they work!

Thanks to various browser vendors’ efforts, HTTPS has become the norm when surfing the web in recent years. And now that you’re armed with knowledge, there is no excuse anymore to serve your websites unencrypted!

Welcome to the second article of a series of three, about the subject of security in networking applications, TLS, certificates, keys, and more.

• Part 1: What is TLS?
• Part 2: Verifying and Connecting
• Part 3: Trust and Certificate Chains

In the first part, we learnt about TLS/SSL, certificates, CAs and Keys. If you haven’t read it yet, please do so now.

Now that we know all about the pieces of the puzzle, how do they all fit together? Let’s have a closer look!

Issuing Certificates

To begin with, we need to obtain a certificate. The process works something like this:

First, we need a public/private key pair. With it, we can now create a new CSR (short for “Certificate Signing Request”). The CSR contains all the details we want to include in the certificate (the “Subject”) as well as the public key. We then send the CSR to the CA (the “Certificate Authority”).

The CA then verifies the contents of our CSR, and signs it with their own certificate and private key. This results in a new certificate with the details from our CSR, its “Issuer” field set to the “Subject” of the CA’s certificate, and our public key embedded. The CA then sends the new certificate back to us.

And that’s it! We now have a certificate that is signed by the CA, and we have the accompanying private key. We can now install the certificate in our web server.

Establishing Connections

The whole purpose of certificates and keys is to create secure channels through which devices can exchange information, without the risk of snooping by third parties. In other words, security experts call this “being able to send secrets on a postcard”, and all things considered, the analogy is quite appropriate.

So, how do we establish a secure connection with SSL? The following things are required:

1. A (web) server with:
   • A CA certificate;
   • A server certificate;
   • And a server certificate key (with its corresponding passphrase).
2. And a client (usually a web browser) with a list of trusted CA certificates.

With all this in place, we can establish a trusted, secure connection between both:

1. The client opens a TLS connection to the server.
2. The server responds with its certificate.
3. The client verifies that the server certificate was indeed signed by a trusted CA.
4. The client verifies that the server certificate’s CN attribute (part of the “Subject”, remember?) matches the requested URL.
5. The client verifies that the server certificate is not expired.
6. The client can optionally check that the server’s certificate has not been revoked, either by using a CRL (a “Certificate Revocation List”), or a mechanism like OCSP (an “Online Certificate Status Protocol”).

After these steps, the client has established an encrypted connection, and verified that the server it is connected to is indeed the one it claims to be.

The client and the server can now exchange data securely with one another, without the risk of a third party being able to read their interactions, or even worse, manipulating them!

Client Certificates

There is however another use case for certificates, and that is Client Authentication. Remember how in the example above the server’s certificate was used to verify its identity? This works the other way around too!

If we have a client certificate that is signed by a CA known to the server, we can establish a mutually trusted, secured connection.

To achieve this, we need the following things:

1. A server with:
   • A CA certificate;
   • A server certificate;
   • And a server certificate key (with its corresponding passphrase).
2. A client with:
   • A CA certificate;
   • A client certificate;
   • And a client certificate key (again, with its corresponding passphrase).

Once all the things are in place, a mutually trusted, secured connection can be established:

1. The client opens a TLS connection to the server, sending its certificate.
2. The server responds with its own certificate.
3. The client verifies that the server certificate was indeed signed by a trusted root.
4. The client verifies that the server certificate’s CN attribute matches the requested URL.
5. The client verifies that the server certificate is not expired.
6. The client can optionally ensure the server’s certificate was not revoked.
7. The server verifies that the client certificate was indeed signed by a trusted root (our CA certificate in this case).
8. The server verifies that the client certificate is not expired.
9. The server can optionally ensure the client’s certificate was not revoked.

After these steps, both client and server have established an encrypted connection, one in which both parties are who they claim they are. The server can now read the client’s certificate details (mostly the Subject) to identify the client.

And just like previously, our connection is now fully encrypted and all communications flowing through it are confidential.

Coming Up

In the last part of this series, we are going to talk about some technical details about chains, roots, and intermediates. Stay tuned!

Tobias Brunner’s lightning talk at Crossplane Community Day Europe 2021.

Watch the recording

The Crossplane Service Broker exposes Crossplane primitives via the Open Service Broker API. Tobias will introduce the concepts behind the Crossplane Service Broker and demonstrate to the audience how it all works together. By leveraging the Open Service Broker API while coupling it with the powerful concept of Crossplane Compositions it’s very easy to enable users of a platform which exposes Open Service Broker API integration (like Kubernetes Service Catalog or Cloudfoundry) to provision services fully automated. In a demonstration a real use case will be shown how a Redis service can be provisioned using the Open Service Broker API, leveraging the Crossplane Compositions and the Helm provider.

Slides: https://speakerdeck.com/tobru/self-service-provisioning-with-the-crossplane-service-broker

Note: This talk was held at the Crossplane Community Day Europe 2021

Also see: Crossplane – The Control-Plane of the future

Welcome to this first article of a series of three, about the subject of security in networking applications, TLS, certificates, keys, and more.

• Part 1: What is TLS?
• Part 2: Verifying and Connecting
• Part 3: Trust and Certificate Chains

We have noticed that many of our customers struggle to understand how TLS certificates work on a conceptual level. For example, how does “signing” work? What does it even mean? What are a “chain”, an “intermediate” or a “root”? Why do intermediates belong to a chain, but not the root? What is a CSR? How do CSR, Key, and certificate work together? And so on.

This series is an attempt to explain these critical concepts to our customers, starting this first part with the basic vocabulary and knowledge to get started.

HTTPS – TLS – SSL

Let’s talk about what we want to achieve first. When you visit a website via plain old HTTP, an attacker could intercept your request and grab any private information like usernames and password. Additionally, there is no way for you to verify that you indeed connected to the server you intended. An attacker could have modified the DNS response (which is also unencrypted) to send your browser to their server instead. See Man-in-the-middle attack on Wikipedia for more examples.

So in order to verify the identity of the server we connected to, and to make sure nobody except the server and our browser can read the data we exchange, websites these days use TLS (“Transport Layer Security”, or its predecessor SSL “Secure Sockets Layer”) to both authenticate the server and encrypt the traffic.

From a technical perspective, TLS sits between TCP and HTTP on the protocol stack, and while it’s mostly known for being the S (for Secure) part in HTTPS, it’s noteworthy that it can be used for other protocols as well. Some examples:

• Email protocols: IMAPS, POP3S, SMPTS, …​
• Database connections: MySQL, PostgreSQL, MongoDB, etcd, …​
• Data transfer: FTPS
• Telephony: SIPS
• Chat: XMPPS, IRCS, …​

The last thing to note here is that there are different versions of TLS (and SSL), and some of them are not considered secure anymore!

From oldest to newest, at the time of writing (March 2021):

• SSLv1: Insecure, not supported anymore
• SSLv2: Insecure, not supported anymore
• SSLv3: Insecure, not supported anymore
• TLS 1.0: Insecure, deprecated
• TLS 1.1: Insecure, deprecated
• TLS 1.2: OK
• TLS 1.3: OK

Keep this in mind when planning your environment! You wouldn’t want to protect your brand new microservice with outdated security protocols!

By the way, if you want to check which TLS versions a website supports, use SSL Labs’ SSL Server Test. It’s a great debugging tool and will show you a lot of information about the topics of this Blog post series!

Certificates

The next thing we should have a look at are Certificates, or more specifically X.509 v3 public key certificates.

A Certificate is a cryptographically signed piece of information, of which the most important part is the Subject, which identifies who this certificate belongs to, as well as the Issuer. Other attributes include details about when the certificate is about to expire as well as a lot of technical information about the key and signature algorithms used, and so on.

The Subject and Issuer of a certificate are characterized by a set of attributes:

• CN – Common Name
• C – Country
• ST – State
• L – Location (City)
• O – Organisation
• OU – Organisational Unit

Together, those attributes form a Distinguished Name (DN). Most attributes are optional, except for the Common Name. In the case of Server Certificates, the CN must match the address used to connect, for example www.vshn.ch.

The Certificate also contains a Public Key (embedded in the certificate) and its matching Private Key.

The last important feature of X.509 certificates is that they are signed by other certificates (or itself). Once a certificate is signed, its contents cannot be changed anymore.

Signing CAs

The Internet is a big place, so how do we know whom to trust? To solve this issue, the concept of Certificate Authorities came up.

A Certificate Authority (commonly referred to as CA) is a central trusted source of certificates. They have a CA Certificate (also known as the “root” certificate) that is well known and trusted. This certificate is only used to sign other certificates. By signing another certificate, the CA confirms “yes, I trust this certificate and the information in it is correct.”

A related concept is a Certificate revocation list, also known as “CRL”, which is a list of digital certificates that have been revoked by the CA before their scheduled expiration date, and should therefore no longer be trusted.

Now that we have explained what a certificate is, let’s talk about keys.

Keys

Each certificate has its own key. The key makes certificates actually useful; without them, they would not work.

Each key consists of two parts: a public key and a private key. While the public key is embedded as part of the certificate itself, the private key is stored in a different file altogether.

In order to sign another certificate, you need the Private key of the signing certificate (but not of the certificate you want to sign).

When you open a connection to a server, the server also needs the private key of the server certificate in order to authenticate itself and establish encrypted communication.

This whole concept is called Public-key cryptography.

Confidentiality

A very important thing to understand is that the key is the only confidential piece in the puzzle; needless to say, it is the most important piece! So please, keep this always in mind: do not ever exchange keys over unsecured channels, under any circumstances!

On the other hand, the certificate itself is not confidential. The CA certificate in particular must be provided to all clients and servers, in order to verify other certificates.

How can a client ask a CA to sign a certificate? Through a CSR (also known as Certificate Signing Request) which, as the name implies, performs exactly this task.

Coming Up

Speaking about verification and issuing new certificates, we are going to talk about these subjects in detail, respectively in the second and third parts of this series. Stay tuned!

I think it can be said that at VSHN we have a certain obsession with documentation. This goes beyond the requirements set by our customers, or by those imposed by certifications such as the ISO 27001 or ISAE-3402. We have found that as a team we can work better when we can refer to written documentation at all times.

In the past few years we have standardized our documentation toolkit: we have adopted Asciidoctor as the lingua franca for our documentation. We have put in production various documentation websites based on Antora: our Handbook, our Knowledge Base, and the Project Syn documentation are some examples.

That works great for text. But what about diagrams? Our documentation has lots of them, and although embedding PNG files works fine, it does not “scale” very well.

For that reason we have set up an instance of Kroki in production (in APPUiO, of course!), to enable the generation of SVG diagrams based on various formats: PlantUML for UML diagrams, Nwdiag for network diagrams, and Ditaa for simple box-and-arrow kind of diagrams, all produced and shown in a crisp and neat SVG format. We love Kroki! It is such an awesome idea, that it has even been integrated to GitLab as a supported diagramming tool. How cool is that? In general, the Asciidoctor ecosystem has very good integration of diagrams into workflows, and this extends to Antora as well.

I am not going to dive into the benefits of vector vs. raster images; I think the readers of this article are well aware of the bonuses the former has over the latter. SVG diagrams look great in both online and print, but they can be a bit complicated to draw by hand (duh!).

But here’s a confession: I’m not a big fan of editing text-based diagrams either. But that’s just a personal choice. And the truth is, many other non-technical VSHNeers find the whole “diagrams as text” thing a bit arcane.

So that means that we needed a more accessible tool for them. The ideal diagram drawing tool for VSHN must have the following requirements:

• Cross-platform: Most of us at VSHN use Linux, but a sizable number also use Macs and Windows laptops.
• Accessible to non-technical VSHNeers: Not everyone at VSHN is a DevOps engineer.
• Simple to use: If possible, using a point-and-click interface; for example, in our Confluence wiki we have enabled the Gliffy extension, which allows non-technical users to create stunning diagrams.

So we started our search. Leaving aside tools like Inkscape, that are too generic, like Asciiflow, that are too limited, or like Dia, that are too platform-specific, it took us a while to find a tool that would fit our requirements. But we found it!

Please welcome Diagrams.net (previously also known as Draw.io). It is a web application geared specifically to the generation of diagrams, very easy to use and ticking all the boxes that we needed.

And even better, there is a Visual Studio Code extension that allows to edit diagrams locally, directly on your laptop, in Linux, Mac and Windows.

How do we integrate this with Antora? Very simple. In the assets/images folder of our documentation components, create your diagrams using the *.drawio.svg extension. These are automatically associated by Visual Studio Code with the extension, and provides a live editor with all the bells and whistles you might expect.

And then, well, just image::diagram.drawio.svg[] in your documents, git commit and push.

This approach is definitely more accessible to non-technical VSHNeers, making it super easy to edit diagrams online, or locally in Visual Studio Code.

To finish, here’s a Pro Tip™®©: switch the Draw.io extension to the “Atlas” theme, since some lines in the extension lack contrast, and aren’t easily seen when using a dark theme. You can change the Draw.io extension theme with the selector at the bottom right of the screen.

PS: there is an open issue in Kroki to add support for Draw.io diagrams. Can’t wait to see it in action!

Docker Inc., the company behind Docker Hub, has recently announced an enforcement of image pull rate limits for Docker Hub users. This change affects all cloud native installations currently configured to consume container images stored in Docker Hub, one of the most popular image repositories available today. This measure has a direct impact in the deployment of many customer applications currently running on APPUiO Public and Private Clusters.

We are following the situation closely. Following recent announcements, Docker Hub is slowly starting to enforce a pull rate limit of 100 pulls per 6 hours for anonymous (unauthenticated) IP-Addresses, and 200 pulls per 6 hours for authenticated non-paying users. This means that a Kubernetes or OpenShift cluster can only pull 100 images in 6 hours without authentication for all of its users from Docker Hub. During maintenance periods, most of the images of a cluster are pulled at some point, and the reduction of pull rate limits can cause downtime when the images can’t be pulled due to the limit.

This situation should not affect today’s maintenance window. Even though the announced start date of the rate limit was Monday, October 2nd, 2020 the analysis of responses from Docker Hub indicates that the new limits are not yet applied, and will only be enforced during a 3-hour window from 18:00 to 21:00 (CET) tomorrow. Unfortunately at this point we don’t know when the new pull rate limits will be enforced fully.

Also noteworthy, these restrictions do not apply to the internal OpenShift registry included with APPUiO Public or Private, which is completely independent of Docker Hub, nor to Docker users with Pro or Team accounts. Authenticated users with Pro or Team accounts enjoy unlimited data transfer to and from Docker Hub.

VSHN is currently evaluating measures to prevent downtime, and reduce the impact of this situation for our customers. The most appropriate solution at this moment consists in switching to the aforementioned Pro or Team Docker account types. Even simpler, the use of an authenticated user account in a pull secret instead of an anonymous account will double the pull rate limit, and will reduce the risk of downtime considerably. Another possibility consists in migrating images to a different public registry, like Red Hat Quay. Another option is the use of a private registry, such as the OpenShift integrated registry, AWS ECR, GitLab Container Registry, Harbor, or other similar technologies. In particular, AWS has recently announced the future availability (“within weeks”) of a free public registry.

We remain at your service for any enquiry. Contact us if you need more information.

This is about how we manage large numbers of ModSecurity WAF deployments, including CRS and custom rules.

ModSecurity CRS Berne Meetup

At VSHN we’re not only 100% committed to Free and Open Source software, we also collaborate actively with the communities of the software we use, deploy and enhance. This June, we had the pleasure to join the OWASP ModSecurity Core Rule Set Berne Meetup group online who had asked us back in November at yet another Meetup to showcase the way we use ModSecurity and the Core Rule Set (CRS).