We’re very excited to share two fantastic news: starting today, users can self-sign up into APPUiO Cloud and create a new organization, and also send invitations to their colleagues and coworkers to join their APPUiO Cloud organization.

APPUiO Cloud is our Kubernetes „Namespace as a Service“ PaaS based on Red Hat OpenShift. It gives you instant access to a shared, fully managed, ready to use Red Hat OpenShift cluster in just a few minutes. Focus on your application rather than on setting up or managing your cluster, and bring your ideas to production in a breeze. It is a particularly good option for various use cases: independent Cloud Native app developers, mobile app backends, education, or startups of any kind can now get instant access to a ready-to-use, secure OpenShift 4 cluster.

APPUiO Cloud also includes various added benefits available off-the-box:

• K8up to trigger and schedule backups of your applications into external S3 buckets.
• AppCat to quickly provision databases, message queues, and more, directly from the comfort of your YAML infrastructure.
• And a comprehensive help system to guide you in your discovery of your new platform.

Head now to portal.appuio.cloud, open an account, create an organization, invite your coworkers, and start deploying applications on a standards-based, secure, and convenient platform, today.

Last September, we announced our new AppCat service. Since then, we’ve expanded it to offer object storage provisioning in both cloudscale.ch and Exoscale. Today, we’re thrilled to announce the immediate availability of a new AppCat service: DBaaS by Exoscale, available in the APPUiO Cloud Zone on Exoscale or, on request, on APPUiO Managed clusters on Exoscale.

Getting an Exoscale DBaaS instance is as simple as specifying any Kubernetes object with YAML. A few minutes later, you’ll get access to a freshly provisioned Exoscale DBaaS instance, with the credentials to access it available in a Kubernetes Secret  object. This lets you specify the dependency of an Exoscale DBaaS instance directly within your application deployment.

DBaaS by Exoscale on AppCat has the following outstanding features:

• Available services: Create one or many instances of PostgreSQL, MySQL, Redis, Apache Kafka, or OpenSearch.
• Multiple plans available: Find a variety of plans with different upscale and downscale possibilities, as well as redundancy options.
• Available in all Exoscale zones: DBaaS plans are available in all current Exoscale zones, enabling multi-zone setups and geo-replication.
• Termination protection: Termination protection against accidental deletion is in place for all DBaaS services to keep your databases safe.
• Automatic backup policy: Backups are automatically done daily. The daily backup frequency and retention depend on the chosen plan.
• Your data stays In Europe: All data is stored in the country of your chosen zone, fully GDPR-compliant. DBaaS is available in every zone across Europe.

Some crunchy technical details:

• The DBaaS offering by Exoscale is powered by Aiven, one of the leading European companies for managing open-source data infrastructure in the cloud. This partnership offers Exoscale and VSHN customers an integrated environment for their complete cloud infrastructure – without any security compromise. All involved companies are GDPR-compliant to ensure the highest data protection standards.
• Using open-source projects ensures customers are not locked into a vendor and always enjoy the latest technology standards.
• This service is made available by our Crossplane providers for cloudscale.ch and Exoscale. It uses a Project Syn Commodore Component to deploy the Crossplane XRDs and Compositions to the APPUiO clusters.

Learn more about AppCat DBaaS by Exoscale, including its pricing and availability, on our product website, and then learn how to use it on our AppCat documentation website.

APPUiO Cloud users are now able to monitor their applications on their own!

APPUiO Cloud has been upgraded to OpenShift 4.11 and thanks to User Workload Monitoring, users are now able to monitor their applications on their own and get notified when something needs attention.

APPUiO Cloud now runs on OpenShift version 4.11. Both zones – cloudscale.ch LPG 2 and Exoscale CH-GVA-2 0 – have been updated in the past two weeks.

Users might notice some warnings when deploying things to APPUiO Cloud or when updating existing workloads. Those warnings come from Pod Security Admission. The Kubernetes community is changing how pod security is handled, and Pod Security Admission got introduced with Kubernetes 1.24, which is part of OpenShift 4.11. You will find more on the subject at our Pod Security Admissions documentation page.

As far as we understand, future versions of Kubernetes will change again and those warnings will likely become errors. Therefore we suggest to look into them sooner rather than later.

…and now that the administrative stuff has been taken care of, let us continue with the exciting stuff:

Thanks to functionality introduced in OpenShift 4.11, we are now able to offer you User Workload Monitoring. This feature allows APPUiO Cloud users to monitor their applications on their own.

Users can now collect metrics of applications and write alerts based on the collected metrics. A how-to on how to do so can be found at Monitor Application With Prometheus. Additionaly, users can now access metrics relevant to their application, which are collected by the cluster monitoring, and thus so far were not accessible to end users. Monitor PVC Disk Usage explains how this can be used to monitor disk usage of persistent volumes. It is also possible to access metrics on memory, CPU usage and many things more.

Users who are interested in nice graphs can install an instance of Grafana for themselves and build informative and nice looking dashboards depicting the metrics of their applications. See the Custom Grafana documentation for details how this can be achieved.

Others do not want to constantly look at dashboards to see whether something is broken, chosing to get notified when this is the case instead. This is also possible, as it’s now an option to not only write your own alerts but also individual alert routes and get email or chat messages when something is in need of attention. Configure Alert Receivers teaches how to do this.

User Workload Monitoring is a long-awaited feature, and we hope that it will make an impact for many APPUiO Cloud users.

What’s next?

We are currently working towards user self registration for APPUiO Cloud. This allows new users to register for APPUiO Cloud all by themselves without having to talk to one of our sales representatives. A milestone within that initiative will be User invitation.

We built APPUiO Cloud to support performance-sensitive production workloads. But that performance also came along with a price tag, one that users had to pay even for non-production workloads — like your development environments — or non-time critical workloads — like your background cron jobs.

We have heard this concern from our customers, and we are pleased to introduce today the concept of Node Classes. They allow you to schedule workloads on nodes with different specifications and costs. See the APPUiO Cloud documentation for more details about node classes.

On the cloudscale.ch – LPG 2 zone, you can now choose between „Flex“ and „Plus“ classes. „Plus“ nodes have the same performance-optimized characteristics all nodes on that cluster had so far. „Flex“ nodes will be a less capable but more affordable option. A more detailed explanation of these two classes is available on the documentation website, where you can also find the price table.

Workloads in our existing namespaces will continue to run on the more performant „Plus“ node class. The default for all new namespaces will be „Flex.“ Check the documentation to learn how to schedule your workloads on the desired node class.

Would your workload profit from nodes optimized for it? Please let us know through our roadmap. Some nodes with GPU support, maybe?

What’s next?

Soon we will release User Workload Monitoring on APPUiO Cloud. This allows users to monitor their applications, write alerts, and receive notifications when those alerts fire. For instance, when your persistent volume runs out of space.

Setting appropriate limit and request values in your deployments is not an easy task. To help our customers, we have enabled the use of the VerticalPodAutoscaler object on all APPUiO Cloud zones. This blog post will show you how to use it.

To use the VerticalPodAutoscaler object, you need an APPUiO Cloud project with some payload deployed and running. The Vertical Pod Autoscaler project on GitHub contains some sample YAML files that deploy a very simple demo application to your project.

Here is the YAML required to create a VerticalPodAutoscaler that will analyze the resource consumption of the vpa-example deployment:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vpa-recommender
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       vpa-example
  updatePolicy:
    updateMode: "Off"

The VPA requires a few moments to gather data and provide recommendations from it. After some time, during which your deployment should have been running in order to gather meaningful data, run the following command and you should see an output similar to this in your terminal:

$ oc get vpa vpa-recommender --output yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  annotations: …
# …
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vpa-example
  updatePolicy:
    updateMode: Auto
status:
  conditions:
  - status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: fortune-container
      lowerBound:
        cpu: 25m
        memory: 262144k
      target:
        cpu: 203m
        memory: 262144k
      uncappedTarget:
        cpu: 203m
        memory: 262144k
      upperBound:
        cpu: 71383m
        memory: "6813174422"

You should analyze with care the values provided by the autoscaler for your deployment. Don’t blindly apply its recommendations; let your application run for a while and study the numbers closely.

Some tips for your analysis, all inside the status.recommendation part of the response:

• The .containerRecommendations[*].target value could be considered indicative for request values.
• The .containerRecommendations[*].upperBound value could be used as an indication to set limit values.

For more hints, the OpenShift dashboard shown on this page of the APPUiO Cloud documentation shows utilization numbers for both CPU and memory limits. Those values function as a suitable supplementary information source and must be taken into consideration.

The APPUiO Cloud documentation has more information about the VerticalPodAutoscaler object.

Wir freuen uns, die sofortige Verfügbarkeit des VSHN Application Catalog (oder kurz „AppCat“) auf APPUiO Cloud bekannt zu geben.

Der VSHN AppCat ist ein Cloud Native-Marktplatz, welcher Services von Anbietern wie Amazon AWS, Google Cloud, Microsoft Azure, Aiven.io, Exoscale oder cloudscale.ch sowie Managed Services von VSHN anbietet. Diese Services können ganz einfach als Kubernetes-Ressourcen angefordert und in jeden GitOps- oder CI/CD-Workflow integriert werden. Bestelle Services so, wie du deine Anwendung auf Kubernetes bereitstellen würdest! Mit dem VSHN Application Catalog konzentrierst du dich auf deine Anwendung, und wir kümmern uns um das Bootstrapping und den Betrieb der benötigten Services.

Der erste AppCat-Services, der ab sofort verfügbar ist, ist der Object Storage Bucket-Services in der APPUiO Cloud Zone LPG 2. Er ermöglicht es Entwicklern, schnell S3-kompatible Storage Buckets in cloudscale.ch bereitzustellen. Wir werden den Services bald auch in der Exoscale CH-GVA-2 0 Zone aktivieren.

Wir werden in Kürze weitere AppCat-Services einführen. Lass uns wissen, welche anderen Services du gerne in AppCat sehen würdest und ob du daran interessiert wärst, AppCat in deinem eigenen APPUiO Managed Cluster zu betreiben.

Weitere Informationen über AppCat findest du auf der Website unserer Produkte und in der APPUiO Cloud Dokumentation.

AppCat basiert auf Crossplane und Project Syn und ist 100% Open Source! Schau dir die Projekte provider-cloudscale, provider-exoscale und component-appcat auf GitHub an.

This blog post is adapted from the speech about APPUiO Cloud given at the Berner Architekten Treffen in June 2022. With this text, we would like to give you an idea of how we built this new product using DevOps as a guiding philosophy.

VSHN

Before we start, I’d like you to introduce VSHN to those who have never heard of it before. We’re a company of 50 people based in Zürich, founded in 2014 with the objective of providing companies with DevOps services of different kinds. The slogan of VSHN is, actually, „The DevOps Company.“ We embrace the DevOps philosophy completely, and as you’ll see today, we use its principles and ideas in everything we do.

What does VSHN do? We provide various services and products:

• We offer „DevOps as a Service“ (or DaaS), with a full team of Kubernetes and cloud experts ready to monitor your applications 24/7, in every hyperscaler, all over the world.
• We help companies become self-sufficient and cloud-enabled; we help software engineers „build bridges“ between Dev and Ops, building CI/CD pipelines in various platforms such as GitLab, OpenShift, or Jenkins.
• We are Kubernetes specialists, to the point that our strategy is 100% oriented towards Kubernetes; all of the code we write, and all of the systems we assemble, for us or others, are meant to be run on Kubernetes.

Aside from our technological choices, the important thing to know about VSHN is that we have chosen to drive the growth of our company in various ways that are completely nonstandard:

• We use Sociocracy as a management and growth framework. This means that all decisions, and I mean all of them, happen through consensus among all VSHNeers (that’s how we call ourselves, by the way.)
• We have created a Handbook, freely available online, which in printed form takes 563 pages, explaining everything we are and do at VSHN with quite an incredible level of detail. I invite you to check it out and learn everything there’s to learn about us.
• We have decided as a company not to grow through venture capital, instead relying upon the good old method of organic growth. We have been self-funded since day one (2014), and we have been consistently profitable and growing since 2017.
• We are „The DevOps Company“, and as such, we embraced the DevOps mantra completely. Everything we do is automated as much as possible, freeing our brains to think.

All of these choices have shaped our culture in ways that are really not common at all, and have interesting consequences in our day-to-day operations. For example, our hiring policies are different from those of most IT companies; not only we do pay attention to the IT skills of those who want to join our team but place a very high degree of attention on the human factor. We want people to feel great at VSHN, and one of the primary factors we evaluate during our hiring process is the „likeness“ of the person, that is, how much would we like to work with them every day?

Another interesting consequence of how VSHN works is that we had embraced remote and asynchronous working well before the pandemic; when the Bundesrat mandated everyone to work from home in March 2020, we simply stayed home and continued working as if nothing had happened. The important bit of information here is the asynchronous word; not so much that we work remotely, but that we work in a non-synchronous way. This particular mindset has shaped our company greatly.

APPUiO

But let’s get back in time a little bit, and see how APPUiO Cloud came to be.

In 2016 VSHN and Puzzle ITC (an IT and software consulting company with offices in Bern and Zurich) launched a joint venture called APPUiO. This word has many meanings: not only it is a word in Esperanto, meaning „support“, but it is also composed of „App“ (for application) and „Ujo“ which is Esperanto for „container.“

APPUiO consists of a series of products built around Red Hat OpenShift.

For those who haven’t heard of it yet, OpenShift is the most widely used Kubernetes-based platform in the enterprise world. It is quite popular with big companies, and it incorporates a hardened and highly available Kubernetes cluster surrounded by lots of relevant software: a container repository, a management console, and CI/CD pipelines, with a very nice and professional GUI on top.

We decided we wanted to be a part of the OpenShift market, but we also realized that installing and operating OpenShift is a huge endeavor, and many companies could not use OpenShift because of the lack of staff or budget. So we decided to join forces with Puzzle ITC.

APPUiO is our response to the complexity of Red Hat OpenShift. With APPUiO, customers can get a ready-to-use cluster, together with the know-how of VSHN and Puzzle ITC. We at VSHN specialize in the setup and maintenance of OpenShift clusters; we have been operating OpenShift clusters since version 3. Puzzle ITC is a specialist in the creation of software solutions for OpenShift, which is something we don’t do. Together, the APPUiO team can help companies make the most out of their OpenShift investment.

Flavors of APPUiO

APPUiO has been historically available in various forms:

• APPUiO Public, based on OpenShift 3, was the first Swiss-based shared OpenShift cluster available to customers. It is a shared platform, where customers can run their projects without having to care about management or anything else. There were APPUiO Public clusters running in various cloud providers, such as Cloudscale in Switzerland and AWS in Germany. Customers can choose whichever they prefer, and deploy their containers there in a few minutes.
• APPUiO Managed is the next step for organizations that are not comfortable running their workloads on a public platform. With APPUiO Managed, they get their own OpenShift cluster, for their exclusive use, and Puzzle ITC and VSHN take care of the operations of the cluster transparently for their users.
• APPUiO Self-Managed is the final step in the evolution of organizations: with APPUiO Self-Managed, organizations not only get an OpenShift cluster „keys in hand“ and ready to run, but we teach their IT teams how to manage and maintain the cluster by themselves. We gradually „fade in the background“ and provide help, until at some point they become completely independent. This is a very interesting option for big corporations with their own IT teams.
• APPUiO Cloud is the latest offering in the APPUiO family, officially launched in November 2021.

APPUiO Cloud

What is APPUiO Cloud? Simply put, APPUiO Cloud is the successor of APPUiO Public. Given the major architectural changes between OpenShift 3 and 4, instead of migrating our APPUiO Public infrastructure to OpenShift 4, we decided to create a new project from scratch, and we gave it a different name and even a different visual identity.

We notified our APPUiO Public customers of the upcoming phasing out of the service, with an offer to help them migrate their payloads to APPUiO Cloud. And on September 1st, 2022 we decommissioned the last APPUiO Public cluster in existence.

As said previously, APPUiO Cloud is based exclusively on OpenShift 4. At the moment we have two APPUiO Cloud zones available to our customers:

• cloudscale.ch in Kanton Aargau
• Exoscale au Canton de Genève

We plan to open more regions in the future, as required and following the demand from our customers.

We started working on APPUiO Cloud in the Spring of 2021, and we released it to the public in the Autumn of that year. We reused lots of code and infrastructure we had created for our work previously:

• K8up is a Kubernetes backup operator that has been picked up by the Cloud Native Computing Foundation as a sandbox project.
• Project Syn is a suite of tools that allow for the remote management of Kubernetes clusters of any kind, from a central location using a GitOps philosophy and workflow.

Who is APPUiO Cloud for?

APPUiO Cloud, just like its predecessor APPUiO Public, is meant to be an „entry-level“ product, catering to the „long tail“ of OpenShift customers, who might be interested in getting access to a working OpenShift cluster without the hassle of installing and operating it.

As such, we identified a few target groups:

• Cloud Native App Development: Instant access to a namespace on a running OpenShift cluster. No provisioning infrastructure in a cloud provider, no installing of OpenShift, no waiting until a new cluster is provisioned for you. Just log in, create a namespace, and you’re ready for oc deploy your new application.
• MVP: Spin a new OpenShift namespace on APPUiO Cloud, deploy your MVP, and go back to raising more venture capital.
• DevOps & CI/CD Pipelines: Run your CI/CD pipelines, deploy, and preview your application on a running OpenShift cluster right now before going live.
• Machine Learning: Use it to train a machine learning model with new data.
• Production App Hosting: Host your application in minutes in production.
• Mobile App Backends: For iOS or Android developers needing to deploy back-ends on a scalable, trusted environment.
• Education: Let students experience the full power of a real OpenShift cluster with their own individual namespace.
• Technology Trial: For users interested in APPUiO Managed cluster, but needing to hedge the risks of trying out new technology.
• Resellers: Resell APPUiO Cloud and let us manage the cluster for you.

Features

What is included in APPUiO Cloud?

• Instant On: Get your own OpenShift namespace in minutes, ready to use.
• Pay-per-use: Only pay for the resources you actually use.
• User Management: Organize your namespaces in teams and organizations, and assign users to those teams; control who can access which namespaces at a glance.
• Backup: Backup all your work with the pre-installed K8up operator.
• Pre-Installed and Configured Operator: APPUiO Cloud provides the following OpenShift operators pre-installed and pre-configured, ready to be used:
  • K8up: Kubernetes Backup Operator.
  • Cert Manager: X.509 certificate management for Kubernetes.
• Community Support: Need help? Check out our APPUiO Cloud forums and community chat. For those needing more help, there are support packages available at extra cost.

Conditions

Because it’s a public platform, APPUiO Cloud comes with some gotchas:

• Maintenance Policies: There are two types of maintenance: APPUiO Cloud Zones receive automatic revision updates (for example, from 4.7.1 to 4.7.2). This includes OpenShift and worker node updates. These updates can happen at any time without prior announcement. Upgrades of a minor (for example from 4.7 to 4.8) and major (4 to 5) OpenShift versions are announced in advance, with a description of all possible breaking changes. On request, we can provide you with access to an already upgraded APPUiO Cloud Zone, for you to test your deployment if needed.
• Status Information: We communicate the status of the platform on status.appuio.cloud.
• Resource Availability: APPUiO Cloud is provided without any guarantees of resource availability.
• SLA: Best effort.
• Fair-Use Policy: APPUiO Cloud is a shared platform. Unless otherwise stated, this fair-use principle applies to the use of our services. APPUiO Cloud users must use their resources moderately, so as not to degrade the service level available to other users.
• Privileged Containers: Privileged containers can’t run on APPUiO Cloud.
• Log retention: The OpenShift integrated logging (Elasticsearch / Kibana) retains collected logs for 72h (3 days), after that time-period logs are permanently deleted.
• Other Operators: It is not possible (at the moment) to run other OpenShift operators than the ones we are offering. We evaluate them on a case-by-case basis, following the requests from our customers.

A DevOps way of working

What do we mean by „a DevOps way of working“? Let’s see first what we mean by DevOps, one of those words that can mean anything and everything depending on who you ask.

We think the best people to talk about DevOps are those who wrote „The DevOps Handbook“ and „The Phoenix Project“: Gene Kim and Jez Humble.

In those books, DevOps is usually defined by the „three ways“:

1. The Principles of Flow
2. The Principles of Feedback
3. The Principles of Continual Learning and Experimentation

How did we create APPUiO Cloud following DevOps? We applied these three principles in and out during the whole process, as follows.

Flow

The first thing was to decide where to start, that is, what value stream we wanted to provide first.

That work brought together the Product Documentation. That’s right, the first thing we created through discussion was written documentation of what we wanted to offer.

Why written? Because we work asynchronously. That means that some of us work better at night, while some work better in the morning; having everything written down helped everyone, commenting down drafts of the documents until there was agreement. Agreement from whom? From the Product Owners to the DevOps engineers who would have to maintain the solution at the end.

This way, the operations team knows exactly what is it that’s going to happen. There are no surprises down the hall, and they feel empowered and listened to. All the features of APPUiO Cloud are, simply put, possible to release; either now or later, but they are possible.

The important thing here is that we started by reverse engineering Conway’s Law. That is, we first structured the team that would work on APPUiO Cloud, and then we got to create the system. The end result of this process is that the architecture of APPUiO Cloud, following Conway’s Law, strictly mirrors the structure of our team. We do not fight against Conway’s Law; we embrace it.

The result of this work of architecture can be summarized in three different documentation websites for APPUiO Cloud; you’ve heard right, we have created three different sets of documentation, and we keep them updated every day:

1. System engineer documentation
2. Product owner documentation
3. End-user documentation

We have made all of the documentation publicly available and viewable, even editable because transparency is one of our values at VSHN. We want all of our customers to know exactly we’re doing things the way we do; this, in turn, generates trust in our existing customers and shows our know-how to prospective customers. These three documentation sites are, simply put, great marketing tools!

The principle of Flow requires teams to make work visible, reduce batch sizes and intervals of work, and build quality. We limited work in progress to the strict minimum, and we automated as much as possible of the process.

This automation involves removing the human factor from the maintenance of those clusters as much as possible. One of the key factors for doing this was Project Syn, a suite of tools we started building in 2019 that allows our small team to manage hundreds of clusters from a central location. We created Project Syn as a way to be able to operate our customers‘ assets with a reduced human footprint, but it turned out to be a great way to handle our own work on APPUiO Cloud, too.

Thanks to Project Syn, DevOps engineers can specify and deploy changes to lots of Kubernetes clusters from a central location, using a GitOps strategy; just commit your changes as „infrastructure as code“ to a Git repository, and wait a few seconds until all clusters apply those changes.

We use Project Syn to deploy Kyverno security policies to our APPUiO Cloud clusters so that all regions conform to the same rulebook.

We also configured each of the APPUiO Cloud zones with the mandatory differences between the cloud providers we use; Exoscale and cloudscale.ch do not offer exactly the same features and being able to see those differences in written form allows us to manage those systems, to take decisions for the future, and to inform our customers of any possible tradeoff.

Feedback

We know that APPUiO Cloud is a complex system, built out of complex systems, that are prone to failure at any given time. It is not a matter of „if“, but rather a matter of „when.“

We have built observability and management tools immediately from the start of our work on APPUiO Cloud. We have reused the management infrastructure provided by OpenShift, the same one we were using for our private customers, and we have built APPUiO Cloud to be observable at all times.

Using „everything as code“ as a basis for our work means that every time we fix an issue on the platform, we have to change a configuration file somewhere. This information is later stored in the Git repo, as part of the project history; not only that, but we also update the required documentation files, both internal and external, so that everyone knows (asynchronously and at their own rhythm) what happened, when, where, and most importantly, why.

And when we say „everything as code“, we mean it:

• Policies
• Builds
• Configuration
• Infrastructure

All of this is described in their corresponding files, and versioned in Git repositories. We use GitLab, and its integrated CI/CD pipelines are configured to automatically build, test, and eventually deploy changes as required. Thanks to Project Syn, all of the feedback they bring back to the system is automatically deployed whenever possible, reducing the amount of human brain work required to keep things running.

Even our documentation is automated: we use the Antora documentation generator tool, which can automatically extract and integrate documentation from various sources into a single website, and we use GitLab pipelines for that as well. With this process, engineers only have to update the documentation sources (using the Asciidoc format, very similar to Markdown) and git push their changes. They are immediately picked up, built, verified (we have automatic styling and syntax checks built-in in our pipelines), and deployed.

Continuous Learning and Experimentation

APPUiO Cloud is not, and will never be, finished. It is a product that changes continuously, sometimes in small ways, and sometimes in bigger ones.

To give an example of continuous learning at work in APPUiO Cloud, let’s refer to the current console screen, where in June 2022 you could see a red banner on top with the following text:

Issue with CPU requests resolved. The resolution includes a slight change to the pricing model.

This, as you can imagine, is the result of a learning process. We realized that, in our preparation, we had not designed our CPU request pricing properly. As a result, as soon as the first users started using the platform this year, we realized that some of them were consuming disproportionate amounts of CPU; this was a huge problem, since they were not aware of that, and we would have to cover for those extra costs at first.

We modified the policies in the clusters, made communication with all of our customers, and updated our documentation as follows:

As of 2022-05-01: The underlying infrastructure has a fixed ratio between memory and CPU resources. CPU requests exceeding that ratio will be counted as well. The requested CPU cores will be multiplied with the platforms‘ ratio. This yields an equivalent in MiB. The memory to CPU ratio can be different per zone. See zone listing for the exact values.

(Source)

This was unexpected and unplanned learning; a local discovery that brought a global improvement in APPUiO Cloud for all users. We did cover some of the costs, but we rectified our policies openly and communicated clearly with our customers. The result? Not only did all of them acknowledge and understand the changes, but we didn’t lose a single customer because of this change. This level of cooperation with our customers is one of the things we’re most proud of.

Another learning: the GitLab Kubernetes agent. Cannot be installed globally, but we figured out how to do it on a namespace basis for those who need it.

As part of the continuous learning process of DevOps, we also have bi-monthly updates by the APPUiO Cloud team on a private Zoom call open to all VSHNeers to join, where we explain what’s going on, what are the upcoming changes to the platform, what new features are planned, and what has happened lately. These calls are recorded for internal use, so that (once again) people can re-watch those internal education sessions whenever they want and how often they want.

Tools

This is the toolkit we used to collaborate during the creation of APPUiO Cloud.

• Zoom
• Discourse
• Jira
• Confluence
• Visual Studio Code Live Share
• RocketChat
• Eating own dogfood: all of our internal systems run on APPUiO Cloud, with the exception of the APPUiO Cloud status page, of course 🙂

Team Size

The current team in charge of APPUiO Cloud is called Tarazed.

• 1 project manager/product manager
• 1 product owner
• 3 DevOps engineers core team

Timeline

This is a rough timeline of key events during the creation of APPUiO Cloud, with links to their corresponding blog posts.

• 2021-02-19: First discussions about product definition for „APPUiO Public 2.0“
• Spring/Summer 2021: Design, development, test
• 2021-06: Benchmarking storage options: Rook, Ceph, and Longhorn (spoiler: Rook won)
• 2021-07-29: „APPUiO Cloud“ name chosen, appuio.cloud domain registered
• 2021-08: Project Syn components for APPUiO Cloud in beta
• 2021-08: cloudscale.ch cluster, including Kyverno and Keycloak, ready for internal use
• 2021-09-17: docs.appuio.cloud published
• 2021-09-20: Public announcement
• 2021-10-28: The testing phase started with beta users on the LPG-2 region migrating apps from APPUiO Public
• 2021-12-01: The testing phase ended
• 2021-12-07: Pricing calculator released
• 2021-12-16: New logo
• 2022-02-02: Exoscale Geneva region available
• 2022-02-07: APPUiO Cloud Portal released
• 2022-02-09: Getting started guide published
• 2022-05-03: First 6 months of APPUiO Cloud
• 2022-09-01: APPUiO Public fully decommissioned

Conclusion

Can other organizations use a similar process to create a product? We believe that yes, it is possible. However, there are a few caveats, that we know some companies should have to work on those items first, in order to have a successful DevOps journey.

First of all, writing skills are fundamental. We need DevOps engineers to be writers and to put everything down. Not only as „everything as code“ (security, infrastructure, business rules, etc) but also as documentation writers, making sure that both engineers and users are able to refer to a written document that explains the reasons why things happen. Yes, keeping that written documentation is part of the work; it is not a chore, it is not a bonus; it is part of the deliverables, and it must be updated, reviewed, and proofread.

Second of all, Cloud Native technologies have been designed to work faster than ever. Containers, Kubernetes, CI/CD pipelines, Open Source, and all of the ecosystem of Cloud Native technology is the greatest enabler of our world. The technological context constitutes a fantastic „giant’s shoulder“ where we can stand, and go faster and better. We definitely could have never done this work without the ecosystem of Open Source Cloud Native technologies available today.

And third of all, trust is paramount. You have to have trust in your teams. We actually think that trust is more important than flat hierarchies; even though these have helped us, without trust there’s no way we could have created APPUiO Cloud in such a short amount of time. Trust allows teams to work independently, moving fast, and without the inherent fear typical of a „blame culture“. And trust is the key ingredient for asynchronous work culture. You cannot really go full async if you do not trust your teams. We stress this point because this factor is the deal breaker for many teams in this country.

These are, we think, the three most important pillars of our DevOps culture: writing, technology, and trust. The ones that have helped us shape APPUiO Cloud into a product that is steadily growing and changing continuously.

Is it easy to work like this in DevOps mode? Of course not, there are lots of things that can go wrong. Is it worth it? Let’s put it this way: after all this time, we have internalized this way of working so much, that we couldn’t do things any other way. We think it is totally worth it, and as a result, we just do things like this.

With APPUiO Cloud, VSHN has demonstrated that we can deliver world-class products in a short amount of time, with a small team of experts, and with fast cycles of feedback and experimentation baked into the process.

Since our original announcement of APPUiO Cloud last September we’ve been very busy collecting feedback from our customers, and working on improving the product with many new features and capabilities. And we are very pleased to see our platform grow more and more every day!

This article contains a summary of everything that has happened in the first six months of APPUiO Cloud, with some hints about what’s coming next.

Security

We have placed a strong focus in securing APPUiO Cloud to the maximum, yet offering flexibility and manageability to our customers.

All security policies are managed through the Kyverno policy engine, a CNCF Sandbox project. On the other hand, the management of organizations and teams is based on Keycloak Groups, with additional syncing to OpenShift provided by an ad hoc, open source Kubernetes operator.

Management

APPUiO Cloud has been possible thanks to the incredible experience of the VSHN team and to their investment in tooling. In particular, Project Syn has made possible the automation of many different tasks in APPUiO Cloud. And of course it is all 100% open source, as the APPUiO Cloud Commodore Component.

The APPUiO Cloud billing system is also a fine piece of automation, helping us process the consumption of resources, and generating the invoices for our customers in a timely manner. Christian Häusler, APPUiO Cloud’s Product Owner, described it in a previous blog post.

API and Portal

We’ve created an extensive APPUiO Cloud Control API, fully open source, based on the OpenShift and Kubernetes APIs, and running on top of vcluster. Another recent blog post also from our Product Owner Christian Häusler gives more information about this topic.

We have also recently launched the APPUiO Cloud Portal, fully open source, offering our users a self-service mechanism to manage their organizations, users, and to quickly access useful information about their APPUiO Cloud usage. The portal uses the same Kubernetes-based APPUiO Cloud Control API mentioned above. This Portal is under heavy development, so you can expect new features to be released regularly.

Cilium

Following our partnership with Isovalent APPUiO Cloud is currently using Cilium Enterprise as its CNI of choice, positioning it at the forefront of technology, and opening fantastic opportunities for the future.

New Zone

After the first APPUiO Cloud zone opened in December (running on cloudscale.ch in Kanton Aargau), we quickly opened the second in February, this time on Exoscale in the Canton de Genève.

Documentation

And none of this effort could go into production without a great deal of documentation. Not only is our user documentation completely open source, we have also been updating our APPUiO Cloud for System Engineers documentation regularly, with lots of juicy details for your technical teams to learn about how APPUiO Cloud works.

Interested?

Would you like to know more about APPUiO Cloud?

• Fill the sign-up form in the APPUiO Cloud website and get your account today!
• Learn more about the features of APPUiO Cloud and check the pricing page.
• Check the technical user documentation with all the information your teams need to use APPUiO Cloud to the maximum.
• Check the APPUiO Cloud status page.
• Join the Community Discussions on GitHub and be notified of the latest APPUiO Cloud news!

In a previous blog post we talked about how we handle billing for APPUiO Cloud. This time we’re going to talk about how we have extended the Kubernetes API to provide a custom API for our product.

Just like in the aforementioned blog post, the links in this text point to pages in the APPUiO Cloud for System Engineers documentation, with more technical details.

As a reminder, APPUiO Cloud consists of several independent OpenShift 4 clusters called zones. They can be individually operated upon through the Kubernetes API and Console.

Being a shared platform, a precondition for users to use APPUiO Cloud is to belong to an organization, that is, the entity we can send invoices to. This organization must be the same for all zones; which means that users get a single invoice, no matter which zone their applications are running in.

Managing organizations individually on each zone would be tricky, and would quickly lead to issues of bi-directional synchronization; this is commonly known as a hard problem in IT, and of course we would rather like to avoid it. And this problem would get worse in the future, every time we open a new APPUiO Cloud zone; so, thanks but no thanks.

We needed a tool to manage organizations, independent from (yet intimately related to) any and each APPUiO Cloud zones. And since we are strong believers of both „API First“ and „Kubernetes First“, we decided to extend Kubernetes‘ own API to do this.

We created Custom Resource Definitions (CRDs) defining organizations and teams, placing them at the core of our APPUiO Cloud API. This gave us access control features for free, and as a huge bonus, we got a free client to talk to said API: kubectl itself (as well as its OpenShift counterpart, oc!)

This single design decision made the APPUiO Cloud API compatible with the larger Kubernetes ecosystem. We used this API to build a whole web application on top of it, and this turned out to be quite easy, largely thanks to said ecosystem.

For example, based on Kubernetes‘ own objects, rules, and groups, we can make the user interface of our web application reactive to the permissions of the current user; this is entirely handled by the API thanks to RBAC. Which means, no need to touch the UI code when permissions change!

The important thing to keep in mind is that the CRDs above are only used to represent data stored in a separate system; we developed adapters (in the shape of native Kubernetes controllers and operators) in charge of synchronizing the organizations and teams data between the API and the systems where those objects are stored.

Specifically, organizations and teams are groups and subgroups stored in a Keycloak instance; but this could be replaced by any other identity provider (IdP) in the future. This means that the identity data model of APPUiO Cloud is completely independent of the underlying store used at any time.

As it is generaly known from Kubernetes, the spec fields in those Custom Resources represent the desired state to be synced to the connected system; status fields represent the actual state on the connected system.

Thanks to the pre-existence of CRDs, the development of the UI and the adapters thereof could progress independently from one another, working as a common de facto contract between different parts of APPUiO Cloud. Even better, during development the behavior of the adapters could be stubbed, by directly manipulating the custom resources.

And to thank you for reading until the end, here’s a bonus detail: we’re using vcluster in our quality management process. What is vcluster? It’s a way to create virtual Kubernetes clusters in another cluster. It allows us to run multiple instance of the API on one Kubernetes or OpenShift cluster, without risking to run into CRD version conflicts or other issues. We use this approach to quickly spin integration and testing environments through our test automation procedures (driven by GitHub Actions), including feature branch testing. This single tool has boosted our productivity tremendously.

APPUiO Cloud is the simplest, most secure, and fastest way for DevOps teams to deploy apps on an OpenShift cluster. Developers all over Europe are enjoying the benefits of quick access to a secure platform for their applications. APPUiO Cloud provides almost instant access to clusters in various zones, with a convenient pay-per-use model which makes it particularly interesting for startups, education, and many other kinds of organizations.

Speaking about APPUiO Cloud’s pay-per-use model, we would like to explain in this blog post how our customers are billed for their usage of APPUiO Cloud resources. In a nutshell, APPUiO Cloud is supported in the background by a series of mechanisms that transform usage data metrics into actual invoices.

This multi-step process can be summarized in three steps: data collection, processing, and invoice generation.

By the way, following our commitment to openness and transparency, the links in the text point to various pages in our APPUiO Cloud for System Engineers documentation, geared for a technical audience. Also don’t hesitate to ask questions in our Community Discussions or the APPUiO Community Chat; we will be delighted to tell you more.

Data Collection

The first step of the invoicing process is based in continuous monitoring of APPUiO Cloud, using Prometheus. The metrics of interest, which are used as the basis of all invoicing calculations, are:

• The amount of effective and requested memory used;
• The amount of allocated persistent storage;
• And the amount of allocated services of type LoadBalancer.

These metrics are sampled regularly from all APPUiO Cloud zones.

Our cluster monitoring systems have a limited retention time, averaging from days to weeks. This means that we need to use long term storage should we need to recalculate invoices for a whole fiscal year, and we use Thanos for that.

Processing

Before we transfer this information into our billing system, a secondary process matches and augments metrics with various metadata. First, we must map the collected metrics to our customers in the billing system. We also need to correlate that data to our products, including their prices, and any eventual applicable discount or promotion.

This secondary process consists of an ETL mechanism extracting metrics through the Thanos API, transforming them as required, and storing results in an intermediate data warehouse system, based on a standard star schema. This ETL process is complemented by an enrichment step, which fills missing data that needs to be collected from other sources as required.

Invoice Generation

The generation of invoices is handled by yet another automated mechanism. This process, running at the end of each billing cycle (in our case, a month) uses the aforementioned data warehouse as an API and data source, and it actually generates the final invoices in our ERP billing system.

This loosely coupled architecture gives us the flexibility to replace the invoice system (our ERP) with minimal effort. It also allows us to pull in data from any sources we could think of in the future.

Following our love for DevOps, all of the processes described above are fully automated, tested, and running in a fault-tolerant way.

But there’s more: the ERP adapter for APPUiO Cloud and the Reporting for APPUiO Cloud projects in GitHub are open source! Check them out and see how this all fits together nicely.

Are you interested in APPUiO Cloud? Get your account and start deploying applications on an OpenShift cluster today.