APPUiO Cloud users are now able to monitor their applications on their own!

APPUiO Cloud has been upgraded to OpenShift 4.11 and thanks to User Workload Monitoring, users are now able to monitor their applications on their own and get notified when something needs attention.

APPUiO Cloud now runs on OpenShift version 4.11. Both zones – cloudscale.ch LPG 2 and Exoscale CH-GVA-2 0 – have been updated in the past two weeks.

Users might notice some warnings when deploying things to APPUiO Cloud or when updating existing workloads. Those warnings come from Pod Security Admission. The Kubernetes community is changing how pod security is handled, and Pod Security Admission got introduced with Kubernetes 1.24, which is part of OpenShift 4.11. You will find more on the subject at our Pod Security Admissions documentation page.

As far as we understand, future versions of Kubernetes will change again and those warnings will likely become errors. Therefore we suggest to look into them sooner rather than later.

…and now that the administrative stuff has been taken care of, let us continue with the exciting stuff:

Thanks to functionality introduced in OpenShift 4.11, we are now able to offer you User Workload Monitoring. This feature allows APPUiO Cloud users to monitor their applications on their own.

Users can now collect metrics of applications and write alerts based on the collected metrics. A how-to on how to do so can be found at Monitor Application With Prometheus. Additionaly, users can now access metrics relevant to their application, which are collected by the cluster monitoring, and thus so far were not accessible to end users. Monitor PVC Disk Usage explains how this can be used to monitor disk usage of persistent volumes. It is also possible to access metrics on memory, CPU usage and many things more.

Users who are interested in nice graphs can install an instance of Grafana for themselves and build informative and nice looking dashboards depicting the metrics of their applications. See the Custom Grafana documentation for details how this can be achieved.

Others do not want to constantly look at dashboards to see whether something is broken, chosing to get notified when this is the case instead. This is also possible, as it’s now an option to not only write your own alerts but also individual alert routes and get email or chat messages when something is in need of attention. Configure Alert Receivers teaches how to do this.

User Workload Monitoring is a long-awaited feature, and we hope that it will make an impact for many APPUiO Cloud users.

What’s next?

We are currently working towards user self registration for APPUiO Cloud. This allows new users to register for APPUiO Cloud all by themselves without having to talk to one of our sales representatives. A milestone within that initiative will be User invitation.

We built APPUiO Cloud to support performance-sensitive production workloads. But that performance also came along with a price tag, one that users had to pay even for non-production workloads — like your development environments — or non-time critical workloads — like your background cron jobs.

We have heard this concern from our customers, and we are pleased to introduce today the concept of Node Classes. They allow you to schedule workloads on nodes with different specifications and costs. See the APPUiO Cloud documentation for more details about node classes.

On the cloudscale.ch – LPG 2 zone, you can now choose between “Flex” and “Plus” classes. “Plus” nodes have the same performance-optimized characteristics all nodes on that cluster had so far. “Flex” nodes will be a less capable but more affordable option. A more detailed explanation of these two classes is available on the documentation website, where you can also find the price table.

Workloads in our existing namespaces will continue to run on the more performant “Plus” node class. The default for all new namespaces will be “Flex.” Check the documentation to learn how to schedule your workloads on the desired node class.

Would your workload profit from nodes optimized for it? Please let us know through our roadmap. Some nodes with GPU support, maybe?

What’s next?

Soon we will release User Workload Monitoring on APPUiO Cloud. This allows users to monitor their applications, write alerts, and receive notifications when those alerts fire. For instance, when your persistent volume runs out of space.

Setting appropriate limit and request values in your deployments is not an easy task. To help our customers, we have enabled the use of the VerticalPodAutoscaler object on all APPUiO Cloud zones. This blog post will show you how to use it.

To use the VerticalPodAutoscaler object, you need an APPUiO Cloud project with some payload deployed and running. The Vertical Pod Autoscaler project on GitHub contains some sample YAML files that deploy a very simple demo application to your project.

Here is the YAML required to create a VerticalPodAutoscaler that will analyze the resource consumption of the vpa-example deployment:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vpa-recommender
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       vpa-example
  updatePolicy:
    updateMode: "Off"

The VPA requires a few moments to gather data and provide recommendations from it. After some time, during which your deployment should have been running in order to gather meaningful data, run the following command and you should see an output similar to this in your terminal:

$ oc get vpa vpa-recommender --output yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  annotations: …
# …
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vpa-example
  updatePolicy:
    updateMode: Auto
status:
  conditions:
  - status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: fortune-container
      lowerBound:
        cpu: 25m
        memory: 262144k
      target:
        cpu: 203m
        memory: 262144k
      uncappedTarget:
        cpu: 203m
        memory: 262144k
      upperBound:
        cpu: 71383m
        memory: "6813174422"

You should analyze with care the values provided by the autoscaler for your deployment. Don’t blindly apply its recommendations; let your application run for a while and study the numbers closely.

Some tips for your analysis, all inside the status.recommendation part of the response:

• The .containerRecommendations[*].target value could be considered indicative for request values.
• The .containerRecommendations[*].upperBound value could be used as an indication to set limit values.

For more hints, the OpenShift dashboard shown on this page of the APPUiO Cloud documentation shows utilization numbers for both CPU and memory limits. Those values function as a suitable supplementary information source and must be taken into consideration.

The APPUiO Cloud documentation has more information about the VerticalPodAutoscaler object.

In a previous blog post we talked about how we handle billing for APPUiO Cloud. This time we’re going to talk about how we have extended the Kubernetes API to provide a custom API for our product.

Just like in the aforementioned blog post, the links in this text point to pages in the APPUiO Cloud for System Engineers documentation, with more technical details.

As a reminder, APPUiO Cloud consists of several independent OpenShift 4 clusters called zones. They can be individually operated upon through the Kubernetes API and Console.

Being a shared platform, a precondition for users to use APPUiO Cloud is to belong to an organization, that is, the entity we can send invoices to. This organization must be the same for all zones; which means that users get a single invoice, no matter which zone their applications are running in.

Managing organizations individually on each zone would be tricky, and would quickly lead to issues of bi-directional synchronization; this is commonly known as a hard problem in IT, and of course we would rather like to avoid it. And this problem would get worse in the future, every time we open a new APPUiO Cloud zone; so, thanks but no thanks.

We needed a tool to manage organizations, independent from (yet intimately related to) any and each APPUiO Cloud zones. And since we are strong believers of both “API First” and “Kubernetes First”, we decided to extend Kubernetes’ own API to do this.

We created Custom Resource Definitions (CRDs) defining organizations and teams, placing them at the core of our APPUiO Cloud API. This gave us access control features for free, and as a huge bonus, we got a free client to talk to said API: kubectl itself (as well as its OpenShift counterpart, oc!)

This single design decision made the APPUiO Cloud API compatible with the larger Kubernetes ecosystem. We used this API to build a whole web application on top of it, and this turned out to be quite easy, largely thanks to said ecosystem.

For example, based on Kubernetes’ own objects, rules, and groups, we can make the user interface of our web application reactive to the permissions of the current user; this is entirely handled by the API thanks to RBAC. Which means, no need to touch the UI code when permissions change!

The important thing to keep in mind is that the CRDs above are only used to represent data stored in a separate system; we developed adapters (in the shape of native Kubernetes controllers and operators) in charge of synchronizing the organizations and teams data between the API and the systems where those objects are stored.

Specifically, organizations and teams are groups and subgroups stored in a Keycloak instance; but this could be replaced by any other identity provider (IdP) in the future. This means that the identity data model of APPUiO Cloud is completely independent of the underlying store used at any time.

As it is generaly known from Kubernetes, the spec fields in those Custom Resources represent the desired state to be synced to the connected system; status fields represent the actual state on the connected system.

Thanks to the pre-existence of CRDs, the development of the UI and the adapters thereof could progress independently from one another, working as a common de facto contract between different parts of APPUiO Cloud. Even better, during development the behavior of the adapters could be stubbed, by directly manipulating the custom resources.

And to thank you for reading until the end, here’s a bonus detail: we’re using vcluster in our quality management process. What is vcluster? It’s a way to create virtual Kubernetes clusters in another cluster. It allows us to run multiple instance of the API on one Kubernetes or OpenShift cluster, without risking to run into CRD version conflicts or other issues. We use this approach to quickly spin integration and testing environments through our test automation procedures (driven by GitHub Actions), including feature branch testing. This single tool has boosted our productivity tremendously.

APPUiO Cloud is the simplest, most secure, and fastest way for DevOps teams to deploy apps on an OpenShift cluster. Developers all over Europe are enjoying the benefits of quick access to a secure platform for their applications. APPUiO Cloud provides almost instant access to clusters in various zones, with a convenient pay-per-use model which makes it particularly interesting for startups, education, and many other kinds of organizations.

Speaking about APPUiO Cloud’s pay-per-use model, we would like to explain in this blog post how our customers are billed for their usage of APPUiO Cloud resources. In a nutshell, APPUiO Cloud is supported in the background by a series of mechanisms that transform usage data metrics into actual invoices.

This multi-step process can be summarized in three steps: data collection, processing, and invoice generation.

By the way, following our commitment to openness and transparency, the links in the text point to various pages in our APPUiO Cloud for System Engineers documentation, geared for a technical audience. Also don’t hesitate to ask questions in our Community Discussions or the APPUiO Community Chat; we will be delighted to tell you more.

Data Collection

The first step of the invoicing process is based in continuous monitoring of APPUiO Cloud, using Prometheus. The metrics of interest, which are used as the basis of all invoicing calculations, are:

• The amount of effective and requested memory used;
• The amount of allocated persistent storage;
• And the amount of allocated services of type LoadBalancer.

These metrics are sampled regularly from all APPUiO Cloud zones.

Our cluster monitoring systems have a limited retention time, averaging from days to weeks. This means that we need to use long term storage should we need to recalculate invoices for a whole fiscal year, and we use Thanos for that.

Processing

Before we transfer this information into our billing system, a secondary process matches and augments metrics with various metadata. First, we must map the collected metrics to our customers in the billing system. We also need to correlate that data to our products, including their prices, and any eventual applicable discount or promotion.

This secondary process consists of an ETL mechanism extracting metrics through the Thanos API, transforming them as required, and storing results in an intermediate data warehouse system, based on a standard star schema. This ETL process is complemented by an enrichment step, which fills missing data that needs to be collected from other sources as required.

Invoice Generation

The generation of invoices is handled by yet another automated mechanism. This process, running at the end of each billing cycle (in our case, a month) uses the aforementioned data warehouse as an API and data source, and it actually generates the final invoices in our ERP billing system.

This loosely coupled architecture gives us the flexibility to replace the invoice system (our ERP) with minimal effort. It also allows us to pull in data from any sources we could think of in the future.

Following our love for DevOps, all of the processes described above are fully automated, tested, and running in a fault-tolerant way.

But there’s more: the ERP adapter for APPUiO Cloud and the Reporting for APPUiO Cloud projects in GitHub are open source! Check them out and see how this all fits together nicely.

Are you interested in APPUiO Cloud? Get your account and start deploying applications on an OpenShift cluster today.