Espejote (big mirror in Spanish) manages arbitrary resources in a Kubernetes cluster. Built from the ground up to take advantage of Server-Side Apply and Jsonnet templating.

VSHN manages a large fleet of Kubernetes clusters for our customers, and we try to automate as much as possible to keep our operations efficient and sustainable. We use GitOps principles, but sometimes external state needs to be merged into the desired state defined in Git. This GitOps journey took us from Ansible playbooks directly applying YAML, to various operators, to bash “reconcilers”, and finally to Espejote, our shiny new GitOps operator.

Chapter 1: The Ansible era and first operator attempts

In the beginning, we used Ansible playbooks and custom roles to manage our OpenShift 3 Kubernetes clusters. We had a set of YAML files that defined the desired state of our clusters, and we would run Ansible playbooks to apply those YAML files to the clusters. This worked, but it was not very efficient. We had to run the playbooks manually, and if we forgot to run them, the clusters would drift from the desired state.

The collection of roles was nicknamed “mungg”, the Swiss German word for “marmot”. Nobody seems to know why, but it stuck.

We were just getting into writing operators and developed espejo to quickly sync resources between namespaces. It was the very early days of our operator journey.

Chapter 2: The sea of operators and tears

To solve the problem of manual intervention (and because we migrated to OpenShift 4, where the install procedure doesn’t use Ansible anymore), we started looking into Kubernetes operators.
It can’t be that hard to patch a Kubernetes manifest. Right? Wrong.
Some of the operators were buggy, some of them were not flexible enough, some of them loved to randomly go into reconcile loops, and most of them used too many resources. Some of them crashed our API servers. We started with resource-locker-operator, migrated to patch-operator, generated outages with Kyverno, and tested all other policy engines we could find. Kubewarden was the only one we really liked, but the cluster context API was not yet flexible enough for our use cases.

Espejo had been a good start, but we did not yet have the experience to build well-designed operators.
It showed. Every event triggered a full reconciliation of every resource, so syncing slowed down dramatically on larger clusters. We missed a lot of flexibility.

Chapter 3: Getting desperate for safe landings

We were fed up with the constant bugs and breaking changes in Kyverno, and patch-operator was barely maintained. Espejo was at its limits.

Desperate times called for desperate measures, so we started using an amalgamation of bash “reconcilers” – hacks with cron jobs, tiny custom controllers, and pre-processing resources in Project Syn.

We were using Jsonnet more and more. Project Syn components primarily use Jsonnet. We use Jsonnet for our cloudscale machine-api provider, for our SSO solution, and many other projects.

A growing issue were our heavily patched OpenShift alerting rules. We curate upstream rules and only enable the ones we need. Some are heavily patched. Every OpenShift release the upstream definitions are moved around and are sometimes only available embedded into Go code. We needed something that was able to patch rules already deployed in the cluster, as this was the only stable interface we had.

Chapter 4: Espejote, the shiny new GitOps operator

Bolstered by our growing operator experience and our love for Jsonnet, we decided to build our own operator to rule them all. We wanted something that was flexible, efficient, and easy to use. We wanted something that could handle all our use cases, from syncing resources between namespaces to patching OpenShift alerting rules.

Espejote is the result of that journey. It merges cluster state with GitOps principles, using Jsonnet to define the desired state of our clusters. It efficiently caches cluster state, and the reconcile trigger logic is explicitly defined. Sane controller-runtime rate limits apply. Jsonnet allows a huge amount of flexibility, and native server-side apply makes adding and removing keys a breeze. Every Espejote “resource manager” – the dynamic controller spawned for a config unit – uses its own ServiceAccount for least privilege.

Espejote is the operator we always wanted, and we are excited to share it with the world.

What is Espejote?

Espejote is a Kubernetes operator allowing you to manage arbitrary resources in a Kubernetes cluster.
It can mix GitOps principles with in-cluster state.

Why Espejote?

There are plenty of similar tools (and policy engines), but Espejote sets itself apart by focusing on three core pillars:

1. Powered by Jsonnet

Espejote uses Jsonnet as its templating engine. Unlike YAML combined with Go templates, Jsonnet treats the configuration as a data structure. It understands objects, arrays, and strings. It can’t accidentally generate broken YAML because Jsonnet ensures the internal data structure is valid before it ever exports the final file.

2. Native Server-Side Apply

Espejote is built from the ground up to leverage server-side apply (SSA). This means Espejote plays nicely with other controllers and operators. It can manage a single annotation or an entire resource; SSA ensures that the changes are merged without stomping on other tools.

3. Reliability

Reliability isn’t an afterthought. Espejote was born out of the frustration of watching operators enter infinite reconcile loops or crash clusters. It features:

• Sane rate limiting and backoff strategies.
• Every configuration unit or “resource manager” runs its own dynamically spawned controller, so a misbehaving unit won’t affect others.
• Least privilege: Every resource manager runs with its own ServiceAccount.
• Explicit control: There are no implicit watches or “magic” triggers. You have complete control over what gets reconciled and when.

Real-World Use Cases

What can you actually do with Espejote? Here are a few ways VSHN is using it in production:

• Secret Syncing: Automatically replicate specific secrets (like image pull secrets or certificates) across multiple namespaces.
• Autoscaler Patching: Patching the OpenShift Cluster Autoscaler using Admission Webhooks.
• Alerting Rule Management: Curate and patch OpenShift alerting rules across different cluster versions.

The Future: WASM and Beyond

The roadmap includes a kro-like API builder for easy custom resource creation and support for WebAssembly plugins, which will allow developers to write custom logic in almost any language and run it safely within the Espejote controller.

Getting Started

• Website: espejote.io
• GitHub: vshn/espejote
• Documentation: Check out the annotated examples to see how Espejote can be used in real-world scenarios.
• Real-World Use Cases: Network policy management, a Gateway listener manager, HTTPRoute certificate management, and many more.

Example

This example ManagedResource patches the RedHat OperatorHub config singleton to disable all default sources. It shows the simplest usecase of unconditionally patching a static manifest.
More complex use cases can be found in the above getting started section.

apiVersion: espejote.io/v1alpha1
kind: ManagedResource
metadata:
  annotations:
  name: disable-default-sources
  namespace: openshift-marketplace
spec:
  serviceAccountRef:
    name: disable-default-sources
  triggers:
    - name: operatorhub
      watchResource:
        apiVersion: config.openshift.io/v1
        kind: OperatorHub
        name: cluster
  template: |-
    {
        "apiVersion": "config.openshift.io/v1",
        "kind": "OperatorHub",
        "metadata": {
            "name": "cluster"
        },
        "spec": {
            "disableAllDefaultSources": true
        }
    }

Large Language Models (LLMs) are rapidly becoming part of modern software systems. From chatbots and copilots to retrieval systems and AI agents, organizations are integrating generative AI into real production environments. But while building AI prototypes has become easier than ever, operating LLMs reliably in production remains a serious challenge.

At Kubernetes Community Days New York, Aarno Aukia shared practical insights into what it takes to run LLMs using proven DevOps practices. His talk highlighted an important reality: AI systems still need strong DevOps foundations – perhaps even more than traditional software systems.

Aarno Aukia’s talk at KCD New York

DevOps Meets AI

DevOps has always been about bridging the gap between development and operations. Developers focus on building application logic and managing data, while operations teams ensure that software runs reliably in production. Over the past decade, DevOps practices have matured around automation, observability, and continuous delivery.

In many organizations today, software follows a well-established pipeline: developers commit code to Git, automated CI/CD pipelines build and package the application, and Kubernetes deploys and runs it in production. Monitoring and logging systems then provide visibility into how the application behaves, allowing developers to continuously improve it.

This feedback loop has become the backbone of modern cloud-native development.

When AI enters the picture, however, the model changes in several important ways.

AI Systems Behave Differently

One of the biggest differences between traditional applications and AI-driven systems is determinism. Traditional software behaves predictably: given the same input, it produces the same output every time. LLMs behave very differently.

Large language models are probabilistic systems. They generate responses by predicting the next token based on context, effectively making statistical decisions about what comes next. This means that even small changes in prompts or user input can produce very different results.

A seemingly harmless modification to a system prompt can completely change the behavior of a model. In one example, simply adding a seasonal theme to a chatbot prompt caused the model to fail at answering basic questions.

For operations teams, this creates a new category of complexity. Instead of debugging deterministic systems, they now have to manage systems whose outputs can change subtly depending on context.

Testing therefore becomes significantly more complicated.

The Challenge of Testing AI

Traditional software testing is relatively straightforward. A test provides an input and verifies that the output exactly matches an expected value.

AI systems do not fit into this model. When an LLM generates an answer, the output might be correct even if the exact wording differs from what was expected. At the same time, the response could contain subtle factual errors or hallucinations.

Determining whether an answer is acceptable often requires semantic evaluation rather than strict comparisons. In some cases, organizations even use another LLM to evaluate the output of the first one. This introduces an entirely new testing paradigm that many teams are still learning how to manage.

More Artifacts to Manage

AI systems also introduce additional artifacts that must be tracked and versioned.

In traditional DevOps pipelines, the primary artifacts are source code and container images. With AI workloads, teams must also manage datasets, training artifacts, prompts, and model files. These models are often very large, sometimes tens of gigabytes in size, and must be stored and versioned carefully.

Without proper versioning, it becomes extremely difficult to debug issues or reproduce results later. If a model behaves unexpectedly, teams need to know exactly which model version, dataset, and configuration were used during deployment.

This dramatically increases the operational complexity of AI systems.

Observability Becomes Critical

Because LLMs are non-deterministic, observability becomes even more important than in traditional systems.

Logging must capture far more context than before. Instead of logging only application events, teams may need to record the full prompt, the model response, the model version, and the surrounding configuration. This allows operators to reconstruct what happened when something goes wrong.

Without detailed observability, debugging AI systems can quickly become impossible.

Open Models vs Hosted APIs

Another important operational consideration is the choice between closed and open models.

Hosted AI APIs offer convenience and powerful capabilities, but they also come with trade-offs. In many cases, organizations cannot control exactly when model updates happen or which minor version is running at a given time. This can make debugging and reproducibility difficult.

Open-weight and open-source models provide more operational control. They can be downloaded, versioned, tested locally, and deployed on internal infrastructure. This allows organizations to decide when and how updates are rolled out.

For many regulated industries such as finance, healthcare, or government, this level of control is essential.

Kubernetes as the Foundation

This is where Kubernetes becomes an important part of the AI infrastructure stack.

Kubernetes already solves many of the operational challenges associated with running distributed systems. It provides mechanisms for container orchestration, resource management, autoscaling, and fault tolerance. Importantly for AI workloads, it can also manage GPU resources.

However, operating Kubernetes itself is not trivial – as discussed in our article Best Kubernetes Distributions in 2026 – And Why You Might Not Want to Run Them Yourself, running clusters in production requires significant operational expertise.

Kubeflow and the Machine Learning Lifecycle

Kubeflow extends Kubernetes with specialized components for machine learning workflows. It helps manage the entire lifecycle of AI models, from training to production inference.

Kubeflow Pipelines allow teams to automate workflows for model development and training. These pipelines can orchestrate complex processes such as data preprocessing, training runs, evaluation steps, and packaging models for deployment.

For many organizations using LLMs, however, the main focus is not training models but serving them reliably in production.

This is where KServe plays a key role.

Serving LLMs with KServe

KServe is a Kubernetes-native model serving framework that simplifies deploying and operating AI models. It allows teams to run inference services on top of Kubernetes using standard APIs.

A typical deployment consists of a container running a model server, often based on runtimes such as vLLM. The container loads the model, uses GPU resources for inference, and exposes an API endpoint for applications.

KServe integrates with Kubernetes autoscaling mechanisms and observability tools, making it possible to scale AI workloads dynamically and monitor their behavior in production.

Because everything runs as Kubernetes resources, teams can apply the same DevOps practices they already use for other applications.

A Rapidly Evolving Ecosystem

The ecosystem around AI infrastructure is evolving extremely quickly. New projects are emerging to address the unique challenges of running LLMs at scale.

One example is LLMD, a Kubernetes operator specifically designed for LLM inference. It builds on existing technologies such as vLLM but adds specialized capabilities like request routing, model selection, caching, and intelligent scaling.

These kinds of tools illustrate how the cloud-native ecosystem is adapting to the operational needs of AI workloads.

AI Still Needs DevOps

Despite the hype surrounding generative AI, one lesson is clear: AI systems still require strong operational foundations.

Running LLMs in production involves far more than simply calling an API. It requires careful management of models, infrastructure, observability, and deployment processes.

Kubernetes and Kubeflow provide a powerful platform for addressing these challenges. By applying proven DevOps principles to AI systems, organizations can build platforms that are not only intelligent but also reliable and scalable.

As AI becomes a standard component of modern applications, the ability to operate these systems effectively will become just as important as the models themselves.

This is also where platform approaches come into play. Instead of every team building and operating complex stacks themselves, platforms can provide ready-to-use services on top of Kubernetes. One example is Servala – Sovereign App Store, a Kubernetes-native marketplace that connects organizations with a catalog of managed cloud-native services such as databases, storage, developer tools, and AI-ready infrastructure components.

At Swiss Cloud Native Day 2025 in Bern, our colleague Liene Luksika shared an honest and entertaining story about VSHN’s journey with Crossplane. What started as a simple use case evolved into a complex architecture, full of learnings, mishaps, and valuable lessons for anyone building managed services on Kubernetes.

From healthcare to cloud native

Liene comes from the healthcare sector, so when she joined the cloud native world at VSHN, she had to quickly get used to Kubernetes lingo – namespaces, instances, and of course, the obsession with laptop stickers. Luckily, VSHN has been around for more than 10 years, providing 24/7 managed services and building cloud native platforms for customers in Switzerland, Germany, and beyond.

Why Crossplane?

As customers increasingly asked VSHN to run their software as a service – databases, Nextcloud, and other critical apps – we needed a solid way to provision and manage infrastructure across private and public clouds. Crossplane seemed like the perfect fit:

• It lets engineers define desired state vs. observed state
• It automatically reconciles the two – like making coffee appear if that is your desired state
• It provides flexible building blocks to expose clean APIs for managed services on Kubernetes

VSHN has used Crossplane in production since early 2021 (around v0.14) and runs the Crossplane Competence Center in Switzerland.

The evolution: from simple to complex

Our first use case was straightforward: a customer wanted two types of databases (Redis and MariaDB), T-shirt sized, no extras. Crossplane handled this beautifully.

Then reality hit. Customers wanted backups and restores, logs and metrics, alerting, maintenance and upgrades, scaling and user management, special features like Collabora for Nextcloud, and the freedom to choose infrastructure. To serve this, we adopted a split architecture:

• A control cluster for all Crossplane logic
• Separate service clusters for customer workloads

This runs today with customers like health organizations in Gesundheitsamt Frankfurt and HIN in Switzerland, on providers such as Exoscale and Cloudscale, keeping data sovereign and operations reliable.

When things go wrong

Building complex platforms means learning in production:

• Deletion protection surprise: a minor Crossplane change removed labels before deletion, wiping our safeguard. Backups saved the day
• Race conditions: a split approach to connection details occasionally made apps unreachable until we cleaned up code
• The big one: during a planned “no-downtime” maintenance for a fleet with 1’300+ databases, objects hit an invalid state and Kubernetes garbage collection deleted 230 database objects. Some restores were fresh, some older. We pulled in 20 people overnight, communicated openly, and recovered together with the customer

Key lessons: test at realistic scale and keep recent, tested backups. Also, practice the restore path, not just the backup.

Crossplane 2.0 – where next?

Crossplane 2.0 introduces major breaking changes. Staying put is not an option, but migrating means real effort, especially for our split control plane architecture. We are evaluating whether Crossplane 2.0 fits our needs or if alternatives are a better match. As always, we will document our decisions openly in VSHN’s Architecture Decision Records.

Final thoughts

Cloud native success is not just about tools. It is about learning fast, designing for failure, and communicating clearly with customers. Crossplane has enabled a lot of innovation for us, and it has also tested us. Whether we proceed with Crossplane 2.0 or chart a different course, we will keep building sovereign, reliable, open managed services for our customers.

👉 Watch the whole video on YouTube:

We’re absolutely thrilled to release the sixth edition of our “DevOps in Switzerland” report – and this time with a special focus on Platform Engineering and Artificial Intelligence (AI)! 🤖

From January to April 2025, we conducted a study with professionals from the Swiss tech community. The result: valuable insights into how DevOps teams in Switzerland work today – what tools they use, how their teams are structured, the challenges they face, and where AI is already being used in practice.

💡 Want a sneak peek?

• 💡Swiss companies are no longer asking whether to adopt DevOps – they’re asking how to scale it.
• 📈 Platform Engineering and AI are reshaping how teams ship software faster, safer, and smarter.
• 💡1 in 3 Swiss DevOps teams already use AI in production – for code reviews, CI/CD optimization, and architecture support. Another third are gearing up to follow.
• 💡54% of Swiss companies now have dedicated Platform Engineering teams.
• Internal Developer Platforms (IDPs) are becoming the secret weapon for enabling autonomy and reducing complexity.
• 💡 Devs say yes to AI! 79% of Swiss developers are comfortable using AI in their workflows – but only 20% believe it’s fully ready.
• The report shows: AI is promising, but needs better measurement and trust to scale.

You’ll find all of this (and much more!) in our compact PDF report (available in English only). Just like last year, the report begins with an executive summary – perfect for those short on time.

📥 Download now

You can download the DevOps Report 2025 here. Have fun reading, and let us know what you think!

Enjoy reading – we’re excited to hear your feedback! 🙌

We are thrilled to announce the fifth edition of our „DevOps in Switzerland” report!

From January to April 2024 we conducted a study to learn how Swiss companies implement and apply DevOps principles.

We compiled the results into a PDF file, and just like in the previous edition, we provided a short summary of our findings in the first pages.

You can get the report here. Enjoy reading and we look forward to your feedback!