How a Slack “Zero Width Space” character broke a Kubernetes Deployment – Lessons from an Instana SRE

How a Slack “Zero Width Space” character broke a Kubernetes Deployment – Lessons from an Instana SRE

What is a Zero Width Space?

A few days ago I learned that the Unicode character for ‘ZERO WIDTH SPACE’ is U+200B.

“The zero-width space is a non-printing character used in computerized typesetting to indicate word boundaries to text processing systems.” Source: https://en.wikipedia.org/wiki/Zero-width_space

How Slack Broke My Kubernetes Deployment

It all started with a built-in alert (from Instana APM) that I got paged on. At first sight it looked like the container of our UI component was not able to start. The issue contains the starting time, the GKE node pool, GKE node name and the K8s pod that was not able to start.

So I started checking the pod status and it came up as “Pending”.

The “Conditions” tab for the pod shows a new scheduled deployment.

Checking the “Events” tab displays the full error message. For each event Instana shows the event type, reason, message, namespace and time so I can easily identify the event that I need to investigate next. The latest event contained a detailed error message that helped me to identify the root cause of the issue.

Here is the error message in a more readable format. As you can see the “zero width space” Unicode character somehow made it into the container image tag. Redeploying the UI component with the correct image version fixed the issue.

Failed to apply default image tag
"/ui-client:2.176.12\u200b7-0": couldn't parse image reference 
"/ui-client:2.176.12\u200b7-0": invalid reference format

So how did it get there? It turns out we received a hotfix deployment request for this UI component in our internal chat system (Slack). We copied the container image version from the message and used the version for the deployment. These “zero width space” characters are added for formatting reasons in Slack. Since this character has zero width, we did not see the hidden character when deploying the hotfix.

Searching the Internet it seems we are not the only ones experiencing this issue.

Lessons Learned – OR – How To Prevent Zero Width Space From Causing Problems

To avoid similar issues in the future we added regex checks to our deployment automation, so only valid characters are allowed when deploying components.

We use dedicated Jenkins servers for production deployments. Only SRE team member have access and can deploy releases and hotfixes to production. Jenkins gives us a nice audit trail, who and when components got deployed.

The Validating String Parameter Jenkins plugin allows us to specify valid parameters using regular expressions. This resolves the issue of whitespace characters ending up in the VERSION parameter, see https://plugins.jenkins.io/validating-string-parameter/

Small enhancement for Jenkins Job DSL plugin

We are using the Job DSL plugin for Jenkins to automatically generate all Jenkins jobs. The latest version does not support the validating string parameter yet. Therefore we created a pull request to add support for the validating string parameter plugin (https://github.com/jenkinsci/job-dsl-plugin/pull/1226).

Here is an example of what a job description looks like using validatingStringParam().

job('deploy-component') {
  parameters {
    validatingStringParam(VERSION, '', 'instana-release-\d+-\d+|1.\d+.\d+', '', '')
  }
  steps {
    shell('echo ${VERSION}')
  }
}

Play with Instana’s APM Observability Sandbox

Developer, Thought Leadership
Kubernetes (also known as k8s) is an orchestration platform and abstract layer for containerized applications and services. As such, k8s manages and limits container available resources on the physical machine, as well...
|
Announcement, Developer, Product
This is the third blog in a series of blogs on Pipeline Feedback. In the first blog, Increasing Agility with Pipeline Feedback Scoping, we discussed the different types of pipeline feedback scope....
|
Announcement, Developer, Product
Co-Authored by: Evgeni Wachnowezki AWS is sharing its Amazon EKS Distro Kubernetes distribution with the community. Amazon EKS Distro is a Kubernetes distribution optimized for security and reliability, and is battle-tested by...
|

Start your FREE TRIAL today!

Instana, an IBM company, provides an Enterprise Observability Platform with automated application monitoring capabilities to businesses operating complex, modern, cloud-native applications no matter where they reside – on-premises or in public and private clouds, including mobile devices or IBM Z.

Control hybrid modern applications with Instana’s AI-powered discovery of deep contextual dependencies inside hybrid applications. Instana also gives visibility into development pipelines to help enable closed-loop DevOps automation.

This provides actionable feedback needed for clients as they to optimize application performance, enable innovation and mitigate risk, helping Dev+Ops add value and efficiency to software delivery pipelines while meeting their service and business level objectives.

For further information, please visit instana.com.