How a Slack “Zero Width Space” character broke a Kubernetes Deployment – Lessons from an Instana SRE

What is a Zero Width Space?

A few days ago I learned that the Unicode character for ‘ZERO WIDTH SPACE’ is U+200B.

“The zero-width space is a non-printing character used in computerized typesetting to indicate word boundaries to text processing systems.” Source: https://en.wikipedia.org/wiki/Zero-width_space

How Slack Broke My Kubernetes Deployment

It all started with a built-in alert (from Instana APM) that I got paged on. At first sight it looked like the container of our UI component was not able to start. The issue contains the starting time, the GKE node pool, GKE node name and the K8s pod that was not able to start.

So I started checking the pod status and it came up as “Pending”.

The “Conditions” tab for the pod shows a new scheduled deployment.

Checking the “Events” tab displays the full error message. For each event Instana shows the event type, reason, message, namespace and time so I can easily identify the event that I need to investigate next. The latest event contained a detailed error message that helped me to identify the root cause of the issue.

Here is the error message in a more readable format. As you can see the “zero width space” Unicode character somehow made it into the container image tag. Redeploying the UI component with the correct image version fixed the issue.

Failed to apply default image tag
"/ui-client:2.176.12\u200b7-0": couldn't parse image reference 
"/ui-client:2.176.12\u200b7-0": invalid reference format

So how did it get there? It turns out we received a hotfix deployment request for this UI component in our internal chat system (Slack). We copied the container image version from the message and used the version for the deployment. These “zero width space” characters are added for formatting reasons in Slack. Since this character has zero width, we did not see the hidden character when deploying the hotfix.

Searching the Internet it seems we are not the only ones experiencing this issue.

Lessons Learned – OR – How To Prevent Zero Width Space From Causing Problems

To avoid similar issues in the future we added regex checks to our deployment automation, so only valid characters are allowed when deploying components.

We use dedicated Jenkins servers for production deployments. Only SRE team member have access and can deploy releases and hotfixes to production. Jenkins gives us a nice audit trail, who and when components got deployed.

The Validating String Parameter Jenkins plugin allows us to specify valid parameters using regular expressions. This resolves the issue of whitespace characters ending up in the VERSION parameter, see https://plugins.jenkins.io/validating-string-parameter/

Small enhancement for Jenkins Job DSL plugin

We are using the Job DSL plugin for Jenkins to automatically generate all Jenkins jobs. The latest version does not support the validating string parameter yet. Therefore we created a pull request to add support for the validating string parameter plugin (https://github.com/jenkinsci/job-dsl-plugin/pull/1226).

Here is an example of what a job description looks like using validatingStringParam().

job('deploy-component') {
  parameters {
    validatingStringParam(VERSION, '', 'instana-release-\d+-\d+|1.\d+.\d+', '', '')
  }
  steps {
    shell('echo ${VERSION}')
  }
}

Play with Instana’s APM Observability Sandbox

Announcement, Product, Thought Leadership
Kubernetes, Kubernetes Monitoring and KubeCon I have vivid memories of the first KubeCon that I attended – it was in Austin, and it SNOWED. I was also pretty blown away by the...
|
Engineering, Featured, Product
At Instana, we store a lot of customer telemetry data in various databases. A part of our production environment runs in Amazon Web Services (AWS). We use encrypted EBS volumes to securely...
|
Developer, Engineering
This is the fourth post in a series on the Life of an SRE at Instana. Check out the first post, second post and third post. Detect certificate problems Here is an...
|

Start your FREE TRIAL today!

As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit https://www.instana.com to learn more.