How a Slack “Zero Width Space” character broke a Kubernetes Deployment – Lessons from an Instana SRE

Post

What is a Zero Width Space?

A few days ago I learned that the Unicode character for ‘ZERO WIDTH SPACE’ is U+200B.

“The zero-width space is a non-printing character used in computerized typesetting to indicate word boundaries to text processing systems.” Source: https://en.wikipedia.org/wiki/Zero-width_space

How Slack Broke My Kubernetes Deployment

It all started with a built-in alert (from Instana APM) that I got paged on. At first sight it looked like the container of our UI component was not able to start. The issue contains the starting time, the GKE node pool, GKE node name and the K8s pod that was not able to start.

So I started checking the pod status and it came up as “Pending”.

The “Conditions” tab for the pod shows a new scheduled deployment.

Checking the “Events” tab displays the full error message. For each event Instana shows the event type, reason, message, namespace and time so I can easily identify the event that I need to investigate next. The latest event contained a detailed error message that helped me to identify the root cause of the issue.

Here is the error message in a more readable format. As you can see the “zero width space” Unicode character somehow made it into the container image tag. Redeploying the UI component with the correct image version fixed the issue.

Failed to apply default image tag
"/ui-client:2.176.12\u200b7-0": couldn't parse image reference 
"/ui-client:2.176.12\u200b7-0": invalid reference format

So how did it get there? It turns out we received a hotfix deployment request for this UI component in our internal chat system (Slack). We copied the container image version from the message and used the version for the deployment. These “zero width space” characters are added for formatting reasons in Slack. Since this character has zero width, we did not see the hidden character when deploying the hotfix.

Searching the Internet it seems we are not the only ones experiencing this issue.

Lessons Learned – OR – How To Prevent Zero Width Space From Causing Problems

To avoid similar issues in the future we added regex checks to our deployment automation, so only valid characters are allowed when deploying components.

We use dedicated Jenkins servers for production deployments. Only SRE team member have access and can deploy releases and hotfixes to production. Jenkins gives us a nice audit trail, who and when components got deployed.

The Validating String Parameter Jenkins plugin allows us to specify valid parameters using regular expressions. This resolves the issue of whitespace characters ending up in the VERSION parameter, see https://plugins.jenkins.io/validating-string-parameter/

Small enhancement for Jenkins Job DSL plugin

We are using the Job DSL plugin for Jenkins to automatically generate all Jenkins jobs. The latest version does not support the validating string parameter yet. Therefore we created a pull request to add support for the validating string parameter plugin (https://github.com/jenkinsci/job-dsl-plugin/pull/1226).

Here is an example of what a job description looks like using validatingStringParam().

job('deploy-component') {
  parameters {
    validatingStringParam(VERSION, '', 'instana-release-\d+-\d+|1.\d+.\d+', '', '')
  }
  steps {
    shell('echo ${VERSION}')
  }
}

Play with Instana’s APM Observability Sandbox

Announcement, Featured, Product, Thought Leadership
Monitoring Kubernetes is an important part of a holistic management strategy considering the impact that orchestration can have on the performance and availability of application services. What is Kubernetes? Kubernetes (commonly k8s)...
|
Conceptual, Featured, Thought Leadership
A microservice architecture is flexible and dynamic but has the challenge of increasing complexity. For example, the picture below is an actual environment where hundreds of services collaborate with each other (a...
|
Customer Stories, Featured, Thought Leadership
Software is not static. As soon as you roll out an update, someone requests a new feature. Even if everything works fine, an infrastructure upgrade can break your code, and resolving one...
|

Start your FREE TRIAL today!

As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit https://www.instana.com to learn more.