Monitoring Google Cloud Run

This page describes Instana's support for Google Cloud Run.

Instana currently only supports the fully managed platform. Cloud Run on Anthos / GKE is currently not supported.

The support for Cloud Run rests on two pillars:

  1. an Instana agent monitoring the GCP project and its resources, and
  2. the Instana in-process collectors integrated into the container image you use on Cloud Run.

Setting up an Instana agent to monitor a GCP project will provide metrics for the Cloud Run service revisions the project contains. Adding an Instana in-process collectors to the Docker image will add visibility into the individual instances that Cloud Run starts, provide runtime specific metrics (JVM metrics, Node.js metrics, .Net metrics, ...) and it will also provide tracing of calls.

IMPORTANT: Monitoring the GCP project with an Instana agent is a prerequisite for using a Cloud Run in-process collector.

GCP Agent Setup

You need to configure one Instana agent to monitor the GCP project you are interested in. A dedicated machine should be used to monitor GCP resources. If you want to monitor multiple GCP projects, you need one agent per project. The remainder of this documents refers to the Instana agent set up to monitor a GCP project as the "GCP agent".

The Instana Google Cloud integration uses service accounts to connect to the GCP admin API from the GCP agent. Below are instructions for creating the service account and providing its credentials to the GCP agent so it can make API calls on your behalf. If you have already set this up for another GCP service (Google Cloud Storage, Google Cloud SQL, ...) you can skip creating a new service account but you might need to add additional permissions for Cloud Run (see below).

  1. Navigate to Google Cloud credentials page for the Google Cloud project you would like to setup the Instana integration.
  2. Press Create credentials and then select Service account key. SelectServiceAccount
  3. In the Service account dropdown, select New service account.
  4. Give the service account a unique name.
  5. Add the role "Cloud Run Viewer" to the account. Alternatively, you can also create a custom role which has at least the following permissions:

    • resourcemanager.projects.get,
    • run.revisions.list,
    • run.services.list, and
    • monitoring.timeSeries.list.
  6. Select JSON as the key type, and press create.
  7. Take note where this file is saved, as it is needed to complete the integration.
  8. Add an attribute named credentials_path to the GCP agent's configuration.yml file under com.instana.plugin.gcp (for all GCP resources) or com.instana.plugin.gcp.run (credentials just for monitoring Cloud Run). The value for this attribute is the path to the credentials file. (See section Configuration for an example.)

In order to monitor a GCP project, the following services need to be enabled:

Configuration

You can configure the rate, at which the GCP agent polls the metric API in seconds by adding the following configuration to the configuration file of the GCP Agent:

com.instana.plugin.gcp.run:
  enabled: true
  poll_rate: 60  # How often Google's metrics API will be polled
  credentials_path: '/opt/instana/credentials/cred-gcp.json'  # Path to Service Account credentials
  include_tags:
    ...
  exclude_tags:
    ...

Filtering

Multiple tags can be defined, separated by a comma. Tags should be provided as a key-value pairs separated by the : character. In order to make configuration easier, it is possible to define which tags you want to include in (or exclude from) the discovery process. In case a tag is present in both lists (include and exclude), the exclude list has higher priority. If there is no need for service filtering, the attributes include_tags and exclude_tags should be omitted.

To include and/or exclude Cloud Run service revisions by tags use the following configuration:

com.instana.plugin.gcp.run:
  include_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)
  exclude_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)

In-Process Collectors

This section describes the required steps to collect tracing data, runtime-specific metrics and additional data about the Cloud Run service revision instances. The steps in this section assume you have already followed the steps described in the GCP Agent Setup steps.

Node.js

The Node.js Cloud Run in-process collector is delivered over the instana/google-cloud-run-nodejs Docker image. You can use in multi-step Docker builds to streamline and speed up the building of Docker images for Node.js applications running on Cloud Run as follows.

Add the following lines to your Dockerfile anywhere after the last FROM clause:

COPY --from=instana/google-cloud-run-nodejs:latest /instana /instana
RUN /instana/setup.sh
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

Build the container and push it to the image repository of your choice.

Finally, follow the configuration steps described in the section Configure Your Cloud Run Service.

Note: If you are using Multi-stage builds (that is, if you have multiple FROM clauses in your Dockerfile), you need to add these lines below the last FROM clause of your Dockerfile.

Note: Auto-profile of Node.js processes running in Cloud Run is currently not supported.

Build Dependencies and Native Add-Ons

If you are using a rather minimal base image for your Node.js containers which does not have the required packages to build native Node.js add-ons, you might see a warning like this in your Docker build:

gyp ERR! ...some error message from trying to rebuild native add-ons, for example:
gyp ERR! stack Error: Could not find any Python installation to use
...
Warning: Rebuilding native add-ons for @instana/google-cloud-run failed. Monitoring the Cloud Run service revision instance will work nonetheless, but you will miss some Node.js metrics (GC metrics, event loop metrics, ...). See https://www.instana.com/docs/ecosystem/google-cloud-run/#build-dependencies-and-native-add-ons for details.

As the log message implies, monitoring your Node.js Cloud Run service revision instance with Instana will work. But certain metrics will be missing. Instana recommends to install the missing operating system packages, at least temporarily, during the build. They do not need to be included in the final image. Which packages to install and how to install them depends on the distribution you are using.

Here is an example Dockerfile snippet for Linux Alpine:

COPY --from=instana/google-cloud-run-nodejs:latest /instana /instana
RUN apk add --no-cache --virtual .gyp \
        build-base \
        python \
    && /instana/setup.sh \
    && apk del .gyp python
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

Here is an example for Debian based distributions (including Ubuntu):

COPY --from=instana/google-cloud-run-nodejs:latest /instana /instana
RUN apt-get update \
    && apt-get install build-essential \
    && /instana/setup.sh \
    && apt-get remove -y build-essential
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

For distributions that use yum (CentOS, RHEL, ...), the command to install the required packages would be:

RUN yum groupinstall "Development Tools"

Getting The Node.js Layer From containers.instana.io

If you also use Instana's Cloud Run support with other runtimes (like Java), for which the image layer is only avaible on our authenticated registry containers.instana.io and you want to keep things consistent, you can also get the Node.js Instana Cloud Run image layer from containers.instana.io rather than Docker Hub, as an alternative to the approach outlined above.

Since the containers.instana.io registry requires authentication, you need to execute the following command before using it in your Docker build:

docker login -u _ -p <your-agent-key> containers.instana.io

The placeholder <your-agent-key> needs to be replaced with your actual Instana agent key. This needs to happen on the machine where you build your Docker image.

With that out of the way, add the following lines to your Dockerfile anywhere after the last FROM clause:

COPY --from=containers.instana.io/instana/release/google/cloud-run/nodejs:latest /instana /instana
RUN /instana/setup.sh
ENV NODE_OPTIONS="--require /instana/node_modules/@instana/google-cloud-run"

Go

The Go Fargate collector is included in github.com/instana/go-sensor starting from v1.22.0 and does not require additional configuration apart from setting up the environment variables described in Configure Your Cloud Run Service section. The tracer detects that the app is running on Cloud Run and automatically activates the right capabilities.

.Net Core

Instana's support for .Net Core applications running in Cloud Run requires the following steps:

  1. include the Instana Nuget packages to your application
  2. set up the environment variables described in Configure Your Cloud Run Service section

And you are good to go, all the rest is entirely automated.

Adding the NuGet packages to your container image

Here is an example of how to modify the Dockerfile to add Instana's NuGet packages at build time of your Docker images:

# If running on Alpine or muslc-based images
RUN dotnet add Instana.Tracing.Core.Rewriter.Alpine
# Glibc-based images like RHEL and Debian derivatives
RUN dotnet add Instana.Tracing.Core.Rewriter.Linux

# set CoreCLR tracing environment variables
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={cf0d821e-299b-5307-a3d8-b283c03916dd}
ENV CORECLR_PROFILER_PATH=/app/instana_tracing/CoreProfiler.so
ENV DOTNET_STARTUP_HOOKS=/app/Instana.Tracing.Core.dll

IMPORTANT: We offer two different packages based on thee flavor of lib your containers use:

  • muslc, used by Alpine-based containers, need the Instana.Tracing.Core.Rewriter.Alpine package
  • glibc, used by pretty much all non-Alpine distributions, needs Instana.Tracing.Core.Rewriter.Linux.

IMPORTANT: Replace in the above /app/ with the actual path to the root of your application.

IMPORTANT: Remember to set up the environment variables described in Configure Your Cloud Run Service section.

Configure Your Cloud Run Service

Note: These instructions are common to the setup of all Instana in-process collectors on Cloud Run.

Note: Before following the steps described in this section, make sure your container images have been built according to the instructions specific to the runtime that your application uses (see above).

All Instana in-process collectors on Cloud Run need a common set of configuration values to be able to connect to and authenticate with the Instana back end:

  1. Create a new revision of the Cloud Run service:

    Creating a new Cloud Run service revision

  2. In the service revision configuration page, go to "Advanced Settings" -> "Variables:

    Edit the container definition

  3. Add the following two environment variables to the container specification:

Cloud Run Service Revision - Metrics Collection

Collected Configuration Data

The following static configuration data is collected for each service revision:

  • region,
  • service name,
  • configuration name,
  • revision name,
  • project ID,
  • numeric poject ID (project number),
  • creation time of the revision,
  • revision labels

For Cloud Run services with multiple revisions the GCP agent collects data for the most recent revision.

Performance Metrics

Metric Description
Request Count Number of requests reaching the revision. Excludes requests that are not reaching your container instances (e.g. unauthorized requests or when maximum number of instances is reached).
Request Latency 99th Percentile 99th percentile for the distribution of request latency in milliseconds (for requests reaching the revision).
Request Latency 95thh Percentile 95th percentile for the distribution of request latency in milliseconds (for requests reaching the revision).
Request Latency 50th Percentile 50th percentile for the distribution of request latency in milliseconds (for requests reaching the revision).
Billable Instance Time Billable time aggregated from all container instances of the revision. For a given container instance, billable time occurs when the container instance is starting or at least one request is being processed. Billable time is rounded up to the nearest 100 milliseconds.
Container Memore Allocation 99th Percentile 99h percentile of the container memory allocation of the revision in Gigabytes-seconds.
Container Memore Allocation 95th Percentile 95h percentile of the container memory allocation of the revision in Gigabytes-seconds.
Container Memore Allocation 50th Percentile 50h percentile of the container memory allocation of the revision in Gigabytes-seconds.

Granularity for all metrics is 60 seconds.