Host Agent Configuration

Backend Configurations

The <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg contains the configurations used by the Host Agent to talk to the Instana backend.

The values of the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg file can be overridden via the following environment variables:

  • INSTANA_AGENT_ENDPOINT
  • INSTANA_AGENT_ENDPOINT_PORT
  • INSTANA_AGENT_KEY

Configure Multiple Backends

In some cases it might be desirable to have an agent report to multiple backends, for example for shared services that are used by otherwise strictly separated environments. It is possible to manually configure agents to do so. Note that the agent will be counted in all backends separately as consuming a license, and this setup will effectively multiply the bandwidth consumption of the agent.

To have the host agent report to multiple backends, perform the following steps:

  1. Rename the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg file to <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-1.cfg
  2. Create copies named <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-2.cfg, <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-3.cfg and so on with the appropriate configurations for the various backends the agent should report to.

Each of files created in the previous step then can be adjusted to describe a different host agent endpoint data and agent keys, and can even contain different proxy settings.

Note: Instead of 1, 2 and 3, any unique alphanumerical id can be used.

Note: If a file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg exists, all other backend files will be silently ignored.

Note: The Instana Host Agent Docker images are specially configured so that adding additional backends beyond the first is easily done by mounting the backend files beyond the first as com.instana.agent.main.sender.Backend-2.cfg and so on. An example argument for a dockerized agent is:

--volume <path-to-additional-backend-config>:/opt/instana/agent/etc/instana/com.instana.agent.main.sender.Backend-2.cfg

Setup an Agent Proxy

To communicate with the backend effectively, Instana uses the HTTP/2 protocol to transfer data. In many cases, direct communication from the host agent to the backend can be granted, simplifying the agents' deployment. In some cases, one dedicated entry into, and exit out of, the network is required. For this reason, Instana can be used in combination with a variety of proxies. In general, http(s), socks4s, and socks5 proxies are supported. The proxy has to support CONNECT pass through.

Note: A proxy that terminates the SSL connection and then tries to manage its own connection to the Instana backend is not supported. We use secure HTTP/2 without ALPN.

For the host agent configuration, you need to modify two files. In the file <instana-agent-dir>/etc/mvn-settings.xml the following information must be present (and not commented out) within the <settings></settings> section:

<proxies>
  <proxy>
    <id>agent-proxy</id>
    <active>true</active>
    <protocol>http</protocol>
    <username>user-if-needed</username>
    <password>password-if-needed</password>
    <host>your-proxy-address-goes-here</host>
    <port>your-proxy-port-goes-here</port>
  </proxy>
</proxies>

This enables the Instana agent to download agent updates and additional features automatically. Furthermore, the file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg needs to be reconfigured to use the proxy for the communication between the agent and the Instana backend. Please make sure the following lines are present and not commented out:

proxy.type=http
proxy.host=your-proxy-address-goes-here
proxy.port=your-proxy-port-goes-here
proxy.user=user-if-needed
proxy.password=password-if-needed
proxy.dns=true

Squid Proxy Configuration Example

This document describes the configuration of a squid proxy (www.squid-cache.org) in combination with Instana, when there is no other proxies available.

There are several ways to install Squid on your system. Most Linux distributions include Squid in their repositories and the software can be installed using the preferred package manager. In case no package is available, or you want to run Squid on Microsoft Windows, you can get Squid binaries from this location: http://wiki.squid-cache.org/SquidFaq/BinaryPackages. Once Squid is installed, an example configuration called squid.conf is created. If you want to use the proxy for the Instana communication exclusively, you can backup the default configuration and just use this minimal squid.conf:

# The tcp port squid is listening on
http_port 3128

# Please specify subnet with instana agents
acl instana_agent_net src 10.0.0.0/8

# This is the ip of the instana backend
acl instana_backend dstdomain saas-eu-west-1.instana.io
#acl instana_backend dstdomain ec2-54-144-114-141.compute-1.amazonaws.com
#acl instana_backend dstdomain saas-us-east-1.instana.io
#acl instana_backend dstdomain saas-us-east-1.instana.io

# This is the port used by Instana
acl instana_backend_port port 443

# This is the repo to download updates and additional sensors
acl instana_repo dstdomain artifact-public.instana.io
acl instana_repo_port port 80
acl instana_repo_port_secure port 443

# Protocol used for instana backend
acl instana_backend_proto proto HTTP

# Protocol used for instana backend
acl instana_repo_proto proto HTTP
acl instana_repo_proto_secure proto HTTPS

http_access allow instana_agent_net instana_backend instana_backend_port
http_access allow instana_agent_net instana_repo instana_repo_port
http_access allow instana_agent_net instana_repo instana_repo_port_secure

# DO NOT REMOVE THIS RULE!
http_access deny all

Configuring Host Agent Mode

Note: The One-Liner host agent installation method and the Instana Host Agent Docker image accept an additional mode, called AWS mode. The AWS mode of the host agent is not meant for monitoring hosts; rather, it is simply the INFRASTRUCTURE mode plus some automatic configuration of AWS data collection, as described in our AWS Agent documentation.

The host agent mode is set via the config file <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg:

mode = APM
# APM, INFRASTRUCTURE or OFF

Changing the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file requires a restart of the host agent to take effect.

Agent Configuration File

Note: The format of this file is YAML, which is very sensitive to whitespace. All indention levels only allow two blank spaces.

Most host agent configuration are applied using one central configuration file at the location:

<instana-agent-dir>/etc/instana/configuration.yaml

Multiple Configuration Files

To provide mudarity in configurations, it is possible to create files called <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml, with <suffix> replaced by an alphanumeric combination. All <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with <instana-agent-dir>/etc/instana/configuration.yaml in file system order. So configuration-cde.yaml comes after configuration-abc.yaml. Nested structures specified in different configuration files like tags are merged; values are overwritten.

Integration with Secret Managers

The host agent can retrieve confidential data like credentials or other information that you do not want to put in clear text in the configuration.yaml file from secret managers.

Vault

Note: The Vault integration requires agent boostrap version 1.2.9 or higher

Instana agent uses HashiCorp Vault in order to securely obtain values for sensitive settings in the agent configuration file. The following Vault integration configuration parameters should be provided to the Agent:

com.instana.configuration.integration.vault:
  connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200)
  token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths 
  path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
  kv_version: 2 # The Key/Value secrets engine version, default is 2

After the initial Vault configuring, you can specify the retrieval of secrets for the various sensors like follows:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key: 
        path: <vault_path>
        key: <vault_secret_key>

For comparison, to configure the password in plain-text in the configuration.yaml file, this would have been the snippet:

com.instana.plugin.mysql:
  user: 'instana'
  password: <my_password>

That is, you can simply swap out the string value for the YAML structure specifying the Vault coordinates, and the host agent will automatically do the rest.

Also, you are not limited to using the Vault integration with a field containing password, but you can use it for every configuration that has a string as value in YAML.

For more information, please refer to the following Vault concepts, relevant for the Instana agent integration:

Configurations From Process Environment & Files

You may have cases in which you do not want to hardcode credentials or other configuration items in your configuration.yaml. Or you may have several instances of one technology (say, several MySQL nodes) on the same node, each needing different configurations. In any cases like these, the host agent is capable of reading configuration values (think of them as literals in YAML, like a string, a boolean or a number) from the process to be monitored (e.g., the MySQL database) looking the values up from specific environment variables or as content of specific files.

Notice that there is an intentional symmetry between these capabilities of the host agent and the way you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.

Note: Numeric values in environment variables and files will be converted to integer values, while the boolean values true and false will be converted to to booleans.

Configurations From Process Environment

For configuration values taken from the process environment, the configuration.yaml looks like the following:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: env
      env_name: <environment variable name>
      default_value: <set this when no env var is found on this process>

If you do not specify a default value and the environment variable is not present on the process to be monitored (in this case, the MySQL database), it is like you never entered a value in the configuration.yaml file.

Configurations From Process Filesystem

For configuration values taken from file the configuration.yaml looks like the following:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: file
      file_path: <absolute path to a file containing the configuration value>
      default_value: <set this when the file is not found>

If the process to be monitored is running inside the container, the file absolute path is resolved against the container's filesystem.

Note: Reference files on the host's operating system when trying to resolve a configuration value for a containerized process is not supported.

Monitor Additional Filesystems

Note: These configurations are applied via the agent configuration files.

By default the agent will monitor local filesystems only. To add additional filesystems, add the name of the filesystem (that is the first column in mtab or df).

For example to monitor the following filesystem:

server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr

please add the following line in the uncommented filesystems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Specify Host Tags

Note: These configurations are applied via the agent configuration files.

To specify tags for an agent in the com.instana.plugin.host, add a section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, the INSTANA_TAGS=production,app1 environment variable can be used (tags from environment variable and config file are additive).

This adds two tags to the specific agent, enabling searching and filtering of tags in the UI:

infrastructure tags

Extract the Installed Packages List

Note: These configurations are applied via the agent configuration files.

The Host Agent can report to Instana which packages are installed on the underpinning Linux operating system.

Note: This configuration is disabled by default.

The following Linux distributions are currently supported:

  • Debian and derivatives that use dpkg as package manager
  • RedHat and derivaties that use rpm and yum as package managers

When the collectInstalledSoftware property is set, installed packages on an operating system are extracted once a day.

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

package list

Custom Zones

Note: These configurations are applied via the agent configuration files.

Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts by default. To customize the default host grouping in Instana, a new group for the specific host can be defined in the section com.instana.plugin.generic.hardware using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

It is also possible to provide this configuration using the INSTANA_ZONE environment variable, which then override the zone specified in the configuration file.

The result is that hosts are grouped into zones on the infrastructure map:

infrastructure zones

Monitor Custom Processes

Note: These configurations are applied via the agent configuration files.

Instana will automatically monitor process metrics of higher level sensors like Java or MySQL by default. Should you want to monitor an OS process not automatically covered by Instana, you can configure it using both the process name and/or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Secrets

Tracing data may contain sensitive data. Therefore, the Instana agent supports the specification of patterns for "secrets", that is, data to be redacted agent-side from the tracing data. Data treated as secrets will not reach the Instana SaaS for processing and, thus, will not be available for analysis in the UI or retrieval via API.

Secrets are specified in the following way:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If a key matches an entry from the list, the value is redacted and not sent to the Instana backend. If you do not specify any custom secrets configuration, Instana will use the aforementioned secrets configuration by default.

Note: Generally, secrets are filtered from static data. Filtering is highly optimized and changes to which secrets to scrub will not be applied while a process is running. To make a secret configuration effective, either restart the agent or the running monitored component, which will respect the new config.

Instana supports secrets at infrastructure and platform level as follows:

  • Process environment variables
  • Docker container info
  • Kubernetes annotations and container environment variables (requires an additional configuration, see Kubernetes secrets for more details)

Instana supports secrets in runtimes as follows:

  • JDBC connection strings
  • HTTP query parameters (virtually for all supported runtimes, see support matrix below), e.g., https://my.domain/accounts/status?account=<secret_1>&user=<secret_2>
  • HTTP matrix path parameters, e.g., https://my.domain/accounts/account=<secret_1>;user=<secret_1>/status (support available for some runtimes, see support matrix below)
Language Secrets in HTTP Query parameters Secrets in HTTP Matrix parameters
Java
Go 1
.NET Framework
.NET Core
.NET Framework
NGINX
Node.js
PHP
Python
Ruby
Crystal

Note: Instana does not support treating as secrets arbitrary segments in a URL paths, e.g., in case the URL is structured as https://my.domain/accounts/<secret>/status. The overhead of running, for example, regular expressions on URL paths to discard segments would simply be prohibitive.

Note: Although it is not really a matter related to secrets, the Instana does not capture the following information related to interaction with databases:

  1. SQL parameters from stored procedures
  2. Connection credentials

Similarly to the case of arbitrary path segments, we do not strip literals from SQL queries, and strongly advise the use of parametrized queries instead. The only exception to this lies with Instana's PHP support, which has an option to strip SQL literals (see the sanitizeSql configuration option), at a potentially rather steep cost in terms of overhead of the Instana agent.

Capture Custom HTTP Headers

Note: These configurations are applied via the agent configuration files.

By default, Instana will not collect HTTP headers when tracing HTTP calls, but this feature can be enabled on demand as follows:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers thus collected will be shown in the call details (in the "callee details" section). You can also the header names and their values to search for calls and traces (in the UI as well as via the API). Finally, they can be used for service configuration.

Restrictions:

  • Instana only captures request headers on HTTP entries (HTTP calls that the instrumented process receives).
  • Instana captures response headers on HTTP entries in Java and Node.js.
  • Instana does not capture headers on HTTP exits (HTTP calls where the instrumented process is the client and the HTTP server is not instrumented).
  • This feature is currently supported in Java, .NET, NodeJs, PHP, Python, Ruby and Go, when using the net/http instrumentation.

Ignore Processes

Note: These configurations are applied via the agent configuration files.

To ignore any monitoring of certain processes, uncomment the com.instana.ignore section and list all process names of processes that should not be monitored. Sometimes scripts or other jobs are started through the same command, but only the execution of a single command should be ignored. In this case use arguments to specify any argument that should cause a process to be ignored.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivate Agent Capabilities Triggered by UI

In order to tail the Instana Agent logs, perform diagnostic actions on the agent and to enable the Trace Code View, users can trigger those capabilities through the Instana UI. In case you want to deactivate all of these capabilities, you can set the config accordingly by adding to: <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg

backchannel.enabled = false

Note: Commands are not send directly to the Instana Agent rather they are received as response to sending new metrics.

Upload Codesource Files to Instana

The Instana agent is capable of retrieving on-demand the source code of the processes it monitors and associate it with the collected tracing data.

In order to disable the on-demand upload of source files into Instana, you can set the following configuration in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg file:

source.download.enabled = false

Note: The Instana Agent needs to be restarted to apply this setting.

Agent Logging

By default, the Instana Agent will log to <instana-agent-dir>/data/log/agent.log, which will also be rotated should the file grow too large. If you are running the agent in a container, it logs to the console instead, which is managed by Docker and accessible via docker logs.

The log level can be increased to debug by changing the level in the configuration file <instana-agent-dir>/etc/org.ops4j.pax.logging.cfg from:

log4j2.logger.instana.level=INFO

to:

log4j2.logger.instana.level=DEBUG

Note: ensure that there is no trailing whitespace on the line.

Older agent installs need this line to be changed from:

log4j.logger.com.instana=INFO, out, osgi:*

to:

log4j.logger.com.instana=DEBUG, out, osgi:*

Log Rotation

By default, the agent uses a log rotation of 10 times 5MB agent log files.

log4j2.appender.rolling.policy.type = SizeBasedTriggeringPolicy
log4j2.appender.rolling.policy.size = 5MB
log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy
log4j2.appender.rolling.strategy.max = 10

Syslog

The Agent now uses Log4j2, which is a modern and flexible logging facility. For example, syslog can be configured like this:

log4j2.rootLogger.appenderRef.Syslog.ref = Syslog
log4j2.rootLogger.appenderRef.Syslog.level = ERROR
log4j2.appender.syslog.type=Syslog
log4j2.appender.syslog.name=Syslog
log4j2.appender.syslog.layout.type=PatternLayout
log4j2.appender.syslog.layout.pattern = ${log4j2.pattern}
log4j2.appender.syslog.facility=SYSLOG
log4j2.appender.syslog.host=localhost
log4j2.appender.syslog.port=514
log4j2.appender.syslog.protocol=UDP

Logs to STDOUT

In custom container images, you may want to log to STDOUT. You can do so by providing the following configuration in <instana-agent-install-dir>etc/org.ops4j.pax.logging.cfg:

log4j2.rootLogger.appenderRef.Console.ref = Console

log4j2.appender.console.type = Console
log4j2.appender.console.name = Console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = ${log4j2.pattern}

Log4j2

Because the logging is using the standard log4j2 logging, it can be configured in many more ways as described in the log4j2 documentation.

Log metrics or traces to file

The agent can temporarily log metrics or traces being sent through that agent to a file on disk. This is often used to debug various issues related to metrics, traces, tracing or spans.

To enable this, locate the file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.File.cfg and update it's contents with:

# Configuration of local logging. Changes will be hot-reloaded.
# Activate logging of outgoing payloads to local disk by setting a non-empty
# prefix. The log file will be written to data/log, and the file will have the
# defined prefix followed by a timestamp.
# Note: There is no automatic rotation of those files.
prefix=locallog

# The file can be filtered to either "metrics" or "traces".
# If empty or absent, there will be no filtering.
type=traces

As noted, the changes will be hot-reloaded and be activated immediately. This should only be done temporarily as leaving this enabled has the potential to fill up all available disk space given enough time and traffic.

Allow the logging to continue for a minute or two while generating traffic to the component in question (in the case of tracing issues). Then revert the changes or comment all lines in that file. Again, changes written to disk will be hot-reloaded by the Instana agent.

If you are working with a support ticket, please attach the resulting log file to the support ticket. The log file will be located in <instana-agent-dir>/data/log/locallog_*.log.

Limiting Host Agent CPU and Memory

In certain scenarios it is crucial to control resource consumption of processes very closely. This can become especially handy in environments that leverage resource sharing and for systems with extremely low resources. While the Instana agent is designed to consume as few resources as possible it can be limited following the instructions below as another safeguard to respect clear resource limits.

All following examples limit Agent's CPU to computing resources equivalent to half a CPU (however, not all resources would necessarily come from the same CPU) and to a maximum of 512 MB memory.

Systemd

Create file called /etc/systemd/system/instana-agent.service.d/20-resource_limits.conf and add the following content to it:

[Service]
CPUAccounting=true
CPUQuota=50%
MemoryAccounting=true
MemoryMax=512M

Run systemctl daemon-reload and restart the instana-agent service.

Docker

Run the instana-agent container with the following additional parameters:

Docker version 1.13 and newer:

--cpus=0.5 --memory=512m

Docker version 1.12 and older:

--cpu-period=100000 --cpu-quota=50000 --memory=512m

Kubernetes

Add the following configuration snippet to the agent's container configuration:

livenessProbe:
  httpGet: # Agent liveness is published on localhost:42699/status
    path: /status
    port: 42699
  initialDelaySeconds: 75
  periodSeconds: 5
resources:
  requests:
    memory: "256Mi"
    cpu: "0.5"
  limits:
    memory: "512Mi"
    cpu: "1.0"

The configuration above also requests less memory and CPU for the instana-agent container.


  1. Only when using the net/http instrumentation