Monitoring .NET

Note: This page is about our support for .NET Full Framework based applications. If you are looking for support on .NET Core, please follow this link.

Supported Versions

Instana supports monitoring of applications running on the .NET-Framework beginning with version 4.0.

Runtime Support status
.NET Framework 4.0 and above GA
.NET Core GA

Supported Libraries & Frameworks for tracing

HTTP Version
ASP.NET MVC >= 5.0, >= 5.1, >=5.2.2
ASP.NET WebAPI >= 5.0, >= 5.1, >=5.2.2
ASP.NET WebForms >= 4.5.2
ASP.NET Core 1.0.2, 1.0.3, 1.1.2, 1.1.3, 1.1.7, 2.0.0, 2.0.1, 2.0.2, 2.2.0
System.Net.Http.HttpClient >= 4.5.2
System.Net.HttpWebRequest >= 4.5.2
Databases Version
Microsoft SQL-Server (System.Data.SqlClient) >= 1
Oracle (ODP, Devart) 2.102.5, 2.112.1, 4.112.2, 4.112.3, 4.121.2
PostgreSQL (NpgSql) >= 2
Cassandra 3.4.0, 3.7.0
Elastic Search 6.0.0
MongoDB >= 1.8.0, 2.4.4, >= 2.10.0
Redis Service Stack >= 5.0
Redis Stack Exchange >= 2.0
Memcached 2.11.0, 2.16.0
Sql (System.Data) >= 4.5.2
ORM Version
Entity Framework >= 4.5.2
Remoting Version
Windows Communication Foundation Services >= 4.5.2
.NET Remoting >= 4.5.2
Messaging Version
Microsoft Message Queue >= 4.5.2
EasyNetQ 3.3.9
RabbitMQ 3.6.9, 4.1.0, 5.0.1
Logging Version
NLog >= 4
Log4Net 1.2.11, 1.2.14, 1.2.15, 2.0.7, 2.0.8
Serilog 2.8.0
Schedulers / Task-Management Version
Hangfire 1.6.20
Quartz 3.0.7

Sensor Data Collection

Tracked Configuration Metrics
Name GC Activity
Version Memory Usage
Arguments Thread-locks
Contention

Health Signatures

For each sensor, there is a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents depending on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any given entity.

For information about built-in events for the .NET sensor, see the Built-in events reference.

Tracing

This page describes how to setup .NET Tracing for your environment and how it works.

Setting up Tracing for .NET

The tracing component setup is a mostly automatic function of the agent. However, there are is a manual step involved, which we describe in the following sections.

Enabling tracing for .NET

When the Instana agent starts on your machine running windows, it will install the tracing-extension for .NET. The extension is composed from several files which contain the actual tracing-components. Since Instana uses the Profiling API defined in the .NET framework, this involves COM-Libraries as well as managed assemblies. When tracing is enabled, the files will be registered appropriately, so that all applications can use them for tracing. The sensor and tracing-extensions are enabled by default.

Upon startup of the agent, InstanaPCP will log information about the process of trace-enablement. This will look like this (log lines shortened):

configuration.yaml with .NET-Tracing enabled

INFO  | com.instana.agent - 1.1.326 | Writing location for instrumentation-advices to registry
INFO  | com.instana.agent - 1.1.326 | Setting up tracing-environment
INFO  | com.instana.agent - 1.1.326 | Registering native profiler for .NET
INFO  | com.instana.agent - 1.1.326 | Agent UUID received:ac:bc:32:ff:fe:90:7d:bb
INFO  | com.instana.agent - 1.1.326 | Registering System.Collections.Immutable.dll in GAC
INFO  | com.instana.agent - 1.1.326 | .NET Tracing is enabled
INFO  | com.instana.agent - 1.1.326 | Registering instrumentation-DLLs in GAC
INFO  | com.instana.agent - 1.1.326 | Registering Instana.ManagedTracing.dll in GAC
INFO  | com.instana.agent - 1.1.326 | Setting environment-variables for .NET tracing
INFO  | com.instana.agent - 1.1.326 | Registry-Key for WAS-Environment does not exist, will create now
INFO  | com.instana.agent - 1.1.326 | Setting environment for WAS
WARN  | com.instana.agent - 1.1.326 | You should consider restarting the service WAS in order to make it use the new profiler. Already running application will not use it until restarted.
INFO  | com.instana.agent - 1.1.326 | Registry-Key for W3SVC-Environment does not exist, will create now
INFO  | com.instana.agent - 1.1.326 | Setting environment for W3SVC
WARN  | com.instana.agent - 1.1.326 | You should consider restarting the service W3SVC in order to make it use the new profiler. Already running application will not use it until restarted.
INFO  | com.instana.agent - 1.1.326 | Setting memoryUsageExcessThreshold set to configured value of 100000000
INFO  | com.instana.agent - 1.1.326 | Self-Monitoring for Windows-Extensions has been started successfully
INFO  | com.instana.agent - 1.1.326 | Setting memoryUsageWarningThreshold set to configured value of 80000000
INFO  | com.instana.agent - 1.1.326 | Starting tracing-session
INFO  | com.instana.agent - 1.1.326 | Windows-Extension REST-Endpoint has been setup successfully at http://localhost:42657
INFO  | com.instana.agent - 1.1.326 | Registering Instana.ManagedTracing.Modules.dll in GAC
INFO  | com.instana.agent - 1.1.326 | Instana ETW-Reader setup and listening

The extension registers a native COM-DLL (profiler), along with 3 managed assemblies, which contain the actual instrumentation code or are required for the instrumentation to work (Instana.ManagedTracing.dll, Instana.ManagedTracing.Base.dll, System.Collections.Immutable.dll). After registering the components, the extension applies changes to the registry, so that any instances of W3SVC or WAS will attach the profiler the next time they are being started. This is why you will find a warning in the logs regarding restarting already running applications. Instana does not restart these services automatically, as we do not want to interrupt your services.

How Tracing Works

Once you enabled tracing by changing the configuration-file, trace-data will be collected for all applications that make use of the supported technologies. Since .NET-Tracing follows the same scheme as all the other tracing-packages, distributed traces in polyglot applications will automatically be collected and displayed in Instana's UI.

When an application with tracing enabled starts up, you might find the startup-time to be increased in comparison to startup without tracing. This is due to the profiler attaching, inspecting the classes contained in the referenced assemblies, and then rejitting some methods' code for instrumentation. After the instrumentation has been applied, the impact on the runtime performance of your applications is extremely low, as with any other tracing-package that Instana provides.

Disabling Tracing for .NET

When you want to stop tracing of .NET applications for whatever reason, you can simply do so by changing the configuration file back to its initial state, or by just setting the corresponding flag to "false."

configuration.yaml with .NET-Tracing disabled

# CLR Tracing
com.instana.plugin.clr:
  tracing:
    enabled: false

After changing the config back, you will have to restart the agent for the changes to take effect.

Due to the nature of how the Profiling API works, disabling tracing in the agent will not remove / detach the profiler from processes that have been started with tracing enabled (i.e. your worker-processes). To make sure the profiler is not loaded into the application, you have to restart the host-process.

Enabling Tracing for Non-IIS Hosted Applications

Our tracing does not enable tracing for every .NET based applications on purpose, since this might produce undesired component behavior that are not in scope of your monitoring needs. While IIS and WAS get configured to attach the profiler automatically by setting the appropriate environment-variables, standalone-processes are not. You can enable tracing for these processes by providing the necessary environment-variables by either setting them on the user level (when you use specific technical users which run your applications), or by setting the environment-variables before starting the actual process.

To enable tracing for a specific process you have to provide the following environment-variables with their respective values:

Environment variables needed for tracing

SET COR_ENABLE_PROFILING=1
SET COR_PROFILER={FA8F1DFF-0B62-4F84-887F-ECAC69A65DD3}

You can put these settings into a batch or powershell-script which starts your processes after setting the env-vars. If you are running Windows-Services you can also put these settings into the registry, on a per-service basis, which gives you better control over which services to trace.

Navigate to HKLM\SOFTWARE\System\CurrentControlSet\Services\<ServiceName> and add a MULTI_SZ value by the name of Environment (if it doesn't already exist), then copy the variables above into that value.

To correctly set the contents of a MULTI_SZvalue, put one key-value-pair per line like displayed below:

COR_ENABLE_PROFILING=1
COR_PROFILER={FA8F1DFF-0B62-4F84-887F-ECAC69A65DD3}

Hit enter after last line.

Custom Tracing

Instana's .NET Tracing SDK enables to your own custom tracing in situation where our automatic instrumentation does not apply (i.e., you are using components we are not automatically supporting).

Custom Performance Counters

We also support monitoring of Custom Performance Counters. To enable them, make sure to enable the plugin in configuration.yaml file like this:

com.instana.plugin.perfcounters:
  enabled: true

Note that this plugin is disabled by default.

Performance counters are defined with 4 different values. They are grouped by category, they have a name, and they can be bounded to a specific instance, or they can be host-wide. To uniquely describe a counter, often we use a syntax like this:

[host]\[category]([instance])\[counter name]

instance can in some cases be optional - it depends on the counter. Typical example of a counter would be:

\Processor Information(_Total)\% Processor Time

In this example

  • host is empty - meaning it is a counter for current host (localhost)
  • category is Processor Information
  • instance is _Total
  • counter name is % Processor Time

Our implementation can only query localhost counters.

It is worth noticing that some categories are only host-wide, meaning that they don't have instance defined. An example of such a category is System. In that case, we would have:

\System\Processor Queue Length

Once the plugin is enabled, the next thing would be to define which counters you would like to monitor. This can be done by extending configuration:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: [name of the counters category]
      instance: [instance name]
      counters:
        - [name of the counter]

Using the previous example, the configuration would look like this:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: Processor Information
      instance: _Total
      counters:
        - "% Processor Time"

Note that counter name is enclosed with "" because of the special character %

If you need more counters per category, just add them to the counters array like this:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: Memory
      counters:
        - Page Faults/sec
        - Cache Faults/sec
        - Page Reads/sec

Note that here we didn't define instance because Memory category is host-wide.

Please note that querying performance counters is an inherently expensive operation. We made sure to optimize our solution, but having a lot of counters will impact the performance of the agent.

Here is another example with multiple categories and counters:

com.instana.plugin.perfcounters:
  enabled: true
  counters:
    - category: System
      counters:
        - Processor Queue Length
        - System Calls/sec
    - category: Process
      instance: _Total
      counters:
        - Private bytes
        - Working Set
        - Thread Count
    - category: Memory
      counters:
        - Page Faults/sec
        - Cache Faults/sec
        - Page Reads/sec
    - category: .NET CLR Exceptions
      instance: _Global_
      counters:
        - "# of Exceps Thrown / sec"
        - "# of Pinned Objects"

Finally, here is an example of a host dashboard with some performance counters:

Example of Performance Counters dashboard

Troubleshooting

Environment variable not defined

Monitoring issue type: clr_env_var_not_defined

One or more environment variables, which are necessary to activate the tracing of .NET application, are not set correctly.

If this happens for applications hosted on IIS, it might be that the iisreset has not been issued at all, or before the agent could register the libraries needed for tracing. As a solution, please issue an iisreset from an elevated prompt.

If this does not help, consider restarting the machine, which might be needed depending on the version of Windows you are running and whether there were (un-)installations happening before running the instana agent.

For applications not hosted on IIS, please refer to the Enabling Tracing for Non-IIS Hosted Applications section for the list of environment variables and the respective values you need to set for tracing your .NET applications.

Environment variable has an invalid value

Monitoring issue type: clr_env_var_invalid_value

At least one environment variable that is necessary to activate the Instana .NET tracing, does not have the right value. (Depending on which environment variables are missing, you may still see metrics in the Instana dashboards for your .NET application, but definitely not traces.) Please refer to the Enabling Tracing for Non-IIS Hosted Applications section for the list of environment variables and the respective values you need to set for tracing your .NET Core applications with Instana.

Also, it might be the case that the reason for the environment variable to have the wrong value is the presence of another APM tool. (In terms of hooking into the .NET runtime, the mechanisms the various APM use are pretty similar.) If we recognize this situation, you will see it in the message in the Instana user interface. Before Instana can trace your .NET Core application, you will have turn off the other APM agent, at least insofar .NET Core monitoring is concerned; for how to do that, refer to the other APM tool's documentation.

InstanaPCP not running

Monitoring issue type: clr_instana_pcp_not_running

You might find warning messages in your agent's log, stating that InstanaPCP does not seem to be running. This message is triggered whenever the agent tries to interact with InstanaPCP but fails to do so. Usually the agent starts InstanaPCP early in the bootstrapping-process and can then communicate with it. Under the following circumstances, InstanaPCP might not be able to start:

  • The agent is not running with sufficient priviliges (i.e. Local System / Administrator)
  • The host does not have .NET Framework 4.5 installed
  • InstanaPCP restarted itself due to memory overconsumption

In case you see this message repeatedly appear, check for the following:

  1. Make sure that your agent runs with a user that has local administrative priviliges. If this is not the case, change the user used to run the agent, or grant it local administrative priviliges. When installing Instana from the installation-package, the service will be run as Local System normally and thus have sufficient priviliges.
  2. Check which version of the .NET Framework you have installed on your machine. The Windows-Installer package will automatically install the minimum version InstanaPCP needs, but if you happen to install the agent from a ZIP-archive, you might have to install the framework yourself. If you are unsure whether you have the correct version or not, you can try to run InstanaPCP manually from a command-prompt by locating it under <agent-dir>\data\repo\com\instana\agent-windows-extensions\<latest>\agent-windows-extensions-<latest>. If you do not have the correct framework-version installed, you will see an according prompt.
  3. If you see messages in the log that point to over-consumption of memory like these:

    InstanaPCP.exe Private-Bytes is above warning-treshhold for XXX times now
    
    Killing InstanaPCP.exe for excessive memory-usage. Possible memory-leak in native Windows-Libraries.

These messages can point to one of two potential issues:

  1. You are running on a Windows-Machine with an old version of IIS7 having a specific memory-leak. This leak can be patched by installing a Microsoft-Update. This might happen on Windows 2008 machines. Before installing this patch, please contact the Instana Support to make sure that we are looking at this specific issue. The support-team will then provide you with the link to the update package.
  2. You are running a lot of .NET applications on the server and those applications are generating a lot of traces. In this case, you might need to increase the memory-limits for InstanaPCPby applying changes to these settings in your configuration.yaml file:

    com.instana.plugin.clr:
     pcp_warn_limit: <MB-treshold before warning>
     pcp_stop_limit: <MB-treshold  to stop / restart the process>

InstanaPCP not connected

Monitoring issue type: clr_instana_pcp_not_connected

The host agent cannot connect to the InstanaPCP process, which prevents the retrieval and processing of tracing data and runtime metrics for .NET processes.

This is usually a transient failure, which will resolve itself quickly after the InstanaPCP has finished intializing. If this is not the case, reach out to the Instana support.