AWS Lambda

Note: Learn about the other supported AWS services with our AWS documentation.

Note: The native tracing of AWS Lambda described in this documentation page replaces the (deprecated) X-Ray integration.

Why Use Instana's Native Tracing

Native tracing of AWS Lambda functions provides large advantages over X-Ray based tracing:

  • End-to-end visibility: The execution Lambda functions now seamlessly correlates with tracing of other components monitored by Instana. Calls from Hosts processes to Serverless and back are now linked, correlated and analyzed as such with no additional steps required.
  • Ease of Analysis: Instana can now provide rich performance analysis between specific versions of the same AWS Lambda function.
  • Reduces Cost: Completely eliminates the need of pulling costly X-Ray data from AWS.
  • Automatic and deep trace visibility: trace every invocation of your Lambda functions rather than manually instrument them with X-Ray.

Supported Runtimes and Versions

Runtime Versions
Node.js 8.10 (see note), 10.x, 12.x
Python 2.7, 3.6, 3.7, 3.8

Note: For Node.js 8.1.0, use the approach outlined in the Manual Wrapping documentation.

Note: Native tracing of AWS Lambda functions is supported in self-hosted Instana from version v170.

Collected Lambda Configurations, Metrics, Versions and Triggers

Tracked Configuration

Lambda Function Details Description
Name The name of the function.
ARN The unqualified Amazon Resource Name (ARN) assigned to the function.
Description The user-provided description.
Runtime The runtime environment for the Lambda function.
Handler The function Lambda calls to begin executing your function.
Timeout The function execution time at which Lambda should terminate the function.
Memory Size The memory size, in MB, you configured for the function.
Last Modified The time stamp of the last time you updated the function.
AWS Region The location of function.

Metrics

Function Metrics Description
Invocations Measures the number of times a function is invoked in response to an event or invocation API call. This includes successful and failed invocations, but does not include throttled attempts. This equals the billed requests for the function.
Errors Measures the number of invocations that failed due to errors in the function (response code 4XX). This includes:
  • Handled exceptions (for example, context.fail(error))
  • Unhandled exceptions causing the code to exit
  • Out of memory exceptions
  • Timeouts
  • Permissions errors
This does not include invocations that fail due to invocation rates exceeding default concurrent limits (error code 429) or failures due to internal service errors (error code 500).
Dead Letter Errors Incremented when Lambda is unable to write the failed event payload to your configured Dead Letter Queues. This could be due to the following:
  • Permissions errors
  • Throttles from downstream services
  • Misconfigured resources
  • Timeouts
Throttles Measures the number of Lambda function invocation attempts that were throttled due to invocation rates exceeding the customer’s concurrent limits (error code 429).
Durations
  • Average
  • Minimum
  • Maximum
  • Sum
Measures the elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing. The maximum data point value possible is the function timeout configuration. The billed duration will be rounded up to the nearest 100 millisecond.
Iterator Age
  • Average
  • Minimum
  • Maximum
  • Sum
Emitted for stream-based invocations only (functions triggered by an Amazon DynamoDB stream or Kinesis stream). Measures the age of the last record for each batch of records processed. Age is the difference between the time Lambda received the batch, and the time the last record in the batch was written to the stream.
Global Metrics Description
Concurrent Executions
  • Average
  • Minimum
  • Maximum
  • Sum
Emitted as an aggregate metric for all functions in the account, and for functions that have a custom concurrency limit specified. Measures the sum of concurrent executions for a given function at a given point in time.
Unreserved Concurrent Executions Emitted as an aggregate metric for all functions in the account only. Represents the sum of the concurrency of the functions that do not have a custom concurrency limit specified.

HTTP Call Attributes, API Gateway & Lambda Proxy Integration

Instana offers detailed capturing of HTTP attributes for Lambda executions that are triggerd by a trigger of type "API Gateway" or "Application Load Balancer". This includes extracting the URL, path templates, the status code, query parameters etc. The standard enpoint extraction uses this attributes, too.

However, for API Gateway calls, HTTP attributes can only be captured if the option "Use Lambda Proxy integration" is used when defining the API Gateway's methods. After the creation of a an API Gateway's methods, this can be checked by inspecting the "Integration Request" box on the API Gateway configuration page. If it says "Type: LAMBDA PROXY", it uses the Lambda Proxy integration.

This constraint does not apply to "Application Load Balancer" triggers.

Triggers

If a Lambda function is used as a trigger, all event source mappings will be displayed in the sidebar with links pointing to the attached AWS service. For more informations, see Invoking Lambda functions. Native Lambda tracing will capture additional meta data for the following triggers:

Trigger Meta Data Supported in Lambda Runtimes
API Gateway HTTP method, URL, Path Template, Query Parameters, Headers2 Node.js, Python
Application Load Balancer HTTP method, URL, Query Parameters, Headers2 Node.js, Python
Cloudwatch Event Event Resources Node.js, Python
Cloudwatch Logs Log Groups, Log Stream, Log Events Node.js, Python
S3 S3 Event Name, Bucket Name, Object Key Node.js, Python
SQS SQS Queue ARN Node.js, Python

Setup

The setup is made of two steps:

  1. installing the Instana tracer in your Lambda function
  2. configuring the AWS Sensor for Lambda monitoring to collect additional information like metrics and versions that Instana uses to enrich the data coming from the tracer

Install Tracing in the AWS Lambda Function

Instana AutoTrace

The preferred way to enable Instana tracing for your AWS Lambda functions is to use the Instana Lambda layer and a custom function handler. This approach requires no modification of the Lambda function code and is done purely through configuration. It is also suited to be automated or included in a Lambda deployment pipeline.

The following instructions are valid for both Node.js and Python AWS Lambda functions.

  1. Add the Instana Lambda layer to your function.

    1. In the configuration page for your Lambda function, click on the Layers box and then on Add a layer. layer
    2. In the popup that opens, select “Provide a layer version ARN” Copy and paste the ARN for your AWS region into the ARN text input field. layer selection
    3. The list of Instana Lambda Layer ARNS are available per runtime. Select the appropriate layer for your region and runtime: Node.js and Python
  2. Configure the Lambda Handler. If you are using the default Lambda handler, then simply replace that value with a custom value.

    • For Node.js, set the handler to instana-aws-lambda-auto-wrap.handler. The Instana Lambda layer will automatically trigger the default Node.js runtime handler: index.handler.
    • For Python, set the handler to instana.lambda_handler. The Instana Lambda layer will automatically trigger the default Python runtime handler: lambda_function.lambda_handler.

    If you are using a custom handler value (not the default runtime handler), specify that value in an environment variable LAMBDA_HANDLER to notify the Instana Lambda Layer. new handler

  3. Configure Environment Variables. Add the following environment variables:

    You can also obtain the correct values for these environment variables by going to your Instana installation, click on "... More" -> "Agents" -> "Installing Instana Agents" -> Platform: "AWS" -> Technology: "AWS Lambda".

  4. Save the Lambda function definition. save

All the steps outlined above can be done either via the AWS web console or any of the usual AWS management tools, such as:

Here is an example aws CLI command that might serve as a starting point if you want to automate the Instana integration of your AWS Lambdas:

# Do not copy and paste this verbatim!
# It will overwrite any previously defined collection of layers and
# environment variables.
aws --region $YOUR_REGION lambda update-function-configuration \
  --function-name $YOUR_LAMBDA_FUNCTION_NAME \
  --layers $INSTANA_LAYER_ARN \
  --handler instana-aws-lambda-auto-wrap.handler
  --environment ""Variables={LAMBDA_HANLDER=yourActual.handler,INSTANA_ENDPOINT_URL=...,INSTANA_AGENT_KEY=...}""

See also the Node.js and Python Lambda runtime pages for more options and details specific to those runtimes.

Manual Wrapping

Note: This section applies only to situations in which the Instana AutoTrace approach is not supported or not desireable. If you have executed the steps defined in the Instana AutoTrace section, you should skip this section.

For the Node.js 8 runtime, the Instana AutoTrace approach is not supported. Refer to the additional AWS Lambda Node.js documentation

AWS Agent Setup for AWS Lambda

Note: This section assumes you have already performed the setup described in the AWS Agent documentation for the AWS account and region in which you deploy the AWS Lambda functions.

Note: The setup steps described in this section are required by several AWS Lambda-related functionalities, including but not limited to, Dynamic Focus Queries using the entity.aws.lambda.version operator, Unbounded Analytics tags like faas.functionversion and the dashboards dedicated to AWS Lambda functions and function versions.

This section describes additional configurations that are required to collect metrics and version data for the AWS Lambda functions traced by Instana. The configurations below need to be applied to the AWS Agent.

Additional IAM permissions

The following additional IAM permissions have to be granted to the AWS Agent:

  • cloudwatch:GetMetricStatistics
  • cloudwatch:GetMetricData
  • lambda:ListTags
  • lambda:ListFunctions
  • lambda:ListVersionsByFunction
  • lambda:GetFunctionConfiguration

For information on granting IAM permissions to the AWS Agent, refer to the AWS Agent Installation documentation.

Monitoring Lambda Versions

Note: Enabling this feature will require the lambda:ListVersionsByFunction IAM permission to be granted tio the AWS Agent. If the required permission is missing, Instana will fall back to aggregating the metrics for all versions.

By default, Instana will monitor the five most recent versions of each Lambda function individually. The number of versions for which Instana fetches data is configurable:

com.instana.plugin.aws.lambda:
  number_of_versions: 3 # default 5

The maximum allowed for number_of_versions is 20. Higher values will be capped to 20. Monitoring individual Lambda versions can be disabled, Instana will then fall back to aggregating the metrics for all versions. Note that in this case the aggregating metrics will be attached to the $LATEST version for the Lambda function.

Please make sure to restart the agent after making this configuration change.

Disabling Retrieval of Lambda Versions and Metrics

To entirely disable monitoring of Lambda instances use the following configuration:

com.instana.plugin.aws.lambda:
  enabled: false

Use the following configuration to disable only the retrieval of versions data:

com.instana.plugin.aws.lambda:
  fetch_versions: false

Poll Rate

Metrics for Lambda are pulled every 5 minutes, this can be changed via agent configuration in <agent_install_dir>/etc/instana/configuration.yml:

com.instana.plugin.aws.lambda:
  cloudwatch_period: 300

Users are able to specify how often sensors will poll the AWS tagged resources using the tagged-services-poll-rate configuration property (default 300 seconds).

To define how often sensors will poll the tagged resources use following configuration:

com.instana.plugin.aws:
  tagged-services-poll-rate: 60 #default 300

Filtering

Multiple tags can be defined, separated by a comma. Tags should be provided as a key-value pair separated by the : character. In order to make configuration easier, it is possible to define which tags you want to include in discovery or exclude from discovery. In case of defining tag in both lists (include and exclude), exclude list has higher priority. If there is no need for services filtering, the configuration should not be defined. It’s not mandatory to define all values in order to enable filtering.

To include services by tags into discovery use following configuration:

com.instana.plugin.aws.lambda:
  include_tags: # Comma separated list of tags in key:value format (e.g. env:prod,env:staging)

To exclude services by tags from discovery use following configuration:

com.instana.plugin.aws.lambda:
  exclude_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)

Monitoring multiple AWS accounts

Refer to the Monitoring multiple AWS accounts documentation to set up monitoring of multiple AWS accounts with one AWS agent in the same region.

To override which profiles should be used to monitor Lambda, use the following configuration:

com.instana.plugin.aws.lambda:
  profile_names:
    - 'profile2'
    - 'profile3'

Note: Defining profiles on service level will override global AWS configuration.

Additional Configuration for the Instana Lambda Tracers

Environment Variables

The following is a list of optional environment variables that are generally supported in native Lambda tracing:

Environment Variable Meaning
INSTANA_TIMEOUT Timeout for the HTTP requests reporting data to the Instana back end
INSTANA_ENDPOINT_PROXY Set this to http://my-proxy.tld or http://user:[email protected] to route the reporting of data to the Instana back end over an HTTP(S) proxy (supported in Node.js, not in Python)
INSTANA_EXTRA_HTTP_HEADERS Semicolon-separated list of HTTP headers to be captured
INSTANA_SERVICE_NAME Custom service name
INSTANA_LOG_LEVEL The log level for the Instana package, possible values are debug, info, warn, and error (default: info)
INSTANA_DEBUG Set this to any value to set the log level to debug.
INSTANA_DISABLE_CA_CHECK Set this to true to disable verifying the server certificate against the list of CAs baked into Lambda runtime when connecting to the Instana back end. Enabling this makes your lambda vulnerable to MITM attacks for this connection. This setting should never be used, unless you use Instana On-Premises and are unable to operate the Instana back end with a certificate with a known root CA. (Available since @instana/[email protected]/layer version 25.)

Capturing HTTP Headers

To capture HTTP headers (for trigger types API gateway with Lambda proxy integration or application load balancer), you need to provide the environment variable INSTANA_EXTRA_HTTP_HEADERS with a semicolon-separated list of headers to capture.