Monitoring AWS Lambda
TABLE OF CONTENTS
- Why Use Instana's Native Tracing
- Lambda Extension
Note: Learn about the other supported AWS services with our AWS documentation.
Note: The native tracing of AWS Lambda described in this documentation page replaces the (deprecated) X-Ray integration.
Native tracing of AWS Lambda functions provides large advantages over X-Ray based tracing:
- End-to-end visibility: The execution Lambda functions now seamlessly correlates with tracing of other components monitored by Instana. Calls from Hosts processes to Serverless and back are now linked, correlated and analyzed as such with no additional steps required.
- Ease of Analysis: Instana can now provide rich performance analysis between specific versions of the same AWS Lambda function.
- Reduces Cost: Completely eliminates the need of pulling costly X-Ray data from AWS.
- Automatic and deep trace visibility: trace every invocation of your Lambda functions rather than manually instrument them with X-Ray.
The setup of the AWS Sensor for Lambda monitoring collects information about metrics and versions for all your Lambda functions, irrespective of the runtime they use.
The setup is made of two steps:
- Configure the AWS Sensor for Lambda monitoring to collect information like metrics and versions that Instana combines with the data coming from the tracer
- Install the Instana tracer in your Lambda function
Note: This section assumes you have already performed the setup described in the AWS Agent documentation for the AWS account and region in which you deploy the AWS Lambda functions.
Note: The setup steps described in this section are required by several AWS Lambda-related functionalities, including but not limited to, Dynamic Focus Queries using the
entity.aws.lambda.version operator, Unbounded Analytics tags like
faas.functionversion and the dashboards dedicated to AWS Lambda functions and function versions.
This section describes additional configurations that are required to collect metrics and version data for the AWS Lambda functions traced by Instana. The configurations below need to be applied to the AWS Agent.
The following additional IAM permissions have to be granted to the AWS Agent:
For information on granting IAM permissions to the AWS Agent, refer to the AWS Agent Installation documentation.
Note: Enabling this feature will require the
lambda:ListVersionsByFunction IAM permission to be granted tio the AWS Agent.
If the required permission is missing, Instana will fall back to aggregating the metrics for all versions.
By default, Instana will monitor the five most recent versions of each Lambda function individually. The number of versions for which Instana fetches data is configurable:
com.instana.plugin.aws.lambda: number_of_versions: 3 # default 5
The maximum allowed for
number_of_versions is 20.
Higher values will be capped to 20.
Monitoring individual Lambda versions can be disabled, Instana will then fall back to aggregating the metrics for all versions.
Note that in this case the aggregating metrics will be attached to the $LATEST version for the Lambda function.
Please make sure to restart the agent after making this configuration change.
To entirely disable monitoring of Lambda instances use the following configuration:
com.instana.plugin.aws.lambda: enabled: false
Use the following configuration to disable only the retrieval of versions data:
com.instana.plugin.aws.lambda: fetch_versions: false
Metrics for Lambda are pulled every 5 minutes, this can be changed via agent configuration in
com.instana.plugin.aws.lambda: cloudwatch_period: 300
Users are able to specify how often sensors will poll the AWS tagged resources using the
tagged-services-poll-rate configuration property (default 300 seconds).
To define how often sensors will poll the tagged resources use following configuration:
com.instana.plugin.aws: tagged-services-poll-rate: 60 #default 300
Multiple tags can be defined, separated by a comma.
Tags should be provided as a key-value pair separated by the
In order to make configuration easier, it is possible to define which tags you want to include in discovery or exclude from discovery.
In case of defining tag in both lists (include and exclude), exclude list has higher priority.
If there is no need for services filtering, the configuration should not be defined.
It’s not mandatory to define all values in order to enable filtering.
To include services by tags into discovery use following configuration:
com.instana.plugin.aws.lambda: include_tags: # Comma separated list of tags in key:value format (e.g. env:prod,env:staging)
To exclude services by tags from discovery use following configuration:
com.instana.plugin.aws.lambda: exclude_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)
AWS services without tags will be monitored by default but can be excluded by setting the
include_untagged field to
com.instana.plugin.aws.lambda: include_untagged: false # True value by default
Refer to the Monitoring multiple AWS accounts documentation to set up monitoring of multiple AWS accounts with one AWS agent in the same region.
To override which profiles should be used to monitor Lambda, use the following configuration:
com.instana.plugin.aws.lambda: profile_names: - 'profile2' - 'profile3'
Note: Defining profiles on service level will override the global AWS profiles configuration.
To override which IAM Roles should be used to monitor Lambda, use the following configuration:
com.instana.plugin.aws.lambda: role_arns: - 'arn:aws:iam::<account_1_id>:role/<role_1_name>' - 'arn:aws:iam::<account_2_id>:role/<role_2_name>'
Note: Defining IAM roles on service level will override the global AWS IAM roles configuration.
Refer to the installation instructions for the specific runtimes:
|Lambda Function Details||Description|
|Name||The name of the function.|
|ARN||The unqualified Amazon Resource Name (ARN) assigned to the function.|
|Description||The user-provided description.|
|Runtime||The runtime environment for the Lambda function.|
|Handler||The function Lambda calls to begin executing your function.|
|Timeout||The function execution time at which Lambda should terminate the function.|
|Memory Size||The memory size, in MB, you configured for the function.|
|Last Modified||The time stamp of the last time you updated the function.|
|AWS Region||The location of function.|
|Layers||AWS Lambda layers that are configured for this function.|
|Environment Variables||The list of environment variables configured for this function (subject to secrets redaction).|
|Invocations||Measures the number of times a function is invoked in response to an event or invocation API call. This includes successful and failed invocations, but does not include throttled attempts. This equals the billed requests for the function.|
|Errors||Measures the number of invocations that failed due to errors in the function (response code 4XX). This includes:
|Dead Letter Errors||Incremented when Lambda is unable to write the failed event payload to your configured Dead Letter Queues. This could be due to the following:
|Throttles||Measures the number of Lambda function invocation attempts that were throttled due to invocation rates exceeding the customer’s concurrent limits (error code 429).|
||Measures the elapsed wall clock time from when the function code starts executing as a result of an invocation to when it stops executing. The maximum data point value possible is the function timeout configuration. The billed duration will be rounded up to the nearest 100 millisecond.|
||Emitted for stream-based invocations only (functions triggered by an Amazon DynamoDB stream or Kinesis stream). Measures the age of the last record for each batch of records processed. Age is the difference between the time Lambda received the batch, and the time the last record in the batch was written to the stream.|
|Post Runtime Extensions Duration
||The cumulative amount of time that the AWS Lambda runtime spends running code for extensions after the function code has completed.|
||Emitted as an aggregate metric for all functions in the account, and for functions that have a custom concurrency limit specified. Measures the sum of concurrent executions for a given function at a given point in time.|
|Unreserved Concurrent Executions||Emitted as an aggregate metric for all functions in the account only. Represents the sum of the concurrency of the functions that do not have a custom concurrency limit specified.|
Instana offers detailed capturing of HTTP attributes for Lambda executions that are triggerd by a trigger of type "API Gateway" or "Application Load Balancer". This includes extracting the URL, path templates, the status code, query parameters etc. The standard enpoint extraction uses this attributes, too.
However, for API Gateway calls, HTTP attributes can only be captured if the option "Use Lambda Proxy integration" is used when defining the API Gateway's methods. After the creation of a an API Gateway's methods, this can be checked by inspecting the "Integration Request" box on the API Gateway configuration page. If it says "Type: LAMBDA PROXY", it uses the Lambda Proxy integration.
This constraint does not apply to "Application Load Balancer" triggers.
If a Lambda function is used as a trigger, all event source mappings will be displayed in the sidebar with links pointing to the attached AWS service. For more informations, see Invoking Lambda functions. Native Lambda tracing will capture additional meta data for the following triggers:
|Trigger||Meta Data||Supported in Lambda Runtimes|
|API Gateway||HTTP method, URL, Path Template, Query Parameters, Headers2||Java, Node.js, Python, Go|
|Application Load Balancer||HTTP method, URL, Query Parameters, Headers2||Java, Node.js, Python, Go|
|Cloudwatch Event||Event Resources||Java, Node.js, Python, Go|
|Cloudwatch Logs||Log Groups, Log Stream, Log Events||Java, Node.js, Python, Go|
|S3||S3 Event Name, Bucket Name, Object Key||Java, Node.js, Python, Go|
|SQS||SQS Queue ARN||Java, Node.js, Python, Go|
You can configure Smart Alerts for excessive cold starts or timeouts on your Lambda functions. To do that, go to an application perspective that contains the Lambda functions you want to alert on and add a new Smart Alert:
- Click the "Add Smart Alert" button,
- switch to "Advanced Mode",
- add a filter for
Cloud Function As A Service > Cold Start is trueor
Cloud Function As A Service > Suspected Timeout is true,
- select the
- select a "threshold" value that fits your use case (use 1 to include every occurence or a higher value to only have the rule violated on a particular number of cold starts/timeouts),
- finally, the "Time Threshold" settings can be used to fine tune the alerting behaviour.
The following is a list of optional environment variables that are generally supported in native Lambda tracing:
||Timeout for the HTTP requests reporting data to the Instana back end|
||Set this to
||Semicolon-separated list of HTTP headers to be captured|
||Custom service name|
||The log level for the Instana package, possible values are
||Set this to any value to set the log level to
||Set this to
To capture HTTP headers (for trigger types API gateway with Lambda proxy integration or application load balancer), you need to provide the environment variable
INSTANA_EXTRA_HTTP_HEADERS with a semicolon-separated list of headers to capture.
The following layers ship a Lambda Extension:
- NodeJS - Since layer version 71.
This extension can improve performance of monitored Lambda functions by shifting the process of sending data to the Instana backend out of band. To take advantage of these performance improvements, upgrade the Instana layer to a version that includes the extension.
Under the hood, the extension works in conjunction with the language library to buffer and proxy all of the requests to the Instana backend. Under normal operation, the extension does not extend the lifetime of the lambda. We only extend the lifetime during:
- A shutdown request to ensure we flush all data held in the buffer to the backend.
- A situation were we think the Lambda function has crashed and stopped sending spans. We wait for three seconds to see if the root span is sent before waiting for the next invocation.
If you would like to disable the use of the extension set
TRUE in your function's environment.
Instana's Lambda Monitoring will increase the size of the deployed artifact by the size of language's given layer or library. To get the size of a given layer, use the following shell commands:
$ aws lambda get-layer-version --layer-name <layer_arn> --version-number <target_version> | jq .Content.Location | xargs -L 1 curl --output layer.zip $ unzip -l layer.zip
The performance impact of monitoring a Lambda function with Instana is primarily driven by the amount of time it takes for the Lambda function to communicate with the Instana backend. At the end of each execution, the Instana provided library makes a blocking call to either our extension or your Instana tenant unit to report the result of the execution.
For functions using the extension, this process will take place in the background and allow the monitored function to return its response as soon as the monitored code is ready.
For functions not using the extension, this process will block until we have sent the data to your tenant unit.
For more context on this, please refer to our extension blog post.