Monitoring Amazon Web Services (AWS)

The AWS Agent collects data from AWS APIs, such as CloudWatch, S3 and X-Ray, for which there is not possible to install a host agent or any other type of agent.

Platforms

Monitored Services

Note: Due to the nature of the AWS CloudWatch API, a delay in metrics retrieval is possible. The displayed values for the metrics are also delayed to ensure consistency. This delay depends on the actual AWS service and availability of data at that time, but usually is around 10 minutes.

Installation

Note: This documentation covers installing the AWS Agent, which allows you to collect data from CloudWatch and other datasources to monitor services managed by Amazon. If you want to monitor an Amazon Elastic Computing (EC2) Virtual Machine, a Kubernetes cluster running on AWS, either installed and managed by you or using Amazon Elastic Kubernetes Service, or an Amazon Elastic Container cluster, refer to the Install the Instana Host Agent documentation.

Note: Install only one AWS Agent per combination of AWS account and AWS region. Installing multiple AWS Agents for the same combination of AWS account and AWS region can incur additional costs from AWS, without added benefit in terms of quality of monitoring via Instana.

Installation on EC2

We recommend that you run the Instana AWS sensor on an Current Generation General Purpose machine running Linux. For example, m5.large instances are ideal.

  1. Sign in to Instana and click More -> Agents -> Instana agent installation -> AWS.
  2. From the Technology list, select Instana AWS Sensor.
  3. From the Run your AWS Agent on list, select Elastic Cloud Compute (EC2) (it's the default)

    Your agent key and location are pre-populated in the script to be used as User Data on EC2. The script looks like the following, except that all the necessary information is provided:

    #!/bin/bash
    
    curl -o setup_agent.sh https://setup.instana.io/agent
    chmod 700 ./setup_agent.sh
    sudo ./setup_agent.sh -y -a <your-agent-key> -m aws -t dynamic -e <location> -s
  4. Spin up a dedicated EC2 virtual machine and use the script as User Data; refer to the Running Commands on Your Linux Instance at Launch documentation for more information on how to user User Data on EC2.
  5. Copy the configuration.json file for the IAM role. This is the required IAM role for the EC2 virtual machine running the Instana AWS sensor, and enables the Instana agent to discover and monitor your AWS resources.
  6. For the IAM role to perform the AssumedRole action, edit the Trust Relationship. For example:

    {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
    }

    In case your EC2 instance is in one of China's regions, Trust Relationship should look like:

    {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "ec2.amazonaws.com.cn"
        },
        "Action": "sts:AssumeRole"
      }
    ]
    }

    Please refer to specifics of the China region monitoring for the additional necessary configuration.

Installation on Fargate on ECS

  1. Sign in to Instana and click More -> Agents -> Instana agent installation -> AWS.
  2. From the Technology list, select Instana AWS Sensor.
  3. From the Run your AWS Agent on list, select Elastic Container Service (ECS)

    The JSON of a Task Definition is pre-generated for you in the Task Definition JSON template, and you can use it with the Configure via JSON functionality of the Task Definition user interface.

  4. Assign to the new Task Definition an IAM role that contains at least the permissions listed under the IAM permissions JSON document.
  5. Create an ECS Service with one instance of the newly created Task Definition.

Configuration

Installation outside your AWS infrastructure

You can also dedicate any agent running outside of your AWS infrastructure. For this you will need to specify the following environment variables within the /opt/instana/agent/bin/setenv file:

  • The region you want to monitor:

    INSTANA_AWS_REGION_CONFIG=
  • The credentials to access AWS resources. Please note that these credentials should belong to a user which is allowed to access the resources already described in the Amazon Web Services IAM Configuration section.

    AWS_ACCESS_KEY_ID=
    AWS_SECRET_ACCESS_KEY=

China region specifics

By default, Instana agent will contact global STS endpoint from the EC2 instance to gather instance profile credentials. Since this approach will not work in CN regions, STS region endpoint needs to be exposed as an environment variable. This will tell Instana agent to use regional (CN) STS endpoints.

Once Instana agent has been installed, specify following environment variable within the <instana-agent-dir>/bin/setenv:

AWS_STS_REGIONAL_ENDPOINTS=regional

Enable Monitoring for Specific Services

For each service, you need to add the required permissions noted below on the individual service pages, linked to from the Monitored Services section. To exclude a service from monitoring, leave out the respective permission.

Alternatively Another way to exclude a service from monitoring is to add the appropriate enabled flag in <agent_install_dir>/etc/instana/configuration.yml as described on the individual service pages below.

Monitoring multiple AWS accounts

The AWS agent supports monitoring services from multiple AWS accounts in the same region. Currently, there are two approaches: using aws named profiles and using AWS Security Token Service(STS).

Note: When you configure the Instana agent to monitor multiple AWS accounts, only ONE of the two available approaches should be used.

AWS named profiles approach

In order to monitor multiple accounts, you need to define named profiles in the same way you would with the AWS CLI, or creating the file manually. The .aws/credentials file must be created within the home directory of the user which is running Instana agent, which is usually root. The AWS CLI uses credentials files ~/.aws/credentials like the following:

[default]
aws_access_key_id = accessKey1
aws_secret_access_key = secretAccessKey1

[profile2]
aws_access_key_id = accessKey2
aws_secret_access_key = secretAccessKey2

[profile3]
aws_access_key_id = accessKey3
aws_secret_access_key = secretAccessKey3

Every profile represents the mapping between AWS named profile and AWS access credentials. Upon installation of an AWS agent, a default AWS profile will be created; there is no need to add the default profile to ~/.aws/credentials file.

The additional profiles to be used by the AWS agent must be listed as follows in the agent configuration file /opt/instana/agent/etc/instana/configuration.yaml:

com.instana.plugin.aws:
  profile_names:
    - 'profile2'
    - 'profile3'

Note: The monitoring of multiple AWS accounts can also be specified for particular AWS services within their specific configuration. The list of profiles specified for a specific service will override the general configuration shown above.

Note: Be sure to follow recommendations when it comes to creation of credentials file. To avoid any security threats, you should either set more strict rules for EC2 instance or create a separate Instana AWS user and that way avoid adding root user security credentials to the ~/.aws/credentials file.

AWS STS approach

This approach uses AWS STS service API to obtain access credentials for all additional AWS accounts that should be monitored by the Instana agent. Once agent was installed and configured to monitor default account, following the steps described in the Installation section, you should provide IAM Role ARNs from all additional AWS accounts, in the following way:

com.instana.plugin.aws:
  role_arns:
    - 'arn:aws:iam::<account_2_id>:role/<role_2_name>'
    - 'arn:aws:iam::<account_3_id>:role/<role_3_name>'

Note: The monitoring of multiple AWS accounts can also be specified for particular AWS services within their specific configuration. The list of IAM roles specified for a specific service will override the general configuration shown above.

Each role, matching the specified ARN, should also comprise required IAM permissions necessary for Instana AWS monitoring. Thus, each role should allow default account to perform sts:AssumeRole by specifying the Trust relationship policy in the following way:

  1. If sts:AssumeRole is done from the AWS user context - when agent is installed outside your AWS infrastructure

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::<default_account_id>:user/<default_account_username>"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  2. If sts:AssumeRole is done from the IAM Role context - when agent is installed on EC2:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::<default_account_id>:role/<default_account_IAM_role_name>"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }

Similarly, default account should be also allowed to perform sts:AssumeRole to all additional accounts. This can be done by specifying additional IAM policies within the IAM Role used for Instana monitoring, in the following way:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        #Instana monitoring policy specifications
      },
      {
        "Action": [
            "sts:AssumeRole"
        ],
        "Resource": "arn:aws:iam::<account_1_id>:role/<role_1_name>",
        "Effect": "Allow"
      },
      {
        "Action": [
            "sts:AssumeRole"
        ],
        "Resource": "arn:aws:iam::<account_2_id>:role/<role_2_name>",
        "Effect": "Allow"
      }
    ]
  }

Filtering and Tagging

Note: To enable filtering, the IAM Permission tag:GetResources should be included in the IAM permissions granted to the AWS Agent.

After enabling the monitoring of a service, it is possible to filter which of its instances Instana will monitor based on AWS tags by modifying the agent configuration file /opt/instana/agent/etc/instana/configuration.yaml of the AWS Agent.

Available options in the configuration file are:

com.instana.plugin.aws:
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is discovered.
  include_tags: ...
  # Comma-separated list of tags in key:value format.
  # Any AWS resource containing at least one of the specified tags is omitted from discovery.
  exclude_tags: ...

Multiple tags can be defined, separated by a comma. Tags should be a key-value pair separated by :. When defining tags in both lists (include and exclude), the exclude list has higher priority. If there is no need for services filtering, the configuration should not be defined.

Note: The Instana agent automatically retrieves data from CloudWatch for all AWS resources that have no tags assigned.

Filtering can also be configured on the service level. In this case, the default configuration is overridden for the particular service. For more details about discovery filtering by specific service, see the specific service documentation.

Tag Poll Rate

To specify how often AWS sensor polls monitored AWS service to get associated tags, use the tagged-services-poll-rate configuration property. The default is 300 seconds. Sensor uses AWS service client to poll tags.

The poll rate configuration for the tagged resources is:

com.instana.plugin.aws:
  tagged-services-poll-rate: 60 #default 300

Polling Interval

The polling interval indicates how often AWS sensor calls the CloudWatch API. It is configurable as a cloudwatch_period in the configuration.yml file. The default value is 300 seconds. It's most common to see monitoring platforms use anywhere between 5 to 10 minutes by default.

Polling interval is configurable on two levels:

  • Agent level for all AWS services monitored by the agent:

    com.instana.plugin.aws:
      cloudwatch_period: 300
  • Per AWS service:

    com.instana.plugin.aws.rds:
      cloudwatch_period: 300

Configuration for single service overrides agent level configuration.

CloudWatch Costs

In order to provide insights into your AWS services, monitoring platforms like Instana must use the CloudWatch API. This API is provided by AWS with consumption-based pricing, and the purpose of this documentation is to provide transparency into how Instana consumes the CloudWatch API in order for users to understand the impact on their AWS bill.

CloudWatch costs are affected by:

  • Number of monitored instances of AWS services
  • Number of collected metrics per AWS sensor
  • Polling interval (configurable)

Each AWS sensor calls GetMetricData CloudWatch API request for metrics collection, and if it fails GetMetricStatistics is called as a fallback. Both endpoints incur the same costs per metrics (for clarity: our fallback usage of GetMetricStatistics does not incur additional costs). You can find details for both on the AWS CloudWatch pricing page.

The following table contains roughly estimated daily costs for different AWS sensors when polling interval is 5 minutes and Amazon charge is $0.01 per 1000 CloudWatch metrics:

AWS Sensor Number of Metrics Daily cost per instance
DynamoDB 71 $0.2045
Elasticsearch 12 $0.0346
Lambda 17 $0.0490
MQ 22 $0.0634
RDS 18 $0.0518
SQS 8 $0.0230
S3 13 $0.0374

Please note that we do not use CloudWatch for EC2 and for polling tags, so there are no CloudWatch costs incurred there.

CloudWatch costs can be reduced by decreasing polling interval. The downside to increasing the polling interval is that you will have reduced granularity metrics from CloudWatch (each request provides a single metric value, so the different between a polling interval of 60s and 300s is 5 metric values or 1 metric value for a 300s period.)

It is up to you to determine whether the default polling interval is right for your business or whether to customize it, to strike the balance between cost and quality of insights.