Monitoring IBM App Connect Enterprise

Supported versions

Confirmed support for metrics and configuration data for versions: IBM App Connect Enterprise v11, IBM Integration Bus v10.

Configuration

Prerequisites

  • If the server is IBM Integration Bus v10, use following commands to check the statistics state. Turn on or turn off statistics collection when needed. Note that metrics return only when the statistics gathering and resource statistics collection are active.

    • Check resource statistics state and turn on if needed. Following is an example:
      See more: mqsireportresourcestats & mqsichangeresourcestats

      # Check the collection state of resource statistics for specified server in specified integration node. 
      mqsireportresourcestats integrationNode -e integrationServer
      # If the state is inactive, turn on the resource statistics collection for specified integration server in specified integration node.    mqsichangeresourcestats integratorNode -e integrationServer -c active
    • Check statistics state and turn on if needed. Following is an example:
      See more: mqsireportflowstats & mqsichangeflowstats

      # Check the statistics gathering state for all message flows in sepcified integration server for specified integrator node. "-a" means archive type
      mqsireportflowstats integratorNode  -a -e integrationServer  -j
      # If the state is inactive, turn on the collection of archive statistics data for all message flows in specified integration server for specified integration node, and emit the data in JSON format.
      mqsichangeflowstats integrationNode -a -e integrationServer -j -c active -o json  

      Note: You can request two types of data collection, snapshot and archive(see more). It is recommended to use archive type for IBM ACE Sensor. Archive data is collected for an interval that you have set for the integration node and the default interval is 60 minutes. You can specify the interval from 1 minute to 43200 minutes with mqsichangebroker command. Here is an example to set interval to 5 minutes:

      mqsistop integrationNode
      mqsichangebroker integrationNode  -v 5 
      mqsistart integrationNode
  • Set up channel if IBM MQ is used For ACE/IIB traditional environment, if the integration node/server is associated with a queue manager, in order to get the performance metrics data for integration server, message flow, and flow node, you must set up a server connection channel and the channel must be accessible by the username/password that is specified in configuration.yaml file. Besides the channel name, username, and password, you also need to specify the queue manager name and listening port in configuration.yaml. If you have any issue about channel configuration, refer to this doc

Configuation for traditional ACE

For traditional ACE, Instana supports monitoring of both remote and local IBM ACE instances. You need to configure the following fields in the agent configuration <agent_install_dir>/etc/instana/configuration.yaml:

To use remote monitoring, set the ip address of target ACE server as value of host. Or else, comment out the host attribute and local monitoring is used by default.

# IBM ACE
com.instana.plugin.ace:
  enabled: true
  poll_rate: 60
  NodesOrServers: # Multiple Integration node instances or multiple standalone Integration Servers can be specified
    BK1:  # specify the Integration node name (required)
          # Following parameters used to acess ACE server's rest api service to retrieve such as status data, configuration data etc
      host: ''                      # ACE server's host, should be defined for remote monitoring only (optional)
      restApiPort: '4414'           # ACE server's rest api port (required)
      aceUsername: 'viewer'         # ACE rest api service username if security enabled (optional)
      acePassword: 'viewerPassword' # ACE rest api service password if security enabled (optional)

      # Following parameters used to access ACE server's pub/sub service, to retrieve performance statistics data
      # For example Message Flow statistics data, Flow node statistics dataparameter
      mqport: '1414'                # Sets the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM1'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security enabled (optional for IBM MQ)
      keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring, separate multiple names by comma (optional)

    IS1:  # specify the standalone integration server name (required)
          # Following parameters used to acess ACE server's rest api service to retrieve such as status data, configuration data etc
      host: ''                      # ACE server's host, should be defined for remote monitoring only (optional)
      restApiPort: '7600'           # ACE server's rest api port (required)
      aceUsername: 'viewer'         # ACE rest api service username if security enabled (optional)
      acePassword: 'viewerPassword' # ACE rest api service password if security enabled (optional)

      # Following parameters used to access ACE server's pub/sub service, to retrieve performance statistics data
      # For example Message Flow statistics data, Flow node statistics data
      mqport: '1414'                # Sets the port for the MQTT server or remote administration IBM MQ channel port (required)
      queuemanagerName: 'QM2'       # Queue Manager name (required for IBM MQ)
      channel: 'SYSTEM.DEF.SVRCONN' # Remote administration channel (required for IBM MQ)
      mqUsername: 'mqUser'          # MQ channel authentication's username if security enabled (optional for IBM MQ)
      mqPassword: 'mqPassword'      # MQ channel authentication's password if security enabled (optional for IBM MQ)
      keystore: '/tmp/server_keystore.jks' # Keystore path for HTTPS connection (required only when HTTPS is enabled)
      keystorePassword: 'password'  # Keystore password for HTTPS connection (required only when HTTPS is enabled)
      excludedServers: '<INTEGRATION_SERVER_NAMES>' # Integration server name to be excluded from monitoring, separate multiple names by comma (optional)

Configuation for cloud native ACE sensor

Configuration for ACE sensor in the cloud native environment is auto-discovered.

Metrics collection

Configuration data

  • Default QueueManager Name
  • Product Name
  • Version
  • Build Level
  • Platform Name
  • Platform Architecture
  • Platform Version
  • Application Name

Performance metrics

Integration server

Metric Description
Initial Heap Memory The initial amount of memory that the JVM requests from the operating system for memory management during startup.
Used Heap Memory The amount of memory that is currently in use.
Committed Heap Memory The amount of memory that is allocated to the JVM by the operating system.
Max Heap Memory The maximum amount of memory that can be used for memory management.
Initial Non-heap Memory The initial amount of memory that the JVM requests from the operating system for memory management during startup.
Used Non-heap Memory The amount of memory that is currently in use.
Committed Non-heap Memory The amount of memory that is allocated to the JVM by the operating system.
Max Non-heap Memory The maximum amount of memory that can be used for memory management.
Cumulative GC Time The time when the data is collected, in seconds.
Cumulative Number of GC The total number of garbage collections that have occurred for this instance of the JVM.

Message flow

Metric Data type Description
Total Elapsed Time Numeric Total elapsed time that a message flow in microseconds spent processing input messages.
Maximum Elapsed Time Numeric Maximum elapsed time that a message flow in microseconds spent processing an input message.
Minimum Elapsed Time Numeric Minimum elapsed time that a message flow in microseconds spent processing an input message.
Total CPU Time Numeric Total CPU time that a message flow in microseconds spent processing an input message.
Max CPU Time Numeric Maximum CPU time that a message flow in microseconds spent processing an input message.
Min CPU Time Numeric Minimum CPU time that a message flow in microseconds spent processing an input message.
CPU Time Waiting For Input Messages Numeric Total CPU time that a message flow in microseconds spent waiting for input messages.
Elapsed Time Waiting For Input Messages Numeric Total elapsed time in microseconds spent waiting for input messages.
Num of All Input Messages Numeric Total number of input messages
Total Size of Input Messages Numeric Total size of input messages in bytes.
Maximum Size of Input Messages Numeric Maximum size of input messages in bytes.
Minimum Size of Input Messages Numeric Minimum size of input messages in bytes.
Number of Threads in Pool Numeric Number of threads in pool
Number of Times That Max Threads Reached Numeric Number of times the maximum number of threads is reached.
Number of MQGET Errors Numeric Number of MQGET errors for MQInput nodes or Web Services errors for HTTPInput nodes.
Number of Messages with Errors Numeric Number of messages that contain errors.
Number of Errors When Processing Messages Numeric Number of errors that occur when processing a message.
Number of Transaction Timeouts Numeric Number of transaction timeouts that occur when processing a message (for AggregateReply nodes only).
Number of Transaction Commits Numeric Number of transaction commits that occur when processing a message.
Number of Transaction Backouts Numeric Number of transaction backouts that occur when processing a message.

Message flow node

Metric Data type Description
Total Elapsed Time Numeric Total elapsed time in microseconds spent processing input messages.
Max Elapsed Time Numeric Maximum elapsed time in microseconds spent processing input messages.
Min Elapsed Time Numeric Minimum elapsed time in microseconds spent processing input messages.
Total CPU Time Numeric Total cpu time in microseconds spent processing input messages.
Max CPU Time Numeric Maximum CPU time in microseconds spent processing an input message.
Min CPU Time Numeric Minimum CPU time in microseconds spent processing an input message.
Number of Invocations Numeric Total number of messages processed by this flow node.
Number of Input-type Terminals Numeric Number of input-type terminals.
Number of Output-type Terminals Numeric Number of output-type terminals.

Known Limitations

  • If HTTPS is enabled for REST API interface, you need to specify keystore and keystorePassword parameters in configurtion.yaml. For keystore type, only JKS or P12 is supported.
  • If integration servers under different integration nodes have the same name, only one integration server and its statistics show up in Infrastructure page.

    Health Signatures

For each sensor, there is a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents depending on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any given entity.

For information about built-in events for the IBM ACE sensor, see the Built-in events reference.

Troubleshooting

  • Test the MQ channel connectivity
    Set up the MQ Explorer, and then test the channel connectivity in the MQ Explorer with the credential that is provided in the configuration.yaml file.
  • You get an error code 2058 when connecting to queue manager
    If you cannot get metrics from UI but get an error code 2058 from agent log, first make sure all MQ related info is correct in configuration file, and then run command ps -ef | grep runmqlsr | grep '<QUEUEMGR NAME>' | grep '<LISTENER_PORT>' to list the detailed info. If the listener has specified ip with -i, you need to remove the limitation or set up a new listener for ACE.
    Here is an example output with corrrect setting:

    [email protected]:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1414
    mqm       9956  9872  0 Aug30 ?        00:00:05 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1414

    Here is an example output with wrong setting:

    [email protected]:~# ps -ef | grep "runmqlsr" | grep QM1 | grep  1417
    mqm       9957 9872  0 Aug30 ?        00:00:06 /opt/mqm/bin/runmqlsr -r -m QM1 -t TCP -p 1417 -i 10.17.59.113 
  • How to get the REST API port that current integration node is using?
    Run following command to determine which port the API interface is using:

    # If HTTP protocal is used, get the port with this command
    mqsireportproperties integrationNode -b webadmin -o HTTPConnector -a | grep port
    # If HTTPS protocal is used, get the port with this command
    mqsireportproperties integrationNode  -b webadmin -o HTTPSConnector -a | grep port
  • How to collect the payload in JSON format for further investigation?
    If you want to get the payload in JSON format, you can use the following command to collect:

    # CMD for IIBv10
    curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv1?depth=6" >> payload.json
    # CMD for ACEv11
    curl -u <USERNAME> --header "Accept: application/json" "http://<ACE_HOST>:<INTEGTATION_NODE_API_PORT>/apiv2?depth=6" >> payload.json

Tracing

This page describes how to install and enable IBM ACE tracing on your IBM ACE hosts.

Install IBM ACE Tracing user exit

Platforms and prerequisites

Note: IBM ACE Tracing support in Instana is in technology preview. For more details, refer to the IBM technology preview.

The IBM ACE tracing user exit supports IBM App Connect Enterprise v11.0.0.8 or later.

The IBM ACE tracing user exit supports Linux x86_64 only, and will support more platforms in the future.

The IBM ACE tracing user exit supports HTTP request and IBM MQ request only.

The IBM ACE tracing user exit supports message flows with input node as the entrypoint only.

IBM ACE must be installed and configured before installing and configuring this component.

Downloading IBM ACE Tracing user exit

  1. Download the package from IBM Fix Central to your IBM ACE host. The package includes the following three files:

    • ACEOpenTracingUserExit.lel, the IBM ACE user exit, intercepting the HTTP request and IBM MQ request and invoking the wrapped go Jaeger client library to create spans.
    • tracelibrary.so, the wrapped go Jaeger client, provides functions to manage the lifecycle of spans and sends spans to the target tracing system.
    • acetracingexit.conf, the configuration file to specify the log level and information about connecting to the host agent.
  2. Extract tar file into /var/mqsi/shared-classes directory on your IBM ACE host.

Configuring IBM ACE Tracing user exit

This section describes how to configure your IBM ACE to enable tracing, including two parts - Configuring user exit and Configuring IBM ACE tracing.

Configuring user exit

Follow steps to enable tracing for IBM ACE.

  1. Stop the integration node.

    mqsistop <integrationNodeName>
  2. Install the user exit code on an integration node by setting the UserExitPath property using the mqsichangeflowuserexits command.

    mqsichangeflowuserexits <integrationNodeName> -o -x /var/mqsi/shared-classes
  3. Activate the user exit.

    User exits can be active or inactive, and are inactive by default. You can activate user exit for an integration node, an integration server or a specific message flow.

    A. Activate the user exit for an integration node.

    mqsichangeflowuserexits <integrationNodeName> -o -a ACEOpenTracingUserExit

    Verify the user exit.

    mqsireportflowuserexits <integrationNodeName> -o

    Example output:

    # mqsireportflowuserexits BK3 -o
    BIP8854I: User Exits active for integration server 'BK3': ACEOpenTracingUserExit.
    BIP8855I: User Exits inactive for integration server 'BK3': .
    BIP8741I: User Exit path for integration server 'BK3': /var/mqsi/shared-classes.
    BIP8071I: Successful command completion.

    Start the integration node.

    mqsistart <integrationNodeName>

    B. Activate the user exit for an integration server.

    Start the integration node.

    mqsistart <integrationNodeName>

    Activate the user exit for an integration server.

    mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ACEOpenTracingUserExit

    C. Activate the user exit for a message flow.

    Start the integration node.

    mqsistart <integrationNodeName>

    Activate the user exit for a message flow.

    mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ACEOpenTracingUserExit

Repeat the steps for other integration nodes, integration servers or message flows that you want to activate the user exit.

See more information at following links.

Configuring IBM ACE Tracing
  1. Go to /var/mqsi/shared-classes directory.
  2. Edit acetracingexit.conf file.

    # configuration for ace tracing exit
    LOG_LEVEL="info" #Log level: info, warn, error, debug
    SPAN_FORMAT="instana"
    INSTANA_AGENT_HOST="localhost" #(optional)
    INSTANA_AGENT_PROTO="http" #(optional)

    Where:

    • LOG_LEVEL specifies the log level, which can be one of info, warn, error and debug, the log file is located at /var/mqsi/trace directory on Linux platform.
    • SPAN_FORMAT specifies where the span data is sent to. Set this variable to instana. IBM ACE Tracing user exit send span data to the host agent endpoint http://localhost:42699 by default. Update the configuration items INSTANA_AGENT_HOST and INSTANA_AGENT_PROTO if you want to send the span data to a remote host agent using https protocol. You should have the same SPAN_FORMAT setting for all IBM ACE instances.
    • INSTANA_AGENT_HOST specifies the agent host where the format span data is sent to, localhost is used by default. If you specify a remote agent host, you also need to add a line http.listen=* in <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg for the remote host agent first as the host agent is not reachable from other hosts by default.
    • INSTANA_AGENT_PROTO specifies the connection type between IBM ACE tracing user exit and the host agent, http and https are supported, http is used by default. If you want to change it to https, you need to follow Setup TLS Encryption for Agent Endpoint to secure the Instana agent endpoint first.
  3. Save the file and restart integration node or integration server.

Repeat these installation and configuration steps on other IBM ACE hosts that you want to enable tracing for.

You can view IBM ACE tracing data on Instana UI.

Unconfiguring IBM ACE Tracing user exit

  1. Unconfigure user exit.

    A. Unconfigure user exit for an integration node.

    Stop the integration node.

    mqsistop <integrationNodeName>

    Deactivate the user exit.

    mqsichangeflowuserexits <integrationNodeName> -o -a ""

    Restart the integration nodes.

    B. Deactivate user exit for an integration server.

    mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -a ""

    C. Deactivate user exit for a message flow.

    mqsichangeflowuserexits <integrationNodeName> -e <integrationServerName> -k <applicationName> -f <messageFlow> -a ""
  2. Repeat steps on other integration nodes.

Troubleshooting IBM ACE tracing

The log files are at /var/mqsi/trace on Linux platform. The file name is aceExit*.log.

You can set log level in acetracingexit.conf as described in Configuring IBM ACE tracing section.