Optimizing Cross AZ Data Transfer Cost with Apache Kafka

May 11, 2020

Optimizing Cross AZ Data Transfer Cost with Apache Kafka

When using a cloud provider like Amazon Web Services (AWS), Google Cloud (GCP) or Microsoft Azure, one is subject to very complicated on-demand pricing models. One even more complicated aspect of the pricing model is about traffic, also referred to as data transfer. While it is obvious that traffic between Europe and Australia is billed, it is unintuitive that traffic from eu-west-1a to eu-west-1b is billed as well. But behind the innocent-looking “a” and “b” are two totally separate data centers. These data centers are usually referred to as “Availability Zone” or short “AZ”. Using several zones in the same geographical region is a cloud architecture best practice, as it protects from bigger local failures, like power outages.

How Kafka Produces Cross AZ Traffic Cost

We will illustrate the cross AZ cost created in several simplified scenarios. Real setups, like Instana’s production setup, have many more instances, but that only makes the math harder to understand and does not significantly contribute to the underlying problem.

Scenario 1: No Replication

We start with a simple setup without replication. We have two producers in eu-west-1a and eu-west-1b, as well as two consumers also in eu-west-1a and eu-west-1b. The data sent via Kafka is put into two partitions, so that data belonging together is processed by the same consumer. In Instana’s case the assignment of message to a partition is based on the hash of a pseudo-random message-id. So 50% of the data will end up on partition 1, while the other half is going to partition 2.

We have a Kafka broker also in each AZ, so each partition will be in one AZ as well. Since only one of the partitions is in the same AZ as the producer, 50% of the data needs to travel to the other AZ to reach its broker.

The consumers by default use the Kafka RangeAssignor, which makes sure that consumers in the same group get their fair share of partitions to read from. Unfortunately it does not care about the AZ of the partition. The arrows in the picture above show the worst case: the consumer reads from the partition in the other AZ. In the best case it would use the same AZ, with a larger number of consumers and partitions, the most likely scenario is that half of the traffic will cross AZ boundaries.

Best case cross AZ traffic: 50%

Worst case cross AZ traffic: 150%

Scenario 2: Replication

This setup enhances the previous one by using replication. Replication is important to protect against data loss when a Kafka broker crashes.

Now the partitions have a leader and a follower, where the follower is the other broker (and as such also in the other AZ).

Now still the consumer has the same chance of selecting a good or bad partition to read from, but in addition to that all the replication needs to go across AZs

Best case cross AZ traffic: 150%

Worst case cross AZ traffic: 250%

Scenario 3: Replication & Fetch Closest Replica

To mitigate the problem from above, Kafka 2.4 features an option to read from followers if they are closer:


Now consumers still randomly get the same amount of partitions assigned, but for each they will consider the closer replica. That is the one in the same AZ.

To make this work, the Kafka Broker and Consumer need to know their AZ. We use the broker.rack and client.rack properties for that.

Best case cross AZ traffic: 150%

Worst case cross AZ traffic: 150%

Scenario 4: Replication in Same AZ

The previous example assigned the replicas to other brokers randomly, but in Instana’s case, we use Kafka as a ephemeral message bus and consider a AZ failure a rare event in which we would accept temporarily losing a few messages. What is more important however is to protect against instance failure (or maintenance). Therefore we still run replication, but not across availability zones.

This setup produces the same worst case cross AZ cost as Setup 3, but since we are running on Kafka since the start, this was our solution before KIP-392 was released in Kafka 2.4. In best and average cases, it still produces less cross AZ traffic.

Best case cross AZ traffic: 50%

Worst case cross AZ traffic: 150%

Scenario 5: Replication in Same AZ & Custom Assignor

By implementing a custom assignor, we can have the consumers prefer partitions that are in the same AZ. The code for our custom assignor can be found here:


It requires the broker.rack setting and a custom property instana.az.

Best case cross AZ traffic: 50%

Worst case cross AZ traffic: 50%


Due to the cost of cross AZ traffic, it is important to know that small changes can have a significant impact. A naive implementation with replication produces 5 times the cross AZ traffic and as such also 5 times the cost than a well designed setup.

Amazon Web Services https://cloud.google.com/compute/network-pricing#general and Google Cloud (https://cloud.google.com/compute/network-pricing#general) charge both $0.01 / GB. Microsoft Azure is going to introduce pricing for this in July 2020

With just 1TB Kafka traffic per day, this translates to $750 per month, while the cheapest solution only costs $150.

Since all scenarios above at least send 50% of the traffic across AZs, it is recommended to also use compression, for example the Kafka built in lz4.

Play with Instana’s APM Observability Sandbox

Developer, Engineering
At Instana we are operating our Clickhouse clusters on m5.12xlarge AWS EC2 instances and similar sized custom build Google Cloud instances. The biggest cost driver however is not the size of the...

Start your FREE TRIAL today!

Instana, an IBM company, provides an Enterprise Observability Platform with automated application monitoring capabilities to businesses operating complex, modern, cloud-native applications no matter where they reside – on-premises or in public and private clouds, including mobile devices or IBM Z.

Control hybrid modern applications with Instana’s AI-powered discovery of deep contextual dependencies inside hybrid applications. Instana also gives visibility into development pipelines to help enable closed-loop DevOps automation.

This provides actionable feedback needed for clients as they to optimize application performance, enable innovation and mitigate risk, helping Dev+Ops add value and efficiency to software delivery pipelines while meeting their service and business level objectives.

For further information, please visit instana.com.