Optimizing Cross AZ Data Transfer Cost with Apache Kafka

May 11, 2020

When using a cloud provider like Amazon Web Services (AWS), Google Cloud (GCP) or Microsoft Azure, one is subject to very complicated on-demand pricing models. One even more complicated aspect of the pricing model is about traffic, also referred to as data transfer. While it is obvious that traffic between Europe and Australia is billed, it is unintuitive that traffic from eu-west-1a to eu-west-1b is billed as well. But behind the innocent-looking “a” and “b” are two totally separate data centers. These data centers are usually referred to as “Availability Zone” or short “AZ”. Using several zones in the same geographical region is a cloud architecture best practice, as it protects from bigger local failures, like power outages.

How Kafka Produces Cross AZ Traffic Cost

We will illustrate the cross AZ cost created in several simplified scenarios. Real setups, like Instana’s production setup, have many more instances, but that only makes the math harder to understand and does not significantly contribute to the underlying problem.

Scenario 1: No Replication

We start with a simple setup without replication. We have two producers in eu-west-1a and eu-west-1b, as well as two consumers also in eu-west-1a and eu-west-1b. The data sent via Kafka is put into two partitions, so that data belonging together is processed by the same consumer. In Instana’s case the assignment of message to a partition is based on the hash of a pseudo-random message-id. So 50% of the data will end up on partition 1, while the other half is going to partition 2.

We have a Kafka broker also in each AZ, so each partition will be in one AZ as well. Since only one of the partitions is in the same AZ as the producer, 50% of the data needs to travel to the other AZ to reach its broker.

The consumers by default use the Kafka RangeAssignor, which makes sure that consumers in the same group get their fair share of partitions to read from. Unfortunately it does not care about the AZ of the partition. The arrows in the picture above show the worst case: the consumer reads from the partition in the other AZ. In the best case it would use the same AZ, with a larger number of consumers and partitions, the most likely scenario is that half of the traffic will cross AZ boundaries.

Best case cross AZ traffic: 50%

Worst case cross AZ traffic: 150%

Scenario 2: Replication

This setup enhances the previous one by using replication. Replication is important to protect against data loss when a Kafka broker crashes.

Now the partitions have a leader and a follower, where the follower is the other broker (and as such also in the other AZ).

Now still the consumer has the same chance of selecting a good or bad partition to read from, but in addition to that all the replication needs to go across AZs

Best case cross AZ traffic: 150%

Worst case cross AZ traffic: 250%

Scenario 3: Replication & Fetch Closest Replica

To mitigate the problem from above, Kafka 2.4 features an option to read from followers if they are closer:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica

Now consumers still randomly get the same amount of partitions assigned, but for each they will consider the closer replica. That is the one in the same AZ.

To make this work, the Kafka Broker and Consumer need to know their AZ. We use the broker.rack and client.rack properties for that.

Best case cross AZ traffic: 150%

Worst case cross AZ traffic: 150%

Scenario 4: Replication in Same AZ

The previous example assigned the replicas to other brokers randomly, but in Instana’s case, we use Kafka as a ephemeral message bus and consider a AZ failure a rare event in which we would accept temporarily losing a few messages. What is more important however is to protect against instance failure (or maintenance). Therefore we still run replication, but not across availability zones.

This setup produces the same worst case cross AZ cost as Setup 3, but since we are running on Kafka since the start, this was our solution before KIP-392 was released in Kafka 2.4. In best and average cases, it still produces less cross AZ traffic.

Best case cross AZ traffic: 50%

Worst case cross AZ traffic: 150%

Scenario 5: Replication in Same AZ & Custom Assignor

By implementing a custom assignor, we can have the consumers prefer partitions that are in the same AZ. The code for our custom assignor can be found here:

https://gist.github.com/CodingFabian/695f859e882cfc42e2007f9de9dca463

It requires the broker.rack setting and a custom property instana.az.

Best case cross AZ traffic: 50%

Worst case cross AZ traffic: 50%

Conclusion

Due to the cost of cross AZ traffic, it is important to know that small changes can have a significant impact. A naive implementation with replication produces 5 times the cross AZ traffic and as such also 5 times the cost than a well designed setup.

Amazon Web Services https://cloud.google.com/compute/network-pricing#general and Google Cloud (https://cloud.google.com/compute/network-pricing#general) charge both $0.01 / GB. Microsoft Azure is going to introduce pricing for this in July 2020

With just 1TB Kafka traffic per day, this translates to $750 per month, while the cheapest solution only costs $150.

Since all scenarios above at least send 50% of the traffic across AZs, it is recommended to also use compression, for example the Kafka built in lz4.

Play with Instana’s APM Observability Sandbox

Developer, Engineering
At Instana we are operating our Clickhouse clusters on m5.12xlarge AWS EC2 instances and similar sized custom build Google Cloud instances. The biggest cost driver however is not the size of the...
|

Start your FREE TRIAL today!

As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit https://www.instana.com to learn more.