DeepAI AI Chat
Log In Sign Up

Kafka Consumer Group Autoscaler

by   Diogo Landau, et al.

Message brokers enable asynchronous communication between data producers and consumers in distributed environments by assigning messages to ordered queues. Message broker systems often provide with mechanisms to parallelize tasks between consumers to increase the rate at which data is consumed. The consumption rate must exceed the production rate or queues would grow indefinitely. Still, consumers are costly and their number should be minimized. We model the problem of determining the required number of consumers, and the partition-consumer assignments, as a variable item size bin packing variant. Data cannot be read when a queue is being migrated to another consumer. Hence, we propose the R-score metric to account for these rebalancing costs. Then, we introduce an assortment of R-score based algorithms, and compare their performance to established heuristics for the Bin Packing Problem for this application. We instantiate our method within an existing system, demonstrating its effectiveness. Our approach guarantees adequate consumption rates something the previous system was unable to at lower operational costs.


page 1

page 2

page 3

page 4


Basic Analysis of Bin-Packing Heuristics

The bin-packing problem continues to remain relevant in numerous applica...

Jiffy: A Fast, Memory Efficient, Wait-Free Multi-Producers Single-Consumer Queue

In applications such as sharded data processing systems, sharded in-memo...

Distributed Kafka Clusters: A Novel Approach to Global Message Ordering

In contemporary distributed systems, logs are produced at an astounding ...

Bin Packing with Partition Matroid can be Approximated within o(OPT) Bins

We consider the Bin Packing problem with a partition matroid constraint....

Quantification of Disaggregation Difficulty with Respect to the Number of Meters

A promising approach toward efficient energy management is non-intrusive...