HA Kafka + HA ZooKeeper
We deliver highly available solution for processing of real time activity stream data
We chose Kafka and ZooKeeper for this purpose.
Kafka cluster typically consists of multiple brokers to maintain load balance. Kafka brokers are stateless, so they use ZooKeeper for maintaining their cluster state. One Kafka broker instance can handle hundreds of thousands of reads and writes per second and each broker can handle TB of messages without performance impact. Kafka broker leader election can be done by ZooKeeper.
ZooKeeper is used for managing and coordinating Kafka broker. ZooKeeper service is mainly used to notify producer and consumer about the presence of any new broker in the Kafka system or failure of the broker in the Kafka system. As per the notification received by the ZooKeeper regarding presence or failure of the broker then producer and consumer takes decision and starts coordinating their task with some other broker.
We build highly available Kafka clusters, which are able to stay alive after several nodes gone down.
Also we recommend to build ZooKeeper cluster that will allows to avoid single point of failure.
Apache Kafka, Apache ZooKeeper