Hi All,

I am starting a series of tutorials / learning for Apache Kafka. I will try to cover all aspects of Apache Kafka. I will start with Basic concepts and move on to advance features. Be ready for lot of theory and try to grasp the concepts.

So What is Kafka ?

Apache Kafka is a fast, scalable, fault tolerant messaging system, which enable communication between producers and consumers using messaging based topics. One of the feature of Kafka is it’s highly available and resilient to node failures and supports automatic recovery. In short Kafka is the most sort after messaging system as compared to ActiveMQ, RabbitMQ.

Messaging System

The main task of a Messaging System is to transfer data from one application to another, so that application can  focus on data and not worry about how to transfer / share data with other system.  There are two types of messaging patterns. available:

  1. Point to Point Messaging
    • In this pattern the message produced continues to remain in queue. The consumer/s can consume the message from the queue, but only one consume can consume a particular message.
  2. Publish – Subscribe Messaging
    • In this pattern the message produced continues to remain in a topic. The consumer/s can take more than one topic and consume every message in that topic. In Kafka message producers are called Kafka producers ( publishers) and Message consumers are Kafka consumers (subscribers).

History ( Source : Apache Kafka – Wikipedia)

Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. Graduation from the Apache Incubator occurred on 23 October 2012. Jay Kreps chose to name the software after the author Franz Kafka because it is “a system optimized for writing”, and he liked Kafka’s work.