Created at LinkedIn and then donated to the Apache Foundation in 2011, Apache Kafka is a popular open source stream processing and data integration platform. Thousands of organizations, and 80% of the Fortune 100, rely on Kafka for their high-performance data pipelines, streaming analytics, data integration, and to stream data for business-critical applications.
This page includes resources and information to help teams using Apache Kafka optimize and improve their deployments.
- Apache Kafka Versions and Support Lifecycle
- How Kafka Works and Related Technologies
- Streaming Data With Kafka
- Apache Kafka Alternatives
- Ensuring Support for Your Kafka Deployments
Kafka has been a top-level Apache Software Foundation project since version 0.8 was developed in 2013. Since Kafka 1.0 debuted in 2017, major releases have been rolled out roughly every four months, or three times per year.
|Kafka Version||Release Date||Support End-of-Life|
|Apache Kafka 1.0||November 1, 2017||July 8, 2019|
|Apache Kafka 1.1||March 28, 2018||July 19, 2019|
|Apache Kafka 2.0||July 30, 2018||November 9, 2019|
|Apache Kafka 2.1||November 20, 2018||February 15, 2020|
|Apache Kafka 2.2||March 22, 2019||June 1, 2020|
|Apache Kafka 2.3||June 25, 2019||October 24, 2020|
|Apache Kafka 2.4||December 16, 2019||February 4, 2021|
|Apache Kafka 2.5||April 15, 2020||August 10, 2021|
|Apache Kafka 2.6||August 3, 2020||April 20, 2022|
|Apache Kafka 2.7||December 21, 2020||June 15, 2022|
|Apache Kafka 2.8||April 19, 2021||November 15, 2022|
|Apache Kafka 3.0||September 21, 2021||January 2023 (Estimated)|
|Apache Kafka 3.1||January 24, 2022||May 2023 (Estimated)|
|Apache Kafka 3.2||May 17, 2022||August 2023 (Estimated)|
|Apache Kafka 3.3||October 3, 2022||February 2024 (Estimated)|
The community support window for Kafka major releases is usually about 16 months. Here’s what the community says about their release cadence and EOL policy:
“Given 3 releases a year and the fact that no one upgrades three times a year, we propose making sure (by testing!) that [a] rolling upgrade can be done from each release in the past year (i.e. last 3 releases) to the latest version. We will also attempt, as a community to do bugfix releases as needed for the last 3 releases.”
By implementing strategies for continuous integration and rolling upgrades, organizations can keep up with the latest versions of Kafka. It’s important to remember that there are no patches in community open source software, only new versions, so being able to upgrade quickly, reliably, and confidently is the best way to avoid security vulnerabilities and bugs.
While Kafka can be used in an application as a message handler, it is not a message broker; it is a stream processor. It can run as a cluster, with records stored in topics, and topics divided into partitions. Knowing when to use a true message broker, such as Apache ActiveMQ, and when to use a streaming event platform like Kafka can make all the difference in a new project.
Read more about how Kafka works >>
Kafka Connect provides a platform to reliably stream data to/from Apache Kafka and external data sources/destinations. It’s a centralized hub for basic data integration between data platforms (such as databases, index engines, file stores, and key-value repositories), integrating them by allowing developers to share and reuse connectors.
Read more about Kafka Connect >>
Kafka partitions work by creating multiple logs from a single topic log and spreading them across one or more brokers. Partitions are what makes Kafka scalable, and the most common partitioning strategies are round robin partitioning and message key partitioning.
Read about how to get started with Kafka partitions >>
ZooKeeper provides distributed configuration services, synchronization services, leadership election services, and a naming registry for a number of open source projects including Kafka, Hadoop, Spark, SOLR, NiFi, and HBase. Kafka and ZooKeeper work together to form a complete Kafka Cluster — with ZooKeeper providing distributed clustering services, and Kafka handling the actual data streams and connectivity to clients. However, much work has been completed on KIP-500, which will deprecate ZooKeeper over time, so understanding the upgrade path is important.
Read more about using Kafka with ZooKeeper >>
Modern systems generate more data than ever before. And, for complex enterprise systems, facilitating that data in real-time is key to business success. Harnessing data streams at scale is a challenge for many organizations, but Kafka is a low-latency, highly durable stream processing platform capable of handling huge volumes of data.
See Why Cassandra, Kafka, and Spark Are Right for Streaming Data
In this white paper by the Bloor Group, learn about the “New Stack” of Cassandra, Kafka, and Spark, and why it’s the ideal approach for enterprises who need to facilitate large amounts of streaming data, and support new types of real-time, event-driven applications.
Within the Kafka platform, the streams API helps with wiring applications to manipulate streams, functioning as a stream processor. Events are managed in real-time and routed from different sources to different destinations continuously. This is done with all the necessary controls for dependable usability to write, store, and process small and large volumes of data.
In the current technology landscape, existing applications and workloads are looking for expedited delivery. With real-time data lakes, it is possible to leverage Spark and Kafka in multi-data environments that demand capturing and processing in seconds.
Watch on-demand webinar about real-time Kafka data lakes >>
Given how harmful a data breach can be to an organization’s reputation, it’s important to keep your Kafka environment secure. This means not running the default configuration and regularly upgrading, as well as implementing encryption, hardening, and authentication processes.
Read more about Kafka security best practices >>
Kafka on Kubernetes
Running Kafka on a container orchestration system like Kubernetes allows it to be deployed in the most native way possible. When combined with Strimzi, managing and running Kafka Clusters on Kubernetes is easy — whether that's on-prem, or across public, private, or hybrid clouds.
Read more about deploying Kafka on Kubernetes >>
Many of the technologies that often get compared to Kafka are not true 1:1 alternatives, but have some overlapping capabilities and similar enterprise use cases.
RabbitMQ is an open-source distributed message broker. While both Kafka and RabbitMQ use a publish/subscribe messaging system, Kafka takes a pull-based approach, whereas RabbitMQ employs a push model.
Read more about Kafka vs. RabbitMQ >>
Although they often get compared, Kafka is an open source stream processing platform, while Redis is a general purpose in-memory data store, so the two technologies serve very different functions within application data infrastructure.
Read more about Kafka vs. Redis >>
Kafka Streams vs. Apache Flink
Although initially created for different use cases, Apache Flink and Kafka Streams API have a lot of overlap in their application. While both can solve stream processing challenges, their differences in deployment, architecture, and more are worth considering before making a decision.
Read more about Kafka Streams vs. Flink >>
Other Kafka Alternatives
Other open source and commercial technologies that frequently get compared to, or discussed alongside, Apache Kafka include Spark, Hadoop, and ActiveMQ.
If you’re using Kafka at scale, when things go wrong, they go wrong at the same scale.
OpenLogic provides SLA-backed, 24/7/365 Kafka technical support and services — our experienced enterprise architects can help configure, improve, and maintain your entire Kafka stack, including Kafka for data lakes, Kafka on bare metal, Kafka with Kubernetes operators, and Kafka with Prometheus and Grafana.