Tag: Apache Kafka

In the world of data processing and messaging systems, terms like “queue” and “streaming” often come up. While they may sound similar, they serve distinct purposes and can significantly impact how systems handle data. Let’s break down their differences in a straightforward way. What Are Message Queues? Imagine a coffee shop where customers place orders […]

Event streaming is a powerful data processing paradigm where events—small, immutable pieces of data—are continuously produced, captured, and processed in real time. Apache Kafka, an open-source distributed event streaming platform, has become the go-to solution for implementing event streaming in modern systems. Understanding Events and Streams An event is a record of an occurrence, such as a […]

Apache Kafka is a powerful distributed messaging system, but like any system, it can face performance bottlenecks. One of the most common challenges is Kafka lag—the delay between producing and consuming messages. Addressing Kafka lag is crucial for maintaining real-time data pipelines and ensuring optimal performance. In this article, we explore practical strategies to reduce […]

Apache Kafka has been a cornerstone of modern event streaming architectures, enabling reliable and scalable data pipelines for businesses worldwide. Traditionally, Kafka has relied on ZooKeeper for managing metadata, configurations, and cluster coordination. However, the introduction of KRaft (Kafka Raft) marks a significant shift in Kafka’s architecture, eliminating the need for ZooKeeper and simplifying cluster management. What […]

Apache Kafka is an open-source platform designed for building real-time data pipelines and streaming applications. Initially developed by LinkedIn and later donated to the Apache Software Foundation, Kafka has become a cornerstone for handling large-scale, high-throughput, and low-latency data streams. At its core, Kafka operates as a distributed messaging system. It allows systems to publish […]

To receive logs from your containers in Kafka topic, we have to do these steps: Install Filebeat echo “deb https://artifacts.elastic.co/packages/8.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list sudo apt-get update && sudo apt-get install filebeat   2. Edit the configuration file /etc/filebeat/filebeat.yml with these lines  3. Enable Filebeat service sudo systemctl enable filebeat 4. Restart Filebeat.