Back to Home
    KafkaKafkaintermediate

    Kafka

    Practical guide to building, operating, and scaling real-world Apache Kafka event streaming systems.

    12 min
    kafkaevent-streamingdistributed-systemsmessagingdata-engineering

    Core Concepts and Design

    3 topics

    Topic, Partition, Offset (Why ordering is local)

    Kafka stores records in topics split into partitions. Ordering is guaranteed only within a partition, so keying strategy directly affects correctness.

    orders topic
├─ partition-0: [offset 0,1,2,...]
├─ partition-1: [offset 0,1,2,...]
└─ partition-2: [offset 0,1,2,...]

Key rule:
- Same key -> same partition (with default partitioner)
- Ordered processing only within that partition
    Kafka Concepts

    Use business keys (e.g., customer_id, order_id) for deterministic routing.

    Do not assume global ordering across all partitions.

    partitioningorderingdata-modeling

    Replication, ISR, and Durability

    Kafka replicates partitions across brokers. The leader handles reads/writes, while followers replicate. In-sync replicas (ISR) determine write safety.

    properties
    # producer durability settings
acks=all
retries=2147483647
enable.idempotence=true

# topic/broker durability factors
replication.factor=3
min.insync.replicas=2
    Replication

    For critical data, combine acks=all with min.insync.replicas >= 2.

    Avoid replication.factor=1 in production.

    durabilityreplicationproduction

    Delivery Semantics in Practice

    Kafka supports at-most-once, at-least-once, and effectively exactly-once patterns depending on producer/consumer configuration and transaction usage.

    At-most-once:
- Commit consumer offset before processing
- May lose messages on failure

At-least-once:
- Process first, commit offset after success
- May reprocess duplicates

Exactly-once (Kafka-to-Kafka pipelines):
- Idempotent producer + transactions + read_committed consumers
    Kafka Delivery Guarantees

    Most business systems use at-least-once plus idempotent consumers.

    Exactly-once is valuable but adds operational complexity.

    semanticsreliabilityarchitecture

    Producer Patterns

    2 topics

    Reliable Python Producer (Confluent Kafka)

    Real-world producer with retries, idempotence, keying, and delivery callback for observability.

    python
    from confluent_kafka import Producer
import json

conf = {
    'bootstrap.servers': 'kafka-1:9092,kafka-2:9092',
    'client.id': 'billing-service',
    'acks': 'all',
    'enable.idempotence': True,
    'retries': 1000000,
    'linger.ms': 20,
    'compression.type': 'snappy'
}

p = Producer(conf)

def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(f"Delivered to {msg.topic()}[{msg.partition()}]@{msg.offset()}")

order_event = {
    'order_id': 'ORD-10021',
    'customer_id': 'CUST-42',
    'amount': 89.90,
    'status': 'PAID'
}

p.produce(
    topic='orders.events.v1',
    key=order_event['customer_id'],
    value=json.dumps(order_event),
    callback=delivery_report
)
p.flush()
    Confluent Python Client

    Use keys to preserve per-entity ordering (e.g., customer timeline).

    Batching (linger.ms) improves throughput at slight latency cost.

    producerpythonreliability

    Schema-Aware Events with Avro + Schema Registry

    Use schema evolution to avoid breaking consumers when event shape changes over time.

    json
    {
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.events",
  "fields": [
    {"name": "order_id", "type": "string"},
    {"name": "customer_id", "type": "string"},
    {"name": "amount", "type": "double"},
    {"name": "status", "type": "string"},
    {"name": "coupon_code", "type": ["null", "string"], "default": null}
  ]
}
    Schema Registry Fundamentals

    Prefer backward-compatible schema changes for consumer safety.

    Add optional fields with defaults instead of renaming/removing fields.

    schema-registryavrocompatibility

    Consumer Patterns

    2 topics

    Manual Offset Commit After Processing

    Classic at-least-once pattern: process message first, commit only on success.

    python
    from confluent_kafka import Consumer, KafkaException
import json

conf = {
    'bootstrap.servers': 'kafka-1:9092',
    'group.id': 'shipping-service-v1',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False
}

c = Consumer(conf)
c.subscribe(['orders.events.v1'])

try:
    while True:
        msg = c.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            raise KafkaException(msg.error())

        event = json.loads(msg.value().decode('utf-8'))
        # business logic: create shipment
        print(f"Creating shipment for {event['order_id']}")

        c.commit(message=msg, asynchronous=False)
finally:
    c.close()
    Kafka Consumer Configs

    If processing fails, do not commit; message will be retried.

    Make handlers idempotent to tolerate duplicate processing.

    consumeroffsetsat-least-once

    Consumer Group Scaling and Rebalance Safety

    Scale horizontally by increasing consumer instances in the same group. Each partition is consumed by only one consumer in that group.

    bash
    # Start a consumer group listener (console tool)
kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic orders.events.v1 \
  --group fraud-detector-v1 \
  --from-beginning

# Inspect lag and partition assignment
kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --group fraud-detector-v1 \
  --describe
    Consumer Groups

    Max parallelism per group is bounded by partition count.

    Use cooperative rebalancing strategies to reduce stop-the-world rebalances.

    consumer-groupsscalingoperations

    Operations and Troubleshooting

    3 topics

    Create Topics with Production-Friendly Defaults

    Define partitions, replication, retention, and cleanup policy intentionally for each workload.

    bash
    kafka-topics --bootstrap-server localhost:9092 --create \
  --topic orders.events.v1 \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000 \
  --config cleanup.policy=delete
    Topic Configs

    Set partitions based on expected throughput and consumer parallelism.

    Avoid frequent partition count changes for keyed topics unless you accept key remapping.

    topicsretentioncli

    Monitor Consumer Lag and Health

    Lag indicates how far consumers are behind producers. Persistent lag usually signals under-provisioning or slow processing.

    bash
    kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --all-groups \
  --describe

# Look at:
# CURRENT-OFFSET, LOG-END-OFFSET, LAG
    Kafka Operations

    Alert on sustained lag, not short spikes.

    Correlate lag with CPU, GC, network, and downstream dependency latency.

    monitoringlagsre

    Common Failure Playbook

    Quick response patterns for frequent incidents in production clusters.

    Symptom: NOT_ENOUGH_REPLICAS
Action: Check broker health, ISR shrink, disk/network; verify min.insync.replicas.

Symptom: Growing consumer lag
Action: Scale consumers, increase partitions, optimize handler, inspect slow dependencies.

Symptom: Rebalance storms
Action: Increase session timeout carefully, use static membership/cooperative assignor, reduce deployment churn.
    Kafka Config Reference

    Keep runbooks with exact commands for your cluster tooling.

    Test failure scenarios in staging (broker restart, network partition, slow consumer).

    incident-responsetroubleshootingrunbook

    Related Articles

    Background reading and deeper explanations for this sheet.

    Building Autonomous Agents Without Overengineering: A Practical Guide for Real-World Teams

    Autonomous agents can automate meaningful work, but many projects fail because teams overbuild too early. This practical guide shows you how to design, ship, and improve useful agents with simple patterns, clear guardrails, and real-world examples—without turning your stack into a science project.

    keyword overlap

    Event-Driven Agents with Kafka Streams: A Practical Guide for Building Real-Time AI Workflows

    Event-driven agents let you move from slow, request/response automation to responsive systems that react in milliseconds to real-world signals. In this hands-on guide, you’ll learn how to design, build, and operate agentic workflows using Apache Kafka and Kafka Streams, with practical patterns, code snippets, and production-ready advice.

    keyword overlap

    GenAI SaaS Architecture: A Practical Blueprint for Building, Scaling, and Securing AI Products

    Designing a SaaS product on top of GenAI is more than calling an LLM API—it requires thoughtful architecture across product, data, safety, and operations. This practical guide walks you through a beginner-friendly yet deep system blueprint, with real-world patterns, code snippets, and deployment strategies you can use immediately.

    keyword overlap

    How to Build and Monetize an AI SaaS: A Practical Step-by-Step Guide for Founders

    Building an AI SaaS is more accessible than ever—but turning it into a profitable business requires more than calling an API. In this practical guide, you’ll learn how to validate your idea, ship an MVP, design pricing, and grow revenue with real-world examples, architecture patterns, and implementation tips.

    keyword overlap

    codesips.com
    All SheetsLearning PathsArchitectureDiagramsCategoriesBlogCommunityRequestsAdmin Login

    © 2026 codesips.com. All rights reserved.