Kafka

Building reliable event streaming with Apache Kafka.

Quick Start

docker compose up -d  # single-node kafka
kafka-topics --bootstrap-server localhost:9092 --create --topic events --partitions 3
kafka-console-producer --bootstrap-server localhost:9092 --topic events

When to Use

• Event-driven architecture and decoupling
• High-throughput log ingestion and analytics
• Stream processing (join, aggregate, window)
• Durable message replay for consumers

Best Practices

Topics & Partitioning

• Model topics by event type, not by consumer
• Partition by key for ordering within a key
• Choose partition count based on throughput
• Set retention per topic (time and size)

Producers

• Use a key for order/co-partitioning needs
• Set acks (all) for durability; tune batching
• Handle retries and idempotence (enable idempotent producer)
• Monitor `record-error-rate` and latency

Consumers

• Use consumer groups for scale
• Handle rebalance gracefully
• Commit offsets after processing (at-least-once)
• Make consumption idempotent for exactly-once semantics

Operations

• Run at least 3 brokers with replication factor 3
• Monitor lag, under-replicated partitions, and disk
• Use Schema Registry for schema evolution
• Set alerts on consumer lag

Dependencies

# Python client
pip install confluent-kafka
# or Java:
# org.apache.kafka:kafka-clients

Examples

from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "localhost:9092"})

def acked(err, msg):
    if err is not None:
        print(f"failed: {err}")

p.produce("events", key="user-42", value=b'{"action":"login"}', callback=acked)
p.flush()

from confluent_kafka import Consumer, KafkaError

c = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "analytics",
    "auto.offset.reset": "earliest",
})
c.subscribe(["events"])

while True:
    msg = c.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        if msg.error().code() == KafkaError._PARTITION_EOF:
            continue
        break
    process(msg.value())        # idempotent processing
    c.commit(asynchronous=False)  # commit after processing

// Java producer with acks and idempotence
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("enable.idempotence", "true");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("events", "user-42", "login"));

# Monitor consumer lag
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --describe --group analytics

Step-by-Step

1. Define event types and topic names up front.

2. Choose partitioning strategy per key.

3. Set replication factor and retention.

4. Implement producers with acks and idempotence.

5. Implement consumers with graceful rebalance and offsets.

6. Make processing idempotent for safe replays.

7. Use Schema Registry for contract evolution.

8. Monitor lag, disk, and error rates.

Validation

1. Producers send without errors under load

2. Consumers process in order within a key

3. Consumer lag stays near zero at steady state

4. Rebalances don't lose or duplicate processed events

5. Replaying from an offset produces identical results

Troubleshooting

• High lag: scale consumers or optimize processing.
• Out-of-order: partition by key and check single-consumer-per-partition.
• Duplicates: make processing idempotent or use exactly-once.