Jacob Wiseberg

I finally set up Kafka on the Beelink instead of just reading about it at work-adjacent distance. It turns out running a message broker for yourself, with real but tiny traffic, teaches you things that production dashboards never will.

Why bother at home

Message queues are one of those technologies I understood in theory — producers, consumers, partitions, offsets — without ever having broken one myself. Reading about Kafka’s guarantees is not the same as watching a consumer group rebalance in real time because you killed the wrong process. I wanted a sandbox where breaking things cost nothing.

The actual trigger was small: I wanted to decouple a couple of scripts on the Beelink that process photos and documents for the household. One script polls a folder, another resizes images, a third backs things up to cold storage. They were all just calling each other directly, which worked until one got slow and backed up the rest.

Getting it running

Docker Compose made this almost insultingly easy compared to the mental overhead I’d built up around it:

services:
  kafka:
    image: apache/kafka:3.7.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://beelink.local:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093

No Zookeeper needed anymore with KRaft mode, which simplified things considerably from the setup I remembered reading about a few years ago.

Rewiring the scripts

Each script became a simple producer or consumer instead of calling the next one directly:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers='beelink.local:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
producer.send('new-photos', {'path': str(photo_path), 'size': photo_path.stat().st_size})

The resize script subscribes to new-photos, does its work, and publishes to resized-photos for the backup script to pick up. If the backup script is slow or offline, photos just queue up instead of the whole pipeline stalling.

What actually surprised me

Consumer groups are more subtle than I expected. I initially ran two instances of the resize script for redundancy, assuming Kafka would load-balance between them. It does, but only across partitions, and my topic had exactly one partition. Both consumers were technically in the group, but only one was ever doing anything. Bumping the topic to three partitions fixed it immediately, and watching the rebalance happen live made the abstraction click in a way no diagram had.

I also underestimated how much visibility you get almost for free. kafka-consumer-groups.sh --describe showing consumer lag per partition turned “is the backup script keeping up” from a guess into a number I could watch.

Where this leaves me

The whole thing runs comfortably in a corner of the Beelink’s RAM, alongside everything else it already does. It’s genuinely overkill for three household scripts, and I’d be lying if I said that wasn’t part of the appeal. Sometimes the best way to actually understand a piece of infrastructure is to give it a job that’s too small to need it.