Jacob Wiseberg

I’d spent enough time reading other people’s logs through OpenSearch that I wanted to know what setting it up myself, for my own tiny scale, actually involved.

What I was trying to solve

Between the Beelink, my laptop, and a couple of scripts running on a timer, I had logs scattered across journalctl, plain text files, and whatever stdout happened to get redirected to that week. Finding anything meant SSH-ing in and grepping blind. I wanted one place to search across all of it, even if “all of it” was a modest amount of data by any real production standard.

Getting it running

A single-node OpenSearch instance is refreshingly simple to stand up for something this size:

docker run -d --name opensearch \
  -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=<strong password>" \
  opensearchproject/opensearch:2.15.0

Getting logs into it meant picking a shipper. I went with Fluent Bit over the heavier alternatives, mostly because its memory footprint on the Beelink mattered to me and it configures entirely through one small file:

[INPUT]
    Name tail
    Path /var/log/myscripts/*.log
    Tag  myscripts.*

[OUTPUT]
    Name  opensearch
    Match *
    Host  localhost
    Port  9200
    Index myscripts-logs

What changed once I could actually search

The first real payoff came within a day. One of my scripts had been silently failing about once a week for months, and I’d never connected the dots because each failure looked like an isolated one-off in whatever terminal I happened to be watching at the time. Querying across weeks of aggregated logs made the pattern obvious immediately: it always failed on the same day of the week, which turned out to coincide with a scheduled backup job hogging disk I/O at the same time.

GET myscripts-logs/_search
{
  "query": { "match": { "level": "ERROR" } },
  "aggs": { "by_day": { "date_histogram": { "field": "@timestamp", "calendar_interval": "day" } } }
}

That kind of aggregation is trivial once your logs are structured and searchable, and basically impossible when they’re scattered text files on three different machines.

The parts that took tuning

Index management needed more attention than I expected for something this small. Without an index lifecycle policy, my logs index just grew forever, and OpenSearch’s defaults are tuned for clusters much bigger than one node with a few gigabytes of RAM to spare. Setting a simple rollover and retention policy brought things under control:

PUT _plugins/_ism/policies/log-rotation
{
  "policy": {
    "description": "Roll over logs after 7 days or 1gb",
    "default_state": "hot",
    "states": [
      { "name": "hot", "transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "14d" } }] },
      { "name": "delete", "actions": [{ "delete": {} }] }
    ]
  }
}

Where it landed

It’s a genuinely oversized tool for the amount of log data three home scripts and a couple of services produce, and I don’t regret it. Having structured, searchable history instead of scrollback buffers changed debugging from archaeology into a query. The single-node setup took an afternoon; tuning it to not eat all the Beelink’s disk took a bit longer, but that part was the actually educational half.