Elasticsearch locks indexes in read-only mode if the disk is full. What steps can you take to prevent this during log surges?

Developers began reporting that recent logs were missing in Kibana. While this issue had cropped up in the past for various reasons (a story for another time), this instance was different. The log structure was intact, and the FluentD shipper appeared to be working fine.

Upon investigation, we discovered that one application had gone rogue, flooding the system with an excessive volume of logs. This surge quickly filled up the disks on our Elasticsearch nodes, causing Elasticsearch to reject incoming logs. When disk space runs out, Elasticsearch automatically switches all indexes to read-only mode.

Curator for Logstash

Curator is a tool that lets you define how long to keep log indexes in Elasticsearch — typically measured in days. This is by far the most common retention approach.

In our setup, we configured Curator to retain the last two weeks of logs. Under normal conditions, with a predictable log volume, this worked without issues. You can estimate the right retention period to avoid filling up disk space on Elasticsearch nodes.

The trouble starts when log volume suddenly spikes — for example, due to a misbehaving application or a denial-of-service attack. In those moments, the newest logs are the most valuable, as they help track the attack’s progression or understand how an outage is spreading.

Curator offers another mode: deleting (or performing other actions) when an index reaches a specific size in gigabytes. This approach can get you closer to ideal disk usage — provided you know the total available disk space for the entire cluster and can manage those values consistently across environments (dev, test, prod).

What I really wanted was straightforward: keep as many logs as the available disk space allows, but never block new log events when disks run out of space (more on that later).

Unfortunately, Curator doesn’t support this mode, and despite searching for alternatives, I couldn’t find a tool that does.

Our answer? A simple Bash script.

Fortunately, Elasticsearch’s API makes it easy to check disk usage on nodes. With just a few curl commands and a bit of Bash scripting, you can implement a solution that prevents data loss when disks fill up.

#!/bin/bash
# Newline\tab as only separator, required for for loop
IFS=$'\n\t'
# Fail on first error
set -euo pipefail

ELASTIC_URL=${ELASTIC_URL:=localhost:9200}
# At 90% usage ES will try to move shards to other nodes. See `disk.watermark.high` in docs.
DISK_WATERMARK=88

NODES_UTILIZATION=$(curl --fail-with-body -s -X GET "$ELASTIC_URL/_cat/allocation?h=disk.percent&pretty")
for DISK_USAGE in $NODES_UTILIZATION; do
    if [ "$DISK_USAGE" -gt "$DISK_WATERMARK" ]; then
        OLDEST_INDEX="$(curl --fail-with-body -s -X GET "$ELASTIC_URL/_cat/indices/logstash-*?h=index&s=index" | head -n 1)"
        curl --fail-with-body -s -X DELETE "$ELASTIC_URL/$OLDEST_INDEX"
        exit 0
    fi
done

As you can see, the approach is fairly simple. One thing to note: it uses the --fail-with-body parameter, introduced in curl 7.76.0, which might not be available on older Linux distributions. This option is only there to display Elasticsearch’s error response for debugging purposes.

Run script periodically

By default, Logstash creates a new index each day, with names following the logstash-YYYY-MM-DD format. The script above assumes this naming pattern in its _cat/indices/logstash-* GET query.

However, running the script only once per day isn’t enough. An application could start logging excessively in the evening and fill up the storage before the day ends. In that case, you’d lose valuable logs capturing the events leading up to the failure.

The fix is straightforward: schedule the script to run via cron every hour. This approach worked flawlessly in our environment.

Why choose these disk usage thresholds?

Deleting an index at roughly 90% disk usage is tied to how Elasticsearch behaves as free space runs low. The official documentation isn’t crystal clear on this, so here’s a quick breakdown of the default thresholds:

  • 85% – Elasticsearch stops allocating new shards to the node (disk.watermark.low).
  • 90% – Elasticsearch attempts to move shards away from the node (disk.watermark.high).
  • 95% – Elasticsearch sets affected indexes to read-only (disk.watermark.flood_stage).

The goal is to avoid hitting 90%, but even that can fail in practice. For example, if one node exceeds 90%, Elasticsearch will try to reallocate shards — but if other nodes are already above 85%, allocation is blocked. With equal disk sizes and evenly distributed shards, this is still a risk.

Even if shard movement is possible, transferring large log shards between nodes can be so resource-intensive that it grinds the cluster to a halt. To avoid this entirely, we aim for 85% usage. At that point, the node is effectively “cordoned” (to borrow Kubernetes terminology) — Elasticsearch won’t shuffle shards, and we still have breathing room before it triggers emergency actions.

Conclusion

Is it really that simple? Well… yes and no. 😉 The idea is so straightforward that I was sure someone had already implemented it, yet my search turned up nothing — not even a passing mention on an obscure blog.

Log aggregation in Elasticsearch is no walk in the park. You need to define the log event structure, decide what should be indexed, and create index templates. That’s if you control the logging clients; without that control, it’s far more challenging.

On top of this, there’s shard replication, hot and cold indexes, and archival strategies to consider. That’s a whole separate discussion — and one I’d be happy to cover in another post if there’s interest.

And above all, treat your logs like Pokémon: you’ve gotta catch ’em all.