Addressing read_only_allow_delete After Disk Space Issues

Occasionally, Elasticsearch administrators may encounter a situation where all indices are automatically set to read_only_allow_delete=true, preventing write operations. This typically occurs when the cluster runs out of available disk space. In this article, we’ll explain why this happens, how to resolve it, and how to prevent it in the future.

Why Do Indices Become read_only_allow_delete=true

Elasticsearch includes built-in safeguards to prevent nodes from running out of disk space. When a node reaches specific disk usage thresholds, Elasticsearch automatically applies a read-only block to protect the cluster’s stability.

Here’s how it works:

1. Thresholds:
   • Low watermark: Elasticsearch warns that disk space is running low.
   • High watermark: New shards are not allocated to nodes with insufficient disk space.
   • Flood stage: Indices with shards on the affected node are set to read_only_allow_delete=true to prevent further writes.
2. Even after clearing up disk space, the read_only_allow_delete setting is not automatically removed. Administrators must reset it manually.

How to Fix the Issue

1. Removing the read_only_allow_delete Block
To remove the block for all indices, use the following API request:

PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": false
}

If you need to apply this change to a specific index, replace _all with the name of the index.
2. Adjust Disk Watermark Settings
To prevent future issues, review and adjust your cluster’s disk watermark thresholds (disk.watermark.low, disk.watermark.high, disk.watermark.flood_stage) based on your infrastructure’s requirements.
Use percentage-based or byte-based values consistently, as Elasticsearch doesn’t allow mixing these formats. Example settings:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}

Is This Behavior Expected?

Yes, this is the expected behavior of Elasticsearch. It’s a protective mechanism designed to maintain cluster stability and prevent data loss when disk space is critically low.

Conclusion

If your indices are marked as “read-only” due to disk space issues, it’s a signal to review your disk usage and thresholds. Clear the read_only_allow_delete setting, ensure sufficient disk space, and validate your watermark configurations to avoid similar incidents in the future. This proactive approach ensures the reliable operation of your Elasticsearch cluster.