OpenSearch Circuit Breakers

Types of Circuit Breakers in OpenSearch

• Parent Circuit Breaker: This breaker sets the overall memory limit for all child circuit breakers. By default, it is configured to 95% of the JVM heap size. It’s crucial to ensure that the indices.breaker.request.limit is set lower than the parent breaker to prevent the parent breaker from being triggered prematurely.
• Fielddata Circuit Breaker: This breaker limits the memory used to load fields into the fielddata cache, which is essential for operations like aggregations and sorting. The default limit is 40% of the JVM heap. To adjust this limit, you can use the following command:

PUT /_cluster/settings
{
  "persistent": {
    "indices.breaker.fielddata.limit": "60%"
  }
}

• Request Circuit Breaker: This breaker estimates the memory required to process a request, including memory for aggregations. The default limit is 60% of the JVM heap. If a request exceeds this limit, it is terminated to prevent memory overload.
• In-Flight Requests Circuit Breaker: This breaker monitors the memory usage of active incoming requests. The default limit is set to 100% of the JVM heap, which is effectively controlled by the parent circuit breaker.
• Script Compilation Circuit Breaker: This breaker limits the number of inline script compilations within a specified time interval. The default setting allows 150 compilations every 5 minutes. To adjust this limit, you can modify the script.max_compilations_rate setting.

Handling Circuit Breaker Exceptions

• Review Query and Mapping: Examine your queries and index mappings to identify operations that consume excessive memory. For instance, using large size values in aggregations can lead to high memory usage.
• Optimize Fielddata Usage: Avoid enabling fielddata on text fields unless absolutely necessary, as it can consume significant memory. Instead, use keyword fields for aggregations and sorting.
• Adjust Circuit Breaker Settings: While it’s generally advisable to keep default settings to prevent OutOfMemoryErrors, you can adjust circuit breaker limits if you have a clear understanding of your workload and memory requirements. For example, to increase the fielddata limit to 60%, use the command mentioned earlier.
• Scale Your Cluster: If your workload consistently exceeds memory limits, consider scaling your cluster by adding more nodes or increasing the JVM heap size to accommodate the increased memory demands.