It happens very rarely, but sometimes your cluster gets red status. Red status means that not only has the primary shard been lost but also that the replica has not been upgraded to primary in its place. However, as in the case of yellow status, you should not panic and start firing commands without finding […]
In Elasticsearch pipelines, you might encounter scenarios where fields contain hexadecimal-encoded text. To decode this text into its original readable format, Logstash offers the ability to use Ruby code within the pipeline configuration. This article demonstrates how to achieve this transformation. Why Use Ruby for HEX Decoding?Hexadecimal-encoded text often represents binary data or strings in […]
Elasticsearch uses a security model to control access to data through roles and users. This allows only authorized users to log in and perform certain actions according to roles. Implementing role-based access control is essential for data security and operational integrity in an Elasticsearch environment.This guide explains how to configure roles and users for secure […]
This standard template for configuring Logstash pipelines, commonly referred to as a “gold template,” ensures consistent metadata enrichment for events processed through Logstash, making it particularly useful in environments where data comes from diverse sources. Configuration Template Below is the template with an explanation of its key components: Key Features Ruby Block for Metadata Enrichment […]
When running an Elasticsearch or OpenSearch cluster, efficient disk space management is essential for ensuring stability and performance. These platforms provide configurable settings to manage how shards are allocated based on available disk space. Here, we discuss three key settings related to disk allocation thresholds: 1. cluster.routing.allocation.disk.threshold_enabledThis setting enables or disables disk-based shard allocation. When set […]
Sometimes, you can get the associated error Limit of total fields [1000] has been exceeded I will explain what it is and how to fix it.You can find that error in OpenSaerch/ElasticSearch logs /var/log/opensearch or /var/log/elasticsearchFor example, in the screenshot, you can see that error: In OpenSearch and Elasticsearch, the number of fields in an index […]
Monitoring Elasticsearch is crucial for maintaining its performance and ensuring cluster health. Metricbeat, a lightweight shipper by Elastic, simplifies this process by collecting and sending metrics from your Elasticsearch nodes to a monitoring system like Kibana or Elasticsearch itself. How to Monitor Elasticsearch with Metricbeat 1. Install MetricbeatDownload and install Metricbeat on your Elasticsearch nodes. […]
By default, the PROCTITLE field contains the command used to start a process, encoded in HEX. Learn how to decode it using a Ruby script within Logstash. Problem Overview When processing auditd events, the PROCTITLE field is encoded in HEX format. This makes it unreadable in its raw form. To make this information human-readable, we can use a […]
This guide is aimed at beginners and provides a step-by-step walkthrough for connecting Elasticsearch to external certificates issued by a Certificate Authority (CA). All instructions and steps are based on the official Elasticsearch documentation to ensure accuracy and compatibility. Generate a CSR for Each Node Step 1: Create a CSR Configuration FileFor each node in […]
When processing event streams with Logstash, it can be useful to record the time an event was processed and the hostname of the Logstash server handling it. This information is important for debugging, monitoring, and ensuring the traceability of event data.Logstash makes this task straightforward using the Ruby filter plugin, which allows embedding Ruby code […]