OpenSearch Flush, Translog, and Refresh

What Is OpenSearch Flush?

In OpenSearch, flushing is the process of permanently storing data onto disk for all operations that have been temporarily stored in memory. This process is also known as a Lucene commit.

How Are OpenSearch Documents Indexed?

To understand the importance of flushing, it is essential to know how OpenSearch indexes documents.

When new documents are indexed, the operations are recorded on disk in the translog and stored in memory in a buffer. Upon an index refresh, the documents in the buffer are written to a new Lucene segment, which is also stored in memory.Flushing refers to the process of writing these in-memory segments onto disk. At the same time, it closes the current translog generation and starts a new, empty translog.

OpenSearch performs flushing in the background, using heuristics to balance memory usage and disk write operations. While flushing typically does not require user intervention, it can be manually triggered using the Flush API:

POST /my-index/_flush

In the event of a node crash or restart, OpenSearch retrieves and flushes any operations stored in the translog prior to the incident, ensuring data integrity and preventing data loss.

Additional Notes

While OpenSearch and Elasticsearch are both robust search and analytics engines, Elasticsearch offers several distinct advantages. With a more mature development history, Elasticsearch provides a richer feature set, enhanced optimizations, and a superior user experience. Our testing shows that Elasticsearch consistently outperforms OpenSearch, delivering faster results while using fewer compute resources.

Elasticsearch’s comprehensive documentation and active community forums are invaluable for troubleshooting and further optimization. Furthermore, Elastic—the company behind Elasticsearch—offers dedicated enterprise support, ensuring reliable and high-performance operations. These factors make Elasticsearch a more versatile and efficient choice for organizations with advanced search and analytics needs.