Architecting the OpenSearch service at CERN

Abstract

The centralised Elasticsearch service has been running at CERN since 2016, providing the search and analytics engine for numerous CERN users. The service has been based on the open-source version of Elasticsearch, surrounded by a set of external open-source plugins offering security, multitenancy, extra visualization types, and more. In October 2020, CERN embarked on an evaluation of OpenDistro for Elasticsearch, an alternative solution that used a different set of modules while retaining the core of open-source Elasticsearch. Notably, OpenDistro offered the advantage of bundling all components together, simplifying the deployment of new versions. This evaluation gained increased significance following a license change imposed by Elastic, the original creators of Elasticsearch. Consequently, the OpenDistro project was rebranded, now based on a forked version of Elasticsearch, called OpenSearch. Motivated by the license change and the streamlined deployment of the featurerich OpenSearch project as a fully open-source environment, the decision was taken to migrate the service at CERN towards it. The migration required a complete architectural redesign to accommodate the new modules while upholding the established standards of resource efficiency. The new service not only introduced a wide range of additional capabilities but also resolved longstanding maintainability issues while meeting the growing demands of various use-cases. At the time of writing, CERN’s service comprises 42 OpenSearch and 41 OpenDistro clusters in active production, plus 28 OpenSearch development clusters. This article covers the motivation, design, and implementation of this transition, highlighting the challenges encountered throughout the process.