All ProjectsDynamic scaling elasticsearch clusters (ECK)

Dynamic scaling elasticsearch clusters (ECK)

By Admin July 20th 2022 00:07Big Data
A 20-node ES cluster has been deployed on GKE (Google Kubernetes Engine) with Hot-Warm Architecture, Automatic Index Rollover, Configuired Index Lifecycle Management (ILM) Policy and Automated Snapshot and Restore Mechanism to have regular backups.
Dynamic scaling elasticsearch clusters (ECK)
Implemented a dynamic scaling elasticsearch cluster using the kubernetes version of elasticsearch called ECK. The 20 node cluster has automated configurations set up for easy maintenance.

The ILM Policies allow us to automatically create a new index based on the index configuration and rollover to the new index. With Hot-Warm architecture in place, the hot nodes are used efficiently for data ingestion. After the ILM policy kicks in, the shards are moved into the warm nodes with a shrink mechanism to optimise search query rates.

The cluster can easily be monitored and managed using the Kibana Dashboard.
Additionally, we have run ES Rally tests on the cluster to benchmark the performance and find the best configuration that produces optimum results for our requirement.
Watch out this space for more details on this project....