case study
AI Tech Company
Location
Europe
Industry
AI Tech
Technologies
Elasticsearch GKE Terraform Helm BigQuery Prometheus
Background
Our client sought to optimize their cloud infrastructure costs. The project, based on Google Cloud Platform (GCP), required a balance between cost efficiency and maintaining high levels of stability, performance, and security
The Challenge
The client faced significant challenges related to the high costs of their existing cloud infrastructure. They were using Elastic Cloud to manage Elasticsearch, a solution that proved to be costly. The company needed a data management strategy and efficient use of cloud resources, leading to higher operational costs. Inefficient compute engine management and reserved instances were key issues that, if addressed, could have significantly reduced expenses and improved efficiency.
The Process
The Audit
To address these challenges, we began with a thorough assessment of the client’s current infrastructure. The audit involved an in-depth analysis of their setup on the Google Cloud Platform and billing data to identify high-cost areas, particularly those associated with Elastic Cloud and the compute engine. Our experts used many different tools to provide the audit correctly.
Migration to Google Kubernetes Engine
Following the assessment, we planned and executed a migration of the Elasticsearch service from Elastic Cloud to Google Kubernetes Engine (GKE). This migration required careful selection of optimal server configurations based on usage data. We divided the Elasticsearch cluster into three nodes, ensuring stability and data redundancy across different environments. Additionally, we implemented Infrastructure as Code (IaC) practices, allowing the client to deploy additional clusters easily in the future.
Log Storage Costs’s Optimization
To further optimize costs, we addressed the issue of high log storage expenses by transferring a significant portion of log data from Google Cloud Logging to Google BigQuery. This shift provided a more cost-effective solution while maintaining the necessary accessibility to the data.
Cost-Efficient Instance Management
We also introduced a more cost-efficient instance management strategy. For virtual machines, we recommended using Reserved Instances, which allowed the client to secure cost savings through long-term commitments to resource usage. Furthermore, we implemented Spot Instances for batch processing tasks, taking advantage of their lower costs despite the trade-off in availability and stability.
Data Lifecycle Management
Lastly, we recognized that the client needed a data lifecycle management strategy, leading to unnecessary storage costs. We established policies to automatically transfer historical and infrequently accessed data to more cost-effective storage solutions within Google Cloud Storage, optimizing their long-term storage cost.
What we have achieved
Through these targeted interventions, we achieved a 60% reduction in costs associated with maintaining Elasticsearch by migrating to Google Kubernetes Engine. This solution will allow expansion in the future if the load or amount of data increases.
The transition of log storage from Google Cloud Logging to Google BigQuery significantly reduced storage costs, making the solution much more economical with simple access at the time. Using Reserved Instances, the client realized up to 54% savings on virtual machine costs through long-term resource commitments.
Additionally, implementing Spot Instances for batch processing tasks resulted in savings of up to 89%, although this came with a trade-off regarding availability. Finally, implementing data lifecycle management policies reduced long-term storage expenses by ensuring that non-critical data was stored in more cost-effective tiers.
Our comprehensive approach to optimizing the client’s cloud infrastructure resulted in substantial cost savings and ensured that their project maintained the necessary levels of performance and security. This case highlights our expertise in delivering tailored, cost-effective solutions that drive value for our clients.
Importantly, this effort represents only the first stage of our collaboration. In the near future, we will conduct additional optimizations to further enhance the client’s infrastructure, ensuring continuous improvement and sustained value.