We leveraged the power of Terraform to streamline infrastructure deployments and created Helm charts for applications to ensure their consistency and scalability. Multiple deployments to production were handled adeptly by Jenkins jobs, as we built jobs which integrated with image scanning, tagged the finalized image, and deployed it to a different environment with the help of Helm charts.
We also created a single-click rollout and rollback Jenkins job to handle 1000+ environment variables and secrets. This setup provided us with the flexibility to update or add new microservices with minimal effort. We defined a proper release strategy to roll out any new features. We also implemented the tagging feature with each image, which identifies the fixes going into the release with just one number, which helps us in case of any rollback requirements.
To ensure comprehensive and centralized logging, we harnessed Splunk, seamlessly forwarding application logs from Cloud Logging by using a Pub-Sub model.
This meticulous design allowed us to set up new environments and application services with ease. To keep a watchful eye on performance and proactively address issues, we implemented Cloud Monitoring with Grafana and PagerDuty to tackle production issues even before their possible escalation. Today, our system operates in eight diverse environments across multiple GCP projects, including four robust production environments, situated in distinct regions. With an increasing demand for additional environments, we stand ready to expand and meet the evolving needs of our clients.
A series of strategic measures were undertaken to enhance our DevOps operations significantly. First and foremost, we standardized the process for infrastructure provisioning, bringing about a new level of consistency and efficiency.
Additionally, we streamlined the process of microservices build and deployment, reducing the complexities that previously hindered our agility. To further improve our operational capabilities, we established a robust release mechanism and implemented comprehensive monitoring and alerting systems for our applications.