Flying with Spinnaker

Continuous integration and delivery is a best practice emerging from decades of learnings from Agile, Extreme Programming, and similar coding disciplines, all urging teams towards shorter and shorter sprints and more and more frequent releases.

Short sprints keep goals simple, and produce frequent, tangible (‘minimum viable,’ or MV) product improvements that increase value to customers. Releasing often -- in full-on CI/CD, that can even mean ‘every time a commit is approved’ -- and doing so completely under automation helps nip bugs before they develop, identifies issues affecting application performance or stability, and generally keeps problems manageable -- while also keeping coders continually productive.

Accelerating Software Delivery - Improving Software Quality

Spinnaker is an open-source, continuous-delivery platform that works on IaaS clouds (i.e., clouds that run virtual machines, e.g., GCP, OpenStack, AWS, soon VMware, etc.) and container orchestration platforms (e.g., clouds that run containers, e.g., Docker Swarm, Kubernetes, etc.). DevOps folks use Spinnaker to automatically deploy application releases to QA/test, staging, and production. Spinnaker was developed in-house by Netflix and is now being integrated in some of the largest-scale and most-demanding cloud operations in the world -- for example, by Google.

Spinnaker is engineered to manage complex and large-scale deployments gracefully, quickly, and repeatedly -- so it’s ideal for doing continuous delivery of ‘big’ application stacks and cloud-native ‘twelve-factor’ apps assembled from horizontally-scaling, containerized microservices. It works under principles of “immutability,” where applications are managed as code within Spinnaker automation ‘pipelines,’ and application components are destroyed and redeployed in the event of failure. Spinnaker provides a web interface for modifying deployment pipelines, as well as APIs for programmatic management. It can address one or several target clouds, or even multiple different cloud platforms (e.g., a premise cloud, and GCP) at the same time; so can be used to coordinate hybrid cloud operations, or application migration from one platform to another.   

Beyond just deploying applications, Spinnaker can also be used to configure and trigger automated testing, roll back failed deployments, or deploy applications to production in sophisticated ways. For example, features enabling ‘blue/green’ (or ‘red/black’) deployments (update a new ‘production’ infrastructure, then cut over traffic) and ‘canary testing’ (providing a new release only to select customers at first, before making it generally available) are built right in. Spinnaker also provides customizable application and load balancer health checks and application metrics, out of the box.

Spinnaker and Functionize

Here at Functionize, we commit numerous changes daily and are now using Spinnaker as our continuous delivery tool, along with Jenkins for continuous integration and automated testing workflow.

We have staging and production environments, both hosted on Google Cloud. Each environment has its own Load Balancer instance. Upstream from staging, code changes committed to our repo are (for the moment) being validated on a physical QA/Test machine, local to our development team. We then use the ‘functionize-release’ Spinnaker pipeline to publish the changeset from the repository to the staging environment. 

Staging (attached to Load Balancer functionize-release-tp in Gcloud) currently comprises a set of multiple Google cloud machine instances, running the pre-production version of the Functionize codebase. As part of the Spinnaker pipeline, an automated run of the Functionize Release test suite is called upon to validate the changeset / existing codebase features on the newly released staging codebase. The QA Team verifies the results of the automated Functionize test run to confirm all is well, and gives the go-ahead for final publishing to the production environment. 

Deployment to Staging

Here’s how deployment to staging works, in detail:

1. Spinnaker finds a premade virtual Linux machine image, already created by Devops in our Gcloud Project. This image contains all the base code of the Functionize app (app.functionize.com) and all our release code  (Release.functionize.com). An Apache web server is already installed on the image, preconfigured with cron and other settings.
2. Spinnaker launches the webtemplate image in the release Load Balancer group, functionize-release LB.
3. Next, Spinnaker pulls the latest code commits from our version control repositories (HG and GIT). This is done by CI tool called Jenkins, with which Spinnaker can integrate out of the box. When Spinnaker triggers Jenkins, Jenkins fires a  job called “functionize-postdeploy.” This job gets the latest code from SVN and puts it in the local web directory on the VM (this web directory is used to for previewing changes -- it’s also known as release.functionize.com.)

In the process of pulling code from repositories, Jenkins also modifies some config files so that release.functionize.com can connect with its database. Now our new changes are ready, and we have a successful database connection. Our app is ready for testing. Next, we test it using the Functionize Test API (plus some human judgments by our QA team).

Testing the Staged Release

Testing and validation works as follows:

1. API Validation:   We now execute the complete suite of Functionize test cases that will validate the release on the Staging deployment environment. We instruct Jenkins to use the Functionize Command Line tool (alternatively, we can also use the Functionize Public API) to fire a new job (Bash script) to execute the test suite - this will include a set of test cases to test and validate the important modules of the web application.
2. Here is the sample API call to run the test cases:
   ‍https://app.functionize.com/api/v1?method=processDeployment&actionFor=execute&deploymentid=1e566014eb7ajfh57b6716a018a8f680&apiKey=XXXXX
3. Now the test cases are running and verifying our new release deployment. Now we need to wait for test case completion. We get status of running task from:
   ‍https://app.functionize.com/api/v1?method=processDeployment&actionFor=status&deploymentid=1e566014e6716a0fdfd18a8f680&apiKey=XXXXXXX&run_id=1e566014eb7ajfh57b6716a0fdf8f680

If a test case fails, the pipeline will halt and the previous deployment will be restored. If each test case passes, the pipeline continues to the next step.

If all tests pass, the QA Team can then click the ‘Continue’ button in the judgement stage to green light the deployment to the production environment.

Deployment to Production

The production environment (attached to Load Balancer functionize-production-tp in Gcloud) currently comprises four Google Cloud machine instances managed by a Load Balancer that handles the web traffic to the website. The codeset is only published to the production environment after it has been successfully deployed and automatically verified successfully using Functionize automated test executions.

If the QA team passes the release deployment, the Promote to Production pipelines fire up. These are similar to the pipelines that deploy the release codebase to Staging:

1. Spinnaker finds a premade virtual machine image -- the same image that we are using as Base in our deployment to Staging.
2. Spinnaker uses the Base image to launch four virtual machines in Production Functionize-Production LB.
3. Once our four VMs are running, Spinnaker pulls the latest code from our SVN Repository, deploys it on the VMs, and modifies config files to attach the database.

Finally, we repoint the production load balancer from the VMs serving the prior released version to our new VMs, serving the latest version. Spinnaker than terminates the old VMs leaving us with a fully functional production rollout on the new VMs.

Conclusion

Spinnaker can be used to rapidly deploy new features and codesets in an Agile environment. The ability to quickly bring up new machines in a Spinnaker pipeline and replace the existing environment with the new deployment can lead to faster turnarounds for feature requests and changesets.

When combined with a CI/CD driven automated testing tool like Functionize that can be integrated directly into a Spinnaker pipeline, the validation of the newly deployed codebase on the new machines becomes easy to perform, even before it is deployed to the production environment. This can help in eradicating bugs that are introduced as part of new code releases and lead to an optimized and accurate deployment cycle.