Managing Application Delivery with Azure DevOps and A/B Testing in Azure

What is your deployment strategy?

Let’s talk about how we can leverage managing our applications in the cloud. The cloud offers so many benefits, opening doors for users and organizations to leverage technologies that they don’t have access to in their traditional datacenter. Yes, the cloud really is someone else’s datacenter, but there are PaaS (Platform as a Service) and SaaS (Software as a Service) offerings that take away that day to day operational burden.

My goal, and many others, is to deliver apps to our end users. In order to deliver real value and quality in our applications we need to ensure they provide the features the customers want, but most importantly we need to test what we’re deploying. Testing is a tough subject, many times we put testing on the back burner, in many cases a ‘we’ll do it later approach.’ As development teams are pushing to get features out the door and meet deadlines, testing is all too often something that gets missed. Many times, we throw our code over the fence into production and cross our fingers that it works.

Let’s walk through how we can easily leverage cloud tooling to find a better setup for our deployment strategy. There are various types of deployment strategies, first you need to figure out what is the deployment strategy that works best for you and your team:

Blue/Green Deployment
Canary Deployment
Feature Flags
A/B Deployment (aka Rolling Deployment)

Blue Green Deployment

In blue green deployments the aim is to release your application in a predictable way, that will hopefully reduce downtime associated to a release and enables a quick rollback. There are usually two identical environments. The ‘green’ environment is production, ‘blue’ is staging (pre-prod or whatever naming works for you, but the key here is that it is identical). Code gets deployed to the blue environment, then it is monitored for stability and performance. Once all tests have passed, there is a cut over to blue. Blue becomes the new green. In case of performance issues, you can roll back.

Canary Deployment

Canary Deployment is a method of deployment that allows you to roll out features to a subset of your environment, that allows you to test and monitor how the release works before rolling out to the entire environment. We see canary deployments used with virtual machine scale sets in Azure or Kubernetes deployments (AKS). You can set a certain percent of the VMs/containers (25% as an example) to receive the feature/update, while the remaining infrastructure (and users) are working off the production version (in this example 75% of traffic gets routed to the prod environment). Once the release is verified, you can begin rolling out the update to the rest of the environment. To setup Canary deployments on Azure DevOps read here.

Feature Flags

Feature flags are part of Azure DevOps and are awesome. Feature flags allow you to deploy your code to a specific group of people. For instance, a new feature will get released to the IT team (called Ring 0), then once approved it will get released to a group of selected VIPs in the organization (Ring 1), then onto Departments (Ring 2), Sites (Ring 3), and then countries (Ring 4). Allowing you at any time to turn off the feature flag to roll back that deployed feature. Easy. Watch here to learn more.

A/B Deployment

A/B deployments are a great way to test features in your application and enabling you to measure the value derived from those changes. A/B deployments really stand out when you need to measure the functionality of an application. We can introduce a new version of code to a certain subset of our organization or end users. In the example below we route out traffic 50/50, so that half the users see one version of our website, the other half see the production version

Application Delivery using Azure DevOps and A/B Deployments

Hosting your applications on virtual machines is common, but when we move customers to the cloud we look at more cost effective ways to do this (also removing operational burden). Enter in the Azure WebApp. WebApps abstract the operating system layer so that we can deploy code first. I am not going to go into all of the features of WebApps, so I suggest reading up on them here.

The feature that I really love most is the deployment slot. A deployment slot is a live app (you can copy your primary app or start net new) with its own host name, effectively giving you a staging deployment slot. Copying your primary app content and configuration allows you to have an exact replica of your app. Test your changes in a like for like environment before swapping that staging slot to production. This saves standing up a whole new infrastructure and paying for that environment, deployment slots are free (okay, kinda) as they are built into the AppService Plan.

First thing is first, we have an environment in Azure, the front end website is hosted on Azure WebApps, with some databases on the backend. We have a website that serves our customers and have been asked to make a change to our website by our marketing team.

They want to know if the free shipping threshold has an effect on how much a customer purchases. Our end goal is to deploy two different versions of our website, allowing the marketing team to pull hard metrics on how it affected our customers and overall margin over the weekend. Our original offer on our website shows free shipping with a spend of $100, we will configure a second version of our website to show a different threshold of $250. We will leverage the built in load balancer in WebApps to route 50% of the traffic to each version of the website.

We are going to make changes to our code in VSCode which will push the changes to our code repository in GitHub. We have setup a trigger that when any code is committed in GitHub, it will trigger a build in our Azure DevOps Pipelines. Our build will run, performing some tests and then package it, ready to deploy to our release pipeline.

Once our code successfully passes all of the tests it will deploy into our release pipeline. The code will first be deployed into our staging environment, where we can test performance of our website and verify the new features. Once successful, a manual approval gate must be approved to allow our deployment to proceed to our staging slot.

Once the deployment to the staging slot is complete, we will have two versions of our website, each with the different shipping thresholds. With an end result of having 50% of our users hitting our canary website (showing the higher shipping threshold) and 50% of the users hitting our production website (showing our original shipping threshold).

In summary, we leveraged a multi-stage Azure Pipeline to release our code into the different environments, once our code has been tested, built and ready to ship. Each stage of our pipeline will deploy into a specified deployment slot. The diagram below gives a good architectural overview of what has been configured: